ArticleJournal of the American Medical Informatics Association : JAMIA
Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance.
Article in Journal of the American Medical Informatics Association : JAMIA. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 34 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
34 citing papers in PubMed.
- Linguistic summarization of in-home sensor data.Journal of biomedical informatics · 2019Trial
- Comparing five generative AI chatbots' answers to LLM-generated clinical questions with medical information scientists' evidence summaries.Journal of the Medical Library Association : JMLA · 2026Article
- Lightweight open-source large language models versus cTAKES for information extraction from discharge summaries: tobacco smoking status test case.JAMIA open · 2026Article
- A methodological framework for validating a multi-domain physiological sensor for divers using a scalable data fusion platform.Frontiers in physiology · 2026Article
- Comparing Five Generative AI Chatbots' Answers to LLM-Generated Clinical Questions with Medical Information Scientists' Evidence Summaries.medRxiv : the preprint server for health sciences · 2025Article
- Accuracy of the diagnosis of pneumonia in Canadian pediatric emergency departments: A prospective cohort study.PloS one · 2024Article
- Linguistic and ontological challenges of multiple domains contributing to transformed health ecosystems.Frontiers in medicine · 2023Article
- Initial Development of an Automated Platform for Assessing Trainee Performance on Case Presentations.ATS scholar · 2022Article
- Measurement Strategies for Evidence-Based Psychotherapy for Posttraumatic Stress Disorder Delivery: Trends and Associations with Patient-Reported Outcomes.Administration and policy in mental health · 2020Article
- Comparison of the cohort selection performance of Australian Medicines Terminology to Anatomical Therapeutic Chemical mappings.Journal of the American Medical Informatics Association : JAMIA · 2019Article
- Computer-based automatic classification of trabecular bone pattern can assist radiographic bone quality assessment at dental implant site.The British journal of radiology · 2018Article
- Diagnostic Accuracy of Ophthalmoscopy vs Telemedicine in Examinations for Retinopathy of Prematurity.JAMA ophthalmology · 2018Article
- Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems.Journal of the American Medical Informatics Association : JAMIA · 2017Article
- Lessons learned in replicating data-driven experiments in multiple medical systems and patient populations.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2013Article
- A study of terminology auditors' performance for UMLS semantic type assignments.Journal of biomedical informatics · 2012Article
- Improving Clinical Trial Participant Tracking Tools Using Knowledge-anchored Design Methodologies.Applied clinical informatics · 2010Article
- Developing a manually annotated clinical document corpus to identify phenotypic information for inflammatory bowel disease.BMC bioinformatics · 2009Article
- Developing a manually annotated clinical document corpus to identify phenotypic information for inflammatory bowel disease.Summit on translational bioinformatics · 2009Article
- Supporting the design of translational clinical studies through the generation and verification of conceptual knowledge-anchored hypotheses.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2008Article
- Conceptual knowledge acquisition in biomedicine: A methodological review.Journal of biomedical informatics · 2007Review
Corrections and comments
- Commented on by
Authors and funding
2 authors.
Funding
Abstract
Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even when performance can be assessed, it is not always clear whether the performance is sufficient or reasonable. These challenges can be addressed if an evaluator enlists the help of clinical domain experts. 1) The experts can carry out the same tasks as the system, and then their responses can be combined to generate a reference standard. 2)The experts can judge the appropriateness of system output directly. 3) The experts can serve as comparison subjects with which the system can be compared. These are separate roles that have different implications for study design, metrics, and issues of reliability and validity. Diagrams help delineate the roles of experts in complex study designs.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.