ArticleJournal of the American Medical Informatics Association : JAMIA2015
Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT.
Article in Journal of the American Medical Informatics Association : JAMIA, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 25 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
25 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Mapping of Crowdsourcing in Health: Systematic Review.Journal of medical Internet research · 2018Pooled it
- Improving Electronic Health Record Note Comprehension With NoteAid: Randomized Trial of Electronic Health Record Note Comprehension Interventions With Crowdsourced Workers.Journal of medical Internet research · 2019Trial
- Using logical constraints to validate statistical information about disease outbreaks in collaborative knowledge graphs: the case of COVID-19 epidemiology in Wikidata.PeerJ. Computer science · 2022Article
- Comparing the Recruitment of Research Participants With Chronic Low Back Pain Using Amazon Mechanical Turk With the Recruitment of Patients From Chiropractic Clinics: A Quasi-Experimental Study.Journal of manipulative and physiological therapeutics · 2021Article
- Crowdsourcing biocuration: The Community Assessment of Community Annotation with Ontologies (CACAO).PLoS computational biology · 2021Article
- A Neuro-ontology for the neurological examination.BMC medical informatics and decision making · 2020Article
- OC-2-KB: integrating crowdsourcing into an obesity and cancer knowledge base curation system.BMC medical informatics and decision making · 2018Article
- Applications of crowdsourcing in health: an overview.Journal of global health · 2018Review
- ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation.Journal of medical Internet research · 2018Article
- Assessing the practice of biomedical ontology evaluation: Gaps and opportunities.Journal of biomedical informatics · 2018Review
- Crowd-sourced Ontology for Photoleukocoria: Identifying Common Internet Search Terms for a Potentially Important Pediatric Ophthalmic Sign.Translational vision science & technology · 2018Article
- Analyzing structural changes in SNOMED CT's Bacterial infectious diseases using a visual semantic delta.Journal of biomedical informatics · 2017Article
- Efficient Results in Semantic Interoperability for Health Care. Findings from the Section on Knowledge Representation and Management.Yearbook of medical informatics · 2016Article
- A unified software framework for deriving, visualizing, and exploring abstraction networks for ontologies.Journal of biomedical informatics · 2016Article
- Crowdsourcing biomedical research: leveraging communities as innovation engines.Nature reviews. Genetics · 2016Review
- Utilizing a structural meta-ontology for family-based quality assurance of the BioPortal ontologies.Journal of biomedical informatics · 2016Article
- Is the crowd better as an assistant or a replacement in ontology engineering? An exploration through the lens of the Gene Ontology.Journal of biomedical informatics · 2016Article
- Crowdsourcing in biomedicine: challenges and opportunities.Briefings in bioinformatics · 2016Review
- Tracking the Remodeling of SNOMED CT's Bacterial Infectious Diseases.AMIA ... Annual Symposium proceedings. AMIA Symposium · 2016Article
- The Drug Data to Knowledge Pipeline: Large-Scale Claims Data Classification for Pharmacologic Insight.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2016Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
Abstract
objectivesThe verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification.
methodsWe developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships.
resultsThe crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (κ) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, 'septic shock is a soft-tissue infection'). DISCUSSION: The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks.
conclusionsWe have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.