ArticlePacific Symposium on Biocomputing. Pacific Symposium on Biocomputing2015
Microtask crowdsourcing for disease mention annotation in PubMed abstracts.
Article in Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing, 2015. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 24 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
24 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Mapping of Crowdsourcing in Health: Systematic Review.Journal of medical Internet research · 2018Pooled it
- Improving Electronic Health Record Note Comprehension With NoteAid: Randomized Trial of Electronic Health Record Note Comprehension Interventions With Crowdsourced Workers.Journal of medical Internet research · 2019Trial
- Individual Factors That Affect Laypeople's Understanding of Definitions of Medical Jargon.Health policy and technology · 2024Article
- Comparing the Recruitment of Research Participants With Chronic Low Back Pain Using Amazon Mechanical Turk With the Recruitment of Patients From Chiropractic Clinics: A Quasi-Experimental Study.Journal of manipulative and physiological therapeutics · 2021Article
- Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology.JMIR medical informatics · 2021Article
- Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation.Drug safety · 2021Article
- A hybrid approach toward biomedical relation extraction training corpora: combining distant supervision with crowdsourcing.Database : the journal of biological databases and curation · 2020Article
- Applying citizen science to gene, drug and disease relationship extraction from biomedical abstracts.Bioinformatics (Oxford, England) · 2020Article
- BioHackathon 2015: Semantics of data for life sciences and reproducible research.F1000Research · 2020Article
- Automated assessment of biological database assertions using the scientific literature.BMC bioinformatics · 2019Article
- ComprehENotes, an Instrument to Assess Patient Reading Comprehension of Electronic Health Record Notes: Development and Validation.Journal of medical Internet research · 2018Article
- Comparing Amazon's Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature.Journal of general internal medicine · 2018Review
- An annotated corpus with nanomedicine and pharmacokinetic parameters.International journal of nanomedicine · 2017Article
- Crowdsourcing: an overview and applications to ophthalmology.Current opinion in ophthalmology · 2016Review
- Crowdsourcing in biomedicine: challenges and opportunities.Briefings in bioinformatics · 2016Review
- Linking rare and common disease: mapping clinical disease-phenotypes to ontologies in therapeutic target validation.Journal of biomedical semantics · 2016Article
- RegenBase: a knowledge base of spinal cord injury biology for translational research.Database : the journal of biological databases and curation · 2016Article
- MET network in PubMed: a text-mined network visualization and curation system.Database : the journal of biological databases and curation · 2016Article
- Community-Reviewed Biological Network Models for Toxicology and Drug Discovery Applications.Gene regulation and systems biology · 2016Article
- Crowdsourcing and curation: perspectives from biology and natural language processing.Database : the journal of biological databases and curation · 2016Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
Abstract
Identifying concepts and relationships in biomedical text enables knowledge to be applied in computational analyses. Many biological natural language processing (BioNLP) projects attempt to address this challenge, but the state of the art still leaves much room for improvement. Progress in BioNLP research depends on large, annotated corpora for evaluating information extraction systems and training machine learning models. Traditionally, such corpora are created by small numbers of expert annotators often working over extended periods of time. Recent studies have shown that workers on microtask crowdsourcing platforms such as Amazon's Mechanical Turk (AMT) can, in aggregate, generate high-quality annotations of biomedical text. Here, we investigated the use of the AMT in capturing disease mentions in PubMed abstracts. We used the NCBI Disease corpus as a gold standard for refining and benchmarking our crowdsourcing protocol. After several iterations, we arrived at a protocol that reproduced the annotations of the 593 documents in the 'training set' of this gold standard with an overall F measure of 0.872 (precision 0.862, recall 0.883). The output can also be tuned to optimize for precision (max = 0.984 when recall = 0.269) or recall (max = 0.980 when precision = 0.436). Each document was completed by 15 workers, and their annotations were merged based on a simple voting method. In total 145 workers combined to complete all 593 documents in the span of 9 days at a cost of $.066 per abstract per worker. The quality of the annotations, as judged with the F measure, increases with the number of workers assigned to each task; however minimal performance gains were observed beyond 8 workers per task. These results add further evidence that microtask crowdsourcing can be a valuable tool for generating well-annotated corpora in BioNLP. Data produced for this analysis are available at http://figshare.com/articles/Disease_Mention_Annotation_with_Mechanical_Turk/1126402.
Indexed as
Identifiers
25592589PMC4299946What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.