ArticleJournal of biomedical informatics2011
Semi-automatic semantic annotation of PubMed queries: a study on quality, efficiency, satisfaction.
Article in Journal of biomedical informatics, 2011. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 45 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
45 citing papers in PubMed.
- Efficient extraction of medication information from clinical notes: an evaluation in 2 languages.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review.Findings of ACL. ACL · 2025Article
- Research on the proximity relationships of psychosomatic disease knowledge graph modules extracted by large language models.Scientific reports · 2025Article
- Adverse drug event detection using natural language processing: A scoping review of supervised learning methods.PloS one · 2023Article
- Identification of Chemical-Disease Associations Through Integration of Molecular Fingerprint, Gene Ontology and Pathway Information.Interdisciplinary sciences, computational life sciences · 2022Article
- AnthraxKP: a knowledge graph-based, Anthrax Knowledge Portal mined from biomedical literature.Database : the journal of biological databases and curation · 2022Article
- Constructing knowledge graphs and their biomedical applications.Computational and structural biotechnology journal · 2020Review
- Chemical-induced disease relation extraction via attention-based distant supervision.BMC bioinformatics · 2019Article
- Knowledge-guided convolutional networks for chemical-disease relation extraction.BMC bioinformatics · 2019Article
- Biomedical text mining for research rigor and integrity: tasks, challenges, directions.Briefings in bioinformatics · 2018Article
- A document level neural model integrated domain knowledge for chemical-induced disease relations.BMC bioinformatics · 2018Article
- OC-2-KB: integrating crowdsourcing into an obesity and cancer knowledge base curation system.BMC medical informatics and decision making · 2018Article
- Towards PubMed 2.0.eLife · 2017Article
- MeSH Now: automatic MeSH indexing at PubMed scale via learning to rank.Journal of biomedical semantics · 2017Article
- PheKnow-Cloud: A Tool for Evaluating High-Throughput Phenotype Candidates using Online Medical Literature.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2017Article
- Chemical-induced disease relation extraction via convolutional neural network.Database : the journal of biological databases and curation · 2017Article
- Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical-disease relation (CDR) task.Database : the journal of biological databases and curation · 2016Article
- Chemical-induced disease relation extraction with various linguistic features.Database : the journal of biological databases and curation · 2016Article
- AuDis: an automatic CRF-enhanced disease normalization in biomedical text.Database : the journal of biological databases and curation · 2016Article
- BioCreative V CDR task corpus: a resource for chemical disease relation extraction.Database : the journal of biological databases and curation · 2016Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
Abstract
Information processing algorithms require significant amounts of annotated data for training and testing. The availability of such data is often hindered by the complexity and high cost of production. In this paper, we investigate the benefits of a state-of-the-art tool to help with the semantic annotation of a large set of biomedical queries. Seven annotators were recruited to annotate a set of 10,000 PubMed® queries with 16 biomedical and bibliographic categories. About half of the queries were annotated from scratch, while the other half were automatically pre-annotated and manually corrected. The impact of the automatic pre-annotations was assessed on several aspects of the task: time, number of actions, annotator satisfaction, inter-annotator agreement, quality and number of the resulting annotations. The analysis of annotation results showed that the number of required hand annotations is 28.9% less when using pre-annotated results from automatic tools. As a result, the overall annotation time was substantially lower when pre-annotations were used, while inter-annotator agreement was significantly higher. In addition, there was no statistically significant difference in the semantic distribution or number of annotations produced when pre-annotations were used. The annotated query corpus is freely available to the research community. This study shows that automatic pre-annotations are found helpful by most annotators. Our experience suggests using an automatic tool to assist large-scale manual annotation projects. This helps speed-up the annotation time and improve annotation consistency while maintaining high quality of the final annotations.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.