ArticleJournal of the American Medical Informatics Association : JAMIA
Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements.
Article in Journal of the American Medical Informatics Association : JAMIA. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 28 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
28 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Clinical Natural Language Processing in 2014: Foundational Methods Supporting Efficient Healthcare.Yearbook of medical informatics · 2015Pooled it
- Semantic Similarity Search Approach to Extract Exemplars of Stigmatizing and Positive Language in Obstetric Clinical Notes: Exploratory Study.JMIR medical informatics · 2026Article
- Computable longitudinal patient journeys from structured and unstructured EHR data.Nature medicine · 2026Article
- Automated sleep stage and event detection algorithms using quality-controlled polysomnography annotations.Sleep advances : a journal of the Sleep Research Society · 2026Article
- Named Entity Recognition for Chinese Cancer Electronic Health Records-Development and Evaluation of a Domain-Specific BERT Model: Quantitative Study.JMIR medical informatics · 2025Article
- A clinical narrative corpus on nut allergy: annotation schema, guidelines and use case.Scientific data · 2025Article
- CACER: Clinical concept Annotations for Cancer Events and Relations.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- Utilizing active learning strategies in machine-assisted annotation for clinical named entity recognition: a comprehensive analysis considering annotation costs and target effectiveness.Journal of the American Medical Informatics Association : JAMIA · 2024Article
- A case study in applying artificial intelligence-based named entity recognition to develop an automated ophthalmic disease registry.Graefe's archive for clinical and experimental ophthalmology = Albrecht von Graefes Archiv fur klinische und experimentelle Ophthalmologie · 2023Article
- MedLexSp - a medical lexicon for Spanish medical natural language processing.Journal of biomedical semantics · 2023Article
- Adverse drug event detection using natural language processing: A scoping review of supervised learning methods.PloS one · 2023Article
- The OpenDeID corpus for patient de-identification.Scientific reports · 2021Article
- Identification of social determinants of health using multi-label classification of electronic health record clinical notes.JAMIA open · 2021Article
- A clinical trials corpus annotated with UMLS entities to enhance the access to evidence-based medicine.BMC medical informatics and decision making · 2021Article
- ADE Eval: An Evaluation of Text Processing Systems for Adverse Event Extraction from Drug Labels for Pharmacovigilance.Drug safety · 2021Article
- Adverse drug event presentation and tracking (ADEPT): semiautomated, high throughput pharmacovigilance using real-world data.JAMIA open · 2020Article
- Constructing knowledge graphs and their biomedical applications.Computational and structural biotechnology journal · 2020Review
- Core Data Elements in Acute Myeloid Leukemia: A Unified Medical Language System-Based Semantic Analysis and Experts' Review.JMIR medical informatics · 2019Article
- Automated Risk Assessment for School Violence: a Pilot Study.The Psychiatric quarterly · 2018Article
- A Pilot Study on Developing a Standardized and Sensitive School Violence Risk Assessment with Manual Annotation.The Psychiatric quarterly · 2017Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
objectiveTo present a series of experiments: (1) to evaluate the impact of pre-annotation on the speed of manual annotation of clinical trial announcements; and (2) to test for potential bias, if pre-annotation is utilized.
methodsTo build the gold standard, 1400 clinical trial announcements from the clinicaltrials.gov website were randomly selected and double annotated for diagnoses, signs, symptoms, Unified Medical Language System (UMLS) Concept Unique Identifiers, and SNOMED CT codes. We used two dictionary-based methods to pre-annotate the text. We evaluated the annotation time and potential bias through F-measures and ANOVA tests and implemented Bonferroni correction.
resultsTime savings ranged from 13.85% to 21.5% per entity. Inter-annotator agreement (IAA) ranged from 93.4% to 95.5%. There was no statistically significant difference for IAA and annotator performance in pre-annotations.
conclusionsOn every experiment pair, the annotator with the pre-annotated text needed less time to annotate than the annotator with non-labeled text. The time savings were statistically significant. Moreover, the pre-annotation did not reduce the IAA or annotator performance. Dictionary-based pre-annotation is a feasible and practical method to reduce the cost of annotation of clinical named entity recognition in the eligibility sections of clinical trial announcements without introducing bias in the annotation process.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.