ArticleAnnals of internal medicine2025
Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews.
Article in Annals of internal medicine, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 17 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
17 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Artificial Intelligence-Supported Evidence Synthesis: A Case Study of Smart Infusion Pump Interoperability.Medical sciences (Basel, Switzerland) · 2026Pooled it
- Comparative efficacy of mind-body exercises on quality of life, anxiety, and depression in patients with breast cancer: a systematic review with pairwise and network meta-analysis of randomized controlled trials.Frontiers in public health · 2026Pooled it
- Association of diffuse idiopathic skeletal hyperostosis with postoperative outcomes following surgery for degenerative lumbar spine disease.European spine journal : official publication of the European Spine Society, the European Spinal Deformity Society, and the European Section of the Cervical Spine Research Society · 2026Article
- Health Care Analytics Challenges: A 3-Pillar Framework Connecting Analytics Maturity, Workforce Agility, and Technical Enablement.Interactive journal of medical research · 2026Article
- How Often Do Large Language Models Agree with Each Other-And with the Truth? A Consensus- and Complexity-Stratified Analysis of Data Extraction for Neuroimaging AI.Journal of clinical medicine · 2026Article
- Adding Artificial Intelligence to the Surgeon's Domain.World journal of surgery · 2026Review
- Automated data extraction for systematic reviews using GPT-5.2 and Google Gemini Pro 3: A dual-large language model approach in orthopaedic research.Knee surgery, sports traumatology, arthroscopy : official journal of the ESSKA · 2026Article
- Pharmacogenetic Predictors of Chemotherapy Treatment-Related Toxicities in Paediatric and Adolescent Acute Lymphoblastic Leukemia: A Systematic Review, Meta-Analysis and Literature-Based Candidate Prioritization.Pharmaceuticals (Basel, Switzerland) · 2026Review
- Large language models vs. conventional search in identifying delirium guidelines for a scoping review.BMC medical research methodology · 2026Article
- Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial.Open forum infectious diseases · 2026Article
- Evidence-Based Medicine Meets Artificial Intelligence: Reframing Critical Appraisal in Emergency Medicine.Academic emergency medicine : official journal of the Society for Academic Emergency Medicine · 2026Article
- Anticoagulation therapy for Lemierre syndrome: a comprehensive review and individual participant data analysis.Thrombosis journal · 2026Article
- Evaluating Large Language Models for Automated Evidence Synthesis in Neuroimaging AI: A Multi-Model Benchmark.Journal of clinical medicine · 2026Article
- Automating data extraction in meta-research: A multi-model benchmark in network psychometrics papers.Behavior research methods · 2026Article
- Automated Tumor and Node Staging from Esophageal Cancer Endoscopic Ultrasound Reports: A Benchmark of Advanced Reasoning Models with Prompt Engineering and Cross-Lingual Evaluation.Diagnostics (Basel, Switzerland) · 2026Article
- Artificial intelligence-assisted photodynamic diagnosis and photodynamic therapy against cancer.Frontiers in oncology · 2026Review
- Responsible Integration of Artificial Intelligence in Rapid Reviews: A Position Statement From the Cochrane Rapid Reviews Methods Group.Cochrane evidence synthesis and methods · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
18 authors.
Funding
Abstract
backgroundData extraction is a critical but error-prone and labor-intensive task in evidence synthesis. Unlike other artificial intelligence (AI) technologies, large language models (LLMs) do not require labeled training data for data extraction.
objectiveTo compare an AI-assisted versus a traditional, human-only data extraction process.
designStudy within reviews (SWAR) using a prospective, parallel-group comparison with blinded data adjudicators.
settingWorkflow validation within 6 ongoing systematic reviews of interventions under real-world conditions.
interventionInitial data extraction using an LLM (Claude, versions 2.1, 3.0 Opus, and 3.5 Sonnet) verified by a human reviewer. MEASUREMENTS: Concordance, time on task, accuracy, sensitivity, positive predictive value, and error analysis.
resultsThe 6 systematic reviews in the SWAR yielded 9341 data elements from 63 studies. Concordance between the 2 methods was 77.2% (95% CI, 76.3% to 78.0%). Compared with the reference standard, the AI-assisted approach had an accuracy of 91.0% (CI, 90.4% to 91.6%) and the human-only approach an accuracy of 89.0% (CI, 88.3% to 89.6%). Sensitivities were 89.4% (CI, 88.6% to 90.1%) and 86.5% (CI, 85.7% to 87.3%), respectively, with positive predictive values of 99.2% (CI, 99.0% to 99.4%) and 98.9% (CI, 98.6% to 99.1%). Incorrect data were extracted in 9.0% (CI, 8.4% to 9.6%) of AI-assisted cases and 11.0% (CI, 10.4% to 11.7%) of human-only cases, with corresponding proportions of major errors of 2.5% (CI, 2.2% to 2.8%) versus 2.7% (CI, 2.4% to 3.1%). Missed data items were the most frequent error type in both approaches. The AI-assisted method reduced data extraction time by a median of 41 minutes per study. LIMITATIONS: Assessing concordance and classifying errors required subjective judgment. Consistently tracking time on task was challenging.
conclusionData extraction assisted by AI may offer a viable, more efficient alternative to human-only methods. PRIMARY FUNDING SOURCE: Agency for Healthcare Research and Quality and RTI International.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.