ArticleJournal of the American Medical Informatics Association : JAMIA2025
Collaborative large language models for automated data extraction in living systematic reviews.
Article in Journal of the American Medical Informatics Association : JAMIA, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
15 citing papers in PubMed.
- Structured Schemas for Provenance-Rich, LLM-Assisted QSP Model Calibration.CPT: pharmacometrics & systems pharmacology · 2026Article
- From API to Action: A Multi-Model Comparison of OpenAI, Anthropic, Google, and Meta LLMs for Clinical Trial Data Extraction.Bioengineering (Basel, Switzerland) · 2026Article
- Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial.Open forum infectious diseases · 2026Article
- Scope, Prospects, and Limitations of Living Systematic Reviews in Medical Literature: A Narrative Overview.Cureus · 2026Review
- Applications of Large Language Models in Medical Research: From Systematic Reviews to Clinical Studies.Bioengineering (Basel, Switzerland) · 2026Review
- Beyond human gold standards: A multimodel framework for automated abstract classification and information extraction.Research synthesis methods · 2026Article
- Collaborative large language models (LLMs) are all you need for screening in systematic reviews.medRxiv : the preprint server for health sciences · 2026Article
- How large language models can help us write a systematic review.Intensive care medicine · 2026Article
- The Phases of Living Evidence Synthesis Using AI AI: Living Evidence Synthesis (Version 1).Journal of medical Internet research · 2026Article
- Comparison of three large language models' ability to assess the risk of bias using ROBINS-I tool.BMJ digital health & AI · 2026Article
- Leveraging AI for Meta-Analysis: Evaluating LLMs in Detecting Publication Bias for Next-Generation Evidence Synthesis.Cochrane evidence synthesis and methods · 2025Article
- Review
- Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers.Cochrane evidence synthesis and methods · 2025Article
- Harnessing the power of large language models for clinical tasks and synthesis of scientific literature.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Advancing the application and evaluation of large language models in health and biomedicine.Journal of the American Medical Informatics Association : JAMIA · 2025Article
Corrections and comments
- Update of
Authors and funding
21 authors.
Funding
Abstract
objectiveData extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. MATERIALS AND
methodsA dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance.
resultsIn the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76. DISCUSSION: Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy.
conclusionLarge language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly "living" systematic reviews.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.