ArticlePloS one2025
ChatGPT-4o can serve as the second rater for data extraction in systematic reviews.
Article in PloS one, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 23 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
23 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Generative artificial intelligence use in evidence synthesis: A systematic review.Research synthesis methods · 2025Pooled it
- Moving towards acceleration with accountability: a conceptual framework for AI-assisted systematic reviews.Global epidemiology · 2026Article
- Association of diffuse idiopathic skeletal hyperostosis with postoperative outcomes following surgery for degenerative lumbar spine disease.European spine journal : official publication of the European Spine Society, the European Spinal Deformity Society, and the European Section of the Cervical Spine Research Society · 2026Article
- How Often Do Large Language Models Agree with Each Other-And with the Truth? A Consensus- and Complexity-Stratified Analysis of Data Extraction for Neuroimaging AI.Journal of clinical medicine · 2026Article
- Automated data extraction for systematic reviews using GPT-5.2 and Google Gemini Pro 3: A dual-large language model approach in orthopaedic research.Knee surgery, sports traumatology, arthroscopy : official journal of the ESSKA · 2026Article
- Treatment for Hyperosmolar Dehydration in Hospitalised Adults: Protocol for a Scoping Review of Current Evidence and Gaps.JMIR research protocols · 2026Article
- Evaluating the Utility of Artificial Intelligence in Conducting Systematic Reviews.Arthroplasty today · 2026Article
- Using Generative AI to Appraise the Quality of Medical Education Research Studies: Agreement Between AI-Generated and Human MERSQI Scores.AEM education and training · 2026Article
- Evaluating Large Language Models for Automated Evidence Synthesis in Neuroimaging AI: A Multi-Model Benchmark.Journal of clinical medicine · 2026Article
- Role of allied health professions in falls prevention for community-dwelling older adults: a scoping review.BMJ open · 2026Article
- Applications of Large Language Models in Medical Research: From Systematic Reviews to Clinical Studies.Bioengineering (Basel, Switzerland) · 2026Review
- Application of large language models to the annotation of cell lines and mouse strains in genomics data.bioRxiv : the preprint server for biology · 2026Article
- Machine learning-assisted abstract screening on learning analytics: a step-by-step tutorial.Systematic reviews · 2026Article
- How large language models can help us write a systematic review.Intensive care medicine · 2026Article
- Application of large language models to the annotation of cell lines and mouse strains in genomics data.Database : the journal of biological databases and curation · 2026Article
- Performance of large language models and prompt engineering strategies for data extraction in systematic reviews.Frontiers in digital health · 2026Article
- Artificial Intelligence-Assisted Data Extraction With a Large Language Model: A Study Within Reviews.Annals of internal medicine · 2025Article
- Artificial intelligence in contrast enhanced ultrasound: A new era for liver lesion assessment.World journal of gastroenterology · 2025Review
- Responsible Integration of Artificial Intelligence in Rapid Reviews: A Position Statement From the Cochrane Rapid Reviews Methods Group.Cochrane evidence synthesis and methods · 2025Article
- Assessing the Feasibility and Acceptability of a Bespoke Large Language Model Pipeline to Extract Data From Different Study Designs for Public Health Evidence Reviews.Cochrane evidence synthesis and methods · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundSystematic reviews provide clarity of a bulk of evidence and support the transfer of knowledge from clinical trials to guidelines. Yet, they are time-consuming. Artificial intelligence (AI), like ChatGPT-4o, may streamline processes of data extraction, but its efficacy requires validation.
objectiveThis study aims to (1) evaluate the validity of ChatGPT-4o for data extraction compared to human reviewers, and (2) test the reproducibility of ChatGPT-4o's data extraction.
methodsWe conducted a comparative study using papers from an ongoing systematic review on exercise to reduce fall risk. Data extracted by ChatGPT-4o were compared to a reference standard: data extracted by two independent human reviewers. The validity was assessed by categorizing the extracted data into five categories ranging from completely correct to false data. Reproducibility was evaluated by comparing data extracted in two separate sessions using different ChatGPT-4o accounts.
resultsChatGPT-4o extracted a total of 484 data points across 11 papers. The AI's data extraction was 92.4% accurate (95% CI: 89.5% to 94.5%) and produced false data in 5.2% of cases (95% CI: 3.4% to 7.4%). The reproducibility between the two sessions was high, with an overall agreement of 94.1%. Reproducibility decreased when information was not reported in the papers, with an agreement of 77.2%.
conclusionValidity and reproducibility of ChatGPT-4o was high for data extraction for systematic reviews. ChatGPT-4o was qualified as a second reviewer for systematic reviews and showed potential for future advancements when summarizing data.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.