Evidence map›Paper›PMID 42878908›Full record

ArticleJournal of medical systems2026

Development and Exploratory Evaluation of a Large Language Model-Supported Scoping Review Workflow.

Frederick H Kuo, Jamie L Fierstein, Madeline de la Concha, Yasmin Saeed, Ryan Shargo, Dina Ashour, Ernest Amankwah, Ellis Crabtree, Mohamed A Rehman, Peter Najjar and 3 more

Abstract readScoping Review
PubMed Publisher
In one paragraph

Article in Journal of medical systems, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.

0numbers the graph read from it
0cells of the map it votes in
0citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

0 citing papers in PubMed.

No citing paper in PubMed yet.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

13 authors.

Frederick H KuoDepartment of Anesthesia and Pain Medicine, Johns Hopkins All Children's Hospital, 601 5th St South, Suite C725, St Petersburg, 33701, FL, USA. frederick.kuo@jhmi.edu.ORCID https://orcid.org/0009-0002-5983-1370
Jamie L FiersteinDepartment of Anesthesiology and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
Madeline de la ConchaDepartment of Biomedical Engineering, California Polytechnic State University, San Luis Obispo, CA, USA.
Yasmin SaeedLake Erie College of Osteopathic Medicine, Bradenton, FL, USA.
Ryan ShargoUniversity of South Florida, Morsani College of Medicine, Tampa, FL, USA.
Dina AshourEpidemiology and Biostatistics Shared Resource, Institute for Clinical and Translational Research, Johns Hopkins All Children's Hospital, St Petersburg, FL, USA.
Ernest AmankwahEpidemiology and Biostatistics Shared Resource, Institute for Clinical and Translational Research, Johns Hopkins All Children's Hospital, St Petersburg, FL, USA.
Ellis CrabtreeDepartment of Anesthesiology and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
Mohamed A RehmanDepartment of Anesthesia and Pain Medicine, Johns Hopkins All Children's Hospital, 601 5th St South, Suite C725, St Petersburg, 33701, FL, USA.
Peter NajjarColorectal Research Unit, Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
Khyzer AzizDepartment of Pediatrics, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
Brant H TudorDepartment of Anesthesiology and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
Luis M AhumadaDepartment of Anesthesiology and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

Large language models (LLMs) have demonstrated human-level performance in literature review screening phases and have been evaluated in systematic-review workflows. However, LLM performance in scoping reviews, which are broader and more heterogeneous, remains understudied. We developed a modular LLM-supported framework for scoping-review workflows and compared the performance of two general-purpose LLMs against human reviewers on a reference scoping review. The reference review's keyword search results (n = 1336) and prompt sets mirroring workflow stages were provided to GPT-5 and gpt-oss-20b (OpenAI, August 2025). Outputs were benchmarked against human decisions for agreement; disagreements were adjudicated. For title-abstract screening and final-inclusion concordance, unanimous and adjudicated decisions were combined into a composite reference standard. In title-abstract screening, GPT-5 and gpt-oss-20b workflows showed substantial interrater agreement with human consensus (AC1: 0.769; 0.722); reference concordance was 90.9% (p = 0.08) and 92.4% (p = 0.004) versus humans' 89.0%. For final inclusion status, interrater agreement was almost-perfect (AC1: 0.948; 0.853); reference concordance was 95.4% (p = 0.23) and 92.4% (p < 0.001) versus humans' 96.1%. Across data categorization/synthesis, interrater agreement was fair to almost-perfect (AC1: 0.493-0.949; 0.342-0.929); total variation distances (representing distribution divergence) were 1.55%-28.0%. All sources found comparably low definition-adherent use rates of the term "human digital twin" (humans 12.1%; GPT-5 13.0%; gpt-oss-20b 18.0%). Both LLM-supported workflows achieved fair to almost-perfect agreement and high screening/final-inclusion reference concordance. The main synthesis finding was directionally consistent across workflows, with all sources identifying infrequent correct use of "human digital twin"; total variation distance may be useful in assessing distribution-level concordance. Limitations include a single domain/vendor, fixed prompts, and predefined taxonomies; broader evaluations are warranted. In this single-domain benchmark, LLM-supported scoping-review workflows showed feasibility for supporting and potentially streamlining selected review stages, with broadly comparable performance on evaluated endpoints and conclusion-level concordance. Generalizability, reproducibility, and operational efficiency beyond this benchmark remain to be established.

Indexed as

Large Language ModelsWorkflowHumansAbstract screeningArtificial intelligenceLarge language modelsResearch questionsScoping reviews

Identifiers

PMID42878908

What OpenQuestion holds

Textmetadata
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.