Evidence map›Paper›PMID 39814458›Full record

ArticleThe British journal of ophthalmology2025

Can large language models fully automate or partially assist paper selection in systematic reviews?

Haichao Chen, Zehua Jiang, Xinyu Liu, Can Can Xue, Samantha Min Er Yew, Bin Sheng, Ying-Feng Zheng, Xiaofei Wang, You Wu, Sobha Sivaprasad and 3 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in The British journal of ophthalmology, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 6 papers.

0numbers the graph read from it
0cells of the map it votes in
6citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

6 citing papers in PubMed.

  1. Article
  2. Article
  3. Article
  4. Article
  5. Review
  6. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

13 authors.

Haichao Chen *Tsinghua Medicine, Tsinghua University, Beijing, China.
Zehua Jiang *Tsinghua Medicine, Tsinghua University, Beijing, China.
Xinyu LiuInstitute of Medical Technology, Peking University Health Science Center, Beijing, China.
Can Can XueSingapore Eye Research Institute, Singapore National Eye Centre, Singapore.
Samantha Min Er YewCentre for Innovation and Precision Eye Health, Yong Loo Lin School of Medicine, National University of Singapore, Singapore.
Bin ShengMOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, China.
Ying-Feng ZhengSingapore Eye Research Institute, Singapore National Eye Centre, Singapore.
Xiaofei WangBeijing Advanced Innovation Center for Biomedical Engineering, Key Laboratory for Biomechanics and Mechanobiology of Ministry of Education, School of Biological Science and Medical Engineering, Beihang University, Beijing, China.ORCID http://orcid.org/0000-0002-3175-0344
You WuTsinghua University School of Medicine, Beijing, China.
Sobha SivaprasadMoorfields Eye Hospital City Road Campus, London, UK.
Tien Yin WongTsinghua Medicine, Tsinghua University, Beijing, China.
Varun ChaudharyDepartment of Surgery, Division of Ophthalmology, McMaster University, Hamilton, Ontario, Canada.
Yih Chung ThamSingapore Eye Research Institute, Singapore National Eye Centre, Singapore thamyc@nus.edu.sg.ORCID http://orcid.org/0000-0002-6752-797X

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

BACKGROUND/

aimsLarge language models (LLMs) have substantial potential to enhance the efficiency of academic research. The accuracy and performance of LLMs in a systematic review, a core part of evidence building, has yet to be studied in detail.

methodsWe introduced two LLM-based approaches of systematic review: an LLM-enabled fully automated approach (LLM-FA) utilising three different GPT-4 plugins (Consensus GPT, Scholar GPT and GPT web browsing modes) and an LLM-facilitated semi-automated approach (LLM-SA) using GPT4's Application Programming Interface (API). We benchmarked these approaches using three published systematic reviews that reported the prevalence of diabetic retinopathy across different populations (general population, pregnant women and children).

resultsThe three published reviews consisted of 98 papers in total. Across these three reviews, in the LLM-FA approach, Consensus GPT correctly identified 32.7% (32 out of 98) of papers, while Scholar GPT and GPT4's web browsing modes only identified 19.4% (19 out of 98) and 6.1% (6 out of 98), respectively. On the other hand, the LLM-SA approach not only successfully included 82.7% (81 out of 98) of these papers but also correctly excluded 92.2% of 4497 irrelevant papers.

conclusionsOur findings suggest LLMs are not yet capable of autonomously identifying and selecting relevant papers in systematic reviews. However, they hold promise as an assistive tool to improve the efficiency of the paper selection process in systematic reviews.

Indexed as

Large Language ModelsSystematic Reviews as TopicAutomationFemaleHumansReview Literature as TopicEpidemiologyPublic health

Identifiers

PMID39814458
PMCPMC12320601

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.