ArticleBJOG : an international journal of obstetrics and gynaecology2026
Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study.
Article in BJOG : an international journal of obstetrics and gynaecology, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 2 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
2 citing papers in PubMed.
- Cultural bias in large language models' ability to follow neuroradiology guidelines.European radiology · 2026Article
- Safety-Oriented Benchmarking of Large Language Models in Risk-Based Management of Abnormal Cervical Screening Results: Scenario-Based Benchmark Study.Journal of medical Internet research · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
15 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveAlthough large language models are increasingly used in clinical and research settings, the validity of the information they provide remains uncertain. This study aimed to evaluate the accuracy, consistency, and reliability of three large language models-ChatGPT 4.0, DeepSeek R1, and Gemini 2.0-in answering cervical cancer-related questions based on the ESGO/ESTRO/ESP guidelines.
designProspective, comparative in silico benchmarking study.
settingFondazione Policlinico Universitario A. Gemelli, Rome, Italy. POPULATION OR SAMPLE: Fifty questions derived from the ESGO/ESTRO/ESP (European Society of Gynaecologic Oncology/European Society for Radiotherapy and Oncology/European Society of Pathology) Guidelines for Cervical Cancer.
methodsEach question was submitted simultaneously to ChatGPT 4.0, DeepSeek R1, and Gemini 2.0, and re-entered twice to assess response repeatability. Answers were evaluated for accuracy using a Global Quality Score (GQS) from 1 (poor) to 5 (completely accurate). Consistency (intra-model response stability) and reliability (alignment with guidelines) were assessed using binary classification.
main outcome measuresMedian GQS, percentage of GQS 5 responses, consistency between repeated answers, and reliability.
resultsChatGPT 4.0 achieved the highest performance, with 42% of responses rated GQS 5, followed by Gemini 2.0 (30%) and DeepSeek R1 (28%). DeepSeek R1 and Gemini 2.0 scored lower in median GQS (3.50) compared to ChatGPT 4.0 (4.00). Response consistency varied significantly, with ChatGPT 4.0 and DeepSeek R1 showing differences from Gemini 2.0 (p = 0.034 and p = 0.044, respectively). No significant difference was observed in reliability (p = 0.602).
conclusionAll models demonstrated suboptimal accuracy in aligning with clinical guidelines. ChatGPT 4.0 was the most accurate and consistent whereas DeepSeek R1 underperformed. Despite similar reliability across models, expert oversight remains essential to ensure safe clinical application and prevent misinformation.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.