ArticleArchives of gynecology and obstetrics2024
Evolution of publicly available large language models for complex decision-making in breast cancer care.
Article in Archives of gynecology and obstetrics, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT07199231 (A Comparative Performance Evaluation of Four Publicly Available Large Language Models Against Gold Standard Medical References), which is not on this map. Cited by 22 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
A Comparative Performance Evaluation of Four Publicly Available Large Language Models Against Gold Standard Medical References
Who cites it
22 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Applications of large language models in tumor boards: a systematic review.Oncology reviews · 2026Pooled it
- Leveraging artificial intelligence to support surgical oncology multidisciplinary team decision-making: a systematic review.Frontiers in artificial intelligence · 2026Pooled it
- Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study.Journal of medical Internet research · 2026Article
- AI-enabled clinical decision support in breast cancer care: a blinded multicenter benchmarking study comparing medically specialized with a general-purpose system.Journal of medical systems · 2026Article
- Feasibility and Concordance of a Large Language Model (ChatGPT-5) as a Clinical Decision Support Tool in Gynecologic Oncology Tumor Boards: A Blinded, Multi-Observer Study.Journal of clinical medicine · 2026Article
- Therapeutic Decisions in Inflammatory Bowel Disease: ChatGPT-4.0 Compared to Medical Expertise.JGH open : an open access journal of gastroenterology and hepatology · 2026Article
- Analysis of Large Language Model Decision Making in Hormone Receptor-Positive/Human Epidermal Growth Factor Receptor 2-Negative Early Breast Cancer.JCO clinical cancer informatics · 2026Article
- Article
- Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation.Rheumatology international · 2026Article
- Artificial Intelligence agents for biological research: a survey.Briefings in bioinformatics · 2026Article
- Large language models for breast cancer treatment planning: a blinded real-world evaluation of DeepSeek, ChatGPT, and oncologist recommendations.Frontiers in digital health · 2026Article
- Artificial intelligence in multidisciplinary tumor boards enhancing decision making and clinical outcomes in oncology.iScience · 2025Review
- Temporal evolution of large language models (LLMs) in oncology.Journal of translational medicine · 2025Article
- Exploring the potential of AI-powered applications for clinical decision-making in gynecologic oncology.International journal of gynaecology and obstetrics: the official organ of the International Federation of Gynaecology and Obstetrics · 2025Article
- AI-driven simplification of surgical reports in gynecologic oncology: A potential tool for patient education.Acta obstetricia et gynecologica Scandinavica · 2025Article
- Fine-Tuning Large Language Models for Specialized Use Cases.Mayo Clinic proceedings. Digital health · 2025Review
- The Role of Artificial Intelligence in Gynecologic Oncology Decision-Making: A Feasibility Study.Gynecologic and obstetric investigation · 2025Article
- Article
- Can Chat-GPT read and understand guidelines? An example using the S2k guideline intrauterine growth restriction of the German Society for Gynecology and Obstetrics.Archives of gynecology and obstetrics · 2024Article
- How do large language models answer breast cancer quiz questions? A comparative study of GPT-3.5, GPT-4 and Google Gemini.La Radiologia medica · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeThis study investigated the concordance of five different publicly available Large Language Models (LLM) with the recommendations of a multidisciplinary tumor board regarding treatment recommendations for complex breast cancer patient profiles.
methodsFive LLM, including three versions of ChatGPT (version 4 and 3.5, with data access until September 3021 and January 2022), Llama2, and Bard were prompted to produce treatment recommendations for 20 complex breast cancer patient profiles. LLM recommendations were compared to the recommendations of a multidisciplinary tumor board (gold standard), including surgical, endocrine and systemic treatment, radiotherapy, and genetic testing therapy options.
resultsGPT4 demonstrated the highest concordance (70.6%) for invasive breast cancer patient profiles, followed by GPT3.5 September 2021 (58.8%), GPT3.5 January 2022 (41.2%), Llama2 (35.3%) and Bard (23.5%). Including precancerous lesions of ductal carcinoma in situ, the identical ranking was reached with lower overall concordance for each LLM (GPT4 60.0%, GPT3.5 September 2021 50.0%, GPT3.5 January 2022 35.0%, Llama2 30.0%, Bard 20.0%). GPT4 achieved full concordance (100%) for radiotherapy. Lowest alignment was reached in recommending genetic testing, demonstrating a varying concordance (55.0% for GPT3.5 January 2022, Llama2 and Bard up to 85.0% for GPT4).
conclusionThis early feasibility study is the first to compare different LLM in breast cancer care with regard to changes in accuracy over time, i.e., with access to more data or through technological upgrades. Methodological advancement, i.e., the optimization of prompting techniques, and technological development, i.e., enabling data input control and secure data processing, are necessary in the preparation of large-scale and multicenter studies to provide evidence on their safe and reliable clinical application. At present, safe and evidenced use of LLM in clinical breast cancer care is not yet feasible.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.