ArticleThe oncologist2024
Comparison of Large Language Models in Answering Immuno-Oncology Questions: A Cross-Sectional Study.
Article in The oncologist, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 28 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
28 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Comparison of the readability of ChatGPT and Bard in medical communication: a meta-analysis.BMC medical informatics and decision making · 2025Pooled it
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Artificial Intelligence (AI) and Healthcare Capabilities: A Systematic Review and Research Directions.F1000Research · 2025Pooled it
- Comparative Evaluation of AI Chatbots for Testicular Cancer Education: Validity, Information Quality, and Readability.Annals of surgical oncology · 2026Article
- AI-driven discovery in protein science for immunology and infectious disease research.Frontiers in bioinformatics · 2026Review
- Artificial Intelligence-Assisted Error Detection in Complex Clinical Documentation: Leveraging Large Language Models to Enhance Patient Safety in Oncology.JCO clinical cancer informatics · 2026Article
- Causal AI for cancer immunotherapy: a narrative framework review of target trial emulation, treatment-effect learning and clinical translation.Frontiers in immunology · 2026Review
- Performance of large language model in cross-specialty medical scenarios.Journal of translational medicine · 2025Article
- Evaluating Artificial Intelligence Models in Dermatology: Comparative Analysis.JMIR dermatology · 2025Article
- ChatGPT's role in the rapidly evolving hematologic cancer landscape.Future science OA · 2025Article
- Comparison of the accuracy and reliability of ChatGPT-4o and Gemini in answering HIV-related questions.BMC infectious diseases · 2025Article
- Assessing ChatGPT's Educational Potential in Lung Cancer Radiotherapy From Clinician and Patient Perspectives: Content Quality and Readability Analysis.JMIR cancer · 2025Article
- Servicing open-source large language models for oncology.The oncologist · 2025Article
- Large language model integrations in cancer decision-making: a systematic review and meta-analysis.NPJ digital medicine · 2025Article
- Predicting Immunotherapy Response in Unresectable Hepatocellular Carcinoma: A Comparative Study of Large Language Models and Human Experts.Journal of medical systems · 2025Article
- Medical accuracy of artificial intelligence chatbots in oncology: a scoping review.The oncologist · 2025Article
- Evaluating the Accuracy of Responses by Large Language Models for Information on Disease Epidemiology.Pharmacoepidemiology and drug safety · 2025Article
- Performance of Large Language Models on the Korean Dental Licensing Examination: A Comparative Study.International dental journal · 2025Article
- Comparison of the performances between ChatGPT and Gemini in answering questions on viral hepatitis.Scientific reports · 2025Article
- Article
Corrections and comments
- Commented on by
- Update of
Authors and funding
6 authors.
Funding
Abstract
backgroundThe capability of large language models (LLMs) to understand and generate human-readable text has prompted the investigation of their potential as educational and management tools for patients with cancer and healthcare providers. MATERIALS AND
methodsWe conducted a cross-sectional study aimed at evaluating the ability of ChatGPT-4, ChatGPT-3.5, and Google Bard to answer questions related to 4 domains of immuno-oncology (Mechanisms, Indications, Toxicities, and Prognosis). We generated 60 open-ended questions (15 for each section). Questions were manually submitted to LLMs, and responses were collected on June 30, 2023. Two reviewers evaluated the answers independently.
resultsChatGPT-4 and ChatGPT-3.5 answered all questions, whereas Google Bard answered only 53.3% (P < .0001). The number of questions with reproducible answers was higher for ChatGPT-4 (95%) and ChatGPT3.5 (88.3%) than for Google Bard (50%) (P < .0001). In terms of accuracy, the number of answers deemed fully correct were 75.4%, 58.5%, and 43.8% for ChatGPT-4, ChatGPT-3.5, and Google Bard, respectively (P = .03). Furthermore, the number of responses deemed highly relevant was 71.9%, 77.4%, and 43.8% for ChatGPT-4, ChatGPT-3.5, and Google Bard, respectively (P = .04). Regarding readability, the number of highly readable was higher for ChatGPT-4 and ChatGPT-3.5 (98.1%) and (100%) compared to Google Bard (87.5%) (P = .02).
conclusionChatGPT-4 and ChatGPT-3.5 are potentially powerful tools in immuno-oncology, whereas Google Bard demonstrated relatively poorer performance. However, the risk of inaccuracy or incompleteness in the responses was evident in all 3 LLMs, highlighting the importance of expert-driven verification of the outputs returned by these technologies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.