ArticleCritical care (London, England)2025
Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study.
Article in Critical care (London, England), 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 18 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
18 citing papers in PubMed.
- Fully flexing the potential of AI in intensive care: reinforcement learning with randomization when needed.Intensive care medicine · 2026Article
- [Application effect of LLMs as decision-support tools in emergency burn treatment].Zhonghua shao shang yu chuang mian xiu fu za zhi · 2026Article
- Hallucination Rate of Peer-Reviewed Citations Generated by Large Language Models in Neurocritical Care.Critical care explorations · 2026Article
- Assessment of AI-Generated Patient Education Materials for Bladder Training and Pelvic Floor Muscle Therapy: Comparison with an International Society Leaflet.International urogynecology journal · 2026Article
- Automatic extraction of structured information from brain MRI reports using an open-weight large language model.European radiology · 2026Article
- Automated Review of Patient Records: Privacy-Preserving Large Language Models for Identifying Incident Nonarteritic Anterior Ischemic Optic Neuropathy at Scale.Ophthalmology science · 2026Article
- Article
- Fine-tuning LLM hyperparameters to align semantic and physiological contexts of aging-related pathways.Molecular diversity · 2026Article
- A Survey on Medical Competence Evaluation Benchmarks for Large Language Models.Health care science · 2026Review
- Performance of successive generative pretrained transformers (GPT) models in medical cases and board style questions.Scientific reports · 2026Article
- Performance of DeepSeek and ChatGPT on the Chinese Health Professional and Technical Examination: A comparative study.PloS one · 2026Article
- Comparing large language models and search engine responses to common orthodontic questions.PloS one · 2026Article
- Large language models in emergency and critical care medicine: a comprehensive review of applications, challenges, and future directions.Burns & trauma · 2026Review
- Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo.Frontiers in public health · 2026Article
- Large Language Models in Critical Care Medicine: Scoping Review.JMIR medical informatics · 2025Article
- ChatMyopia: An AI agent for myopia-related consultation in primary eye care settings.iScience · 2025Article
- AI assisted prediction of unplanned intensive care admissions using natural language processing in elective neurosurgery.NPJ digital medicine · 2025Article
- A large language model improves clinicians' diagnostic performance in complex critical illness cases.Critical care (London, England) · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge language models (LLMs) show increasing potential for their use in healthcare for administrative support and clinical decision making. However, reports on their performance in critical care medicine is lacking.
methodsThis study evaluated five LLMs (GPT-4o, GPT-4o-mini, GPT-3.5-turbo, Mistral Large 2407 and Llama 3.1 70B) on 1181 multiple choice questions (MCQs) from the gotheextramile.com database, a comprehensive database of critical care questions at European Diploma in Intensive Care examination level. Their performance was compared to random guessing and 350 human physicians on a 77-MCQ practice test. Metrics included accuracy, consistency, and domain-specific performance. Costs, as a proxy for energy consumption, were also analyzed.
resultsGPT-4o achieved the highest accuracy at 93.3%, followed by Llama 3.1 70B (87.5%), Mistral Large 2407 (87.9%), GPT-4o-mini (83.0%), and GPT-3.5-turbo (72.7%). Random guessing yielded 41.5% (p < 0.001). On the practice test, all models surpassed human physicians, scoring 89.0%, 80.9%, 84.4%, 80.3%, and 66.5%, respectively, compared to 42.7% for random guessing (p < 0.001) and 61.9% for the human physicians. However, in contrast to the other evaluated LLMs (p < 0.001), GPT-3.5-turbo's performance did not significantly outperform physicians (p = 0.196). Despite high overall consistency, all models gave consistently incorrect answers. The most expensive model was GPT-4o, costing over 25 times more than the least expensive model, GPT-4o-mini.
conclusionsLLMs exhibit exceptional accuracy and consistency, with four outperforming human physicians on a European-level practice exam. GPT-4o led in performance but raised concerns about energy consumption. Despite their potential in critical care, all models produced consistently incorrect answers, highlighting the need for more thorough and ongoing evaluations to guide responsible implementation in clinical settings.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.