ArticleNPJ digital medicine2024
Autonomous medical evaluation for guideline adherence of large language models.
Article in NPJ digital medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 28 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
28 citing papers in PubMed.
- Guideline-augmented prompting improves comparative preference and response consistency of large language model outputs for orthopaedic anaesthesia questions: A controlled prompting study.Journal of experimental orthopaedics · 2026Article
- Improving Reliability and Explainability of Medical Question Answering Through Atomic Fact-Checking in Retrieval-Augmented Large Language Models: Creation and Validation Study.Journal of medical Internet research · 2026Article
- Large language models for tympanostomy patient education: readability and guideline adherence.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Artificial Intelligence and Machine Learning in Rheumatology and Systemic Inflammatory Diseases: From Pattern Recognition to Signal Analysis and Clinical Decision Support.Journal of clinical medicine · 2026Review
- Benchmarking large language models for question answering on German clinical practice guidelines.BMJ health & care informatics · 2026Article
- A Supervised Fine-Tuned Large Language Model for Lifestyle Management in Patients With Prostate Cancer: Development and Evaluation Study.Journal of medical Internet research · 2026Article
- A comprehensive survey of AI agents in healthcare.Journal of biomedical informatics · 2026Review
- Arkangel AI, OpenEvidence, ChatGPT, Medisearch: Are They Objectively up to Medical Standards? A Real-Life Assessment of LLM Chatbots in Health Care.Mayo Clinic proceedings. Digital health · 2026Article
- Evaluating reasoning models for therapy recommendations in gastrointestinal stromal tumors: expert and LLM-based evaluations of OpenAI o1 and DeepSeek-R1.Journal of cancer research and clinical oncology · 2026Article
- Evaluating large language models for evidence-based clinical question answering.Patterns (New York, N.Y.) · 2026Article
- Benchmarking LLM decision support in inflammatory aneurysms: DeepSeek-R1, DeepSeek-V3 and ChatGPT-4o.Clinical and experimental medicine · 2026Article
- Privacy-Preserving Generation of Structured Lymphoma Progression Reports from Cross-sectional Imaging: A Comparative Analysis of Llama 3.3 and Llama 4.Journal of imaging informatics in medicine · 2026Article
- Large Language Models as Clinical Nutrition Decision Tools: Quantitative Bias and Guideline Deviation in Type 2 Diabetes Meal Planning.Healthcare (Basel, Switzerland) · 2026Article
- Accuracy, quality, and readability analyses of responses from large language models to questions on pediatric dental sedation.BMC oral health · 2026Article
- Benchmark evaluation of large language models for clinical decision support in headache management.Journal of oral & facial pain and headache · 2026Article
- A Survey on Medical Competence Evaluation Benchmarks for Large Language Models.Health care science · 2026Review
- Adherence to monitoring iron indices at the initiation of erythropoiesis-stimulating agents or hypoxia-inducible factor prolyl hydroxylase inhibitors.Clinical and experimental nephrology · 2026Article
- Blinded two-phase evaluation of large language models in complex cardiac surgery: task-specific performance and human-AI collaboration.Frontiers in digital health · 2026Article
- A roadmap for medical large language models: a review of foundations, applications, and challenges.Military Medical Research · 2026Review
- Evaluation of large language models in percutaneous coronary intervention decision-making.Frontiers in cardiovascular medicine · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Autonomous Medical Evaluation for Guideline Adherence (AMEGA) is a comprehensive benchmark designed to evaluate large language models' adherence to medical guidelines across 20 diagnostic scenarios spanning 13 specialties. It includes an evaluation framework and methodology to assess models' capabilities in medical reasoning, differential diagnosis, treatment planning, and guideline adherence, using open-ended questions that mirror real-world clinical interactions. It includes 135 questions and 1337 weighted scoring elements designed to assess comprehensive medical knowledge. In tests of 17 LLMs, GPT-4 scored highest with 41.9/50, followed closely by Llama-3 70B and WizardLM-2-8x22B. For comparison, a recent medical graduate scored 25.8/50. The benchmark introduces novel content to avoid the issue of LLMs memorizing existing medical data. AMEGA's publicly available code supports further research in AI-assisted clinical decision-making, aiming to enhance patient care by aiding clinicians in diagnosis and treatment under time constraints.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.