Trial reportNature medicine2026
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.
Trial report in Nature medicine, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. An erratum has been issued. Cited by 34 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
34 citing papers in PubMed.
- Preliminary Evaluation of a Large Language Model-Powered Chatbot for Osteoporosis Self-Management Education: Formative Randomized Controlled Trial.JMIR formative research · 2026Trial
- Misaligned by design: evaluating and deploying generative AI for the real-world conditions of primary care: a Nordic and European perspective.Scandinavian journal of primary health care · 2026Article
- Telehealth and access to acute primary care in Europe: Balancing opportunity and equity.The European journal of general practice · 2026Article
- Accountability for large language models in health care.Bulletin of the World Health Organization · 2026Article
- Sex and gender bias in large language models: an old problem at a new scale.BMJ health & care informatics · 2026Article
- What Are Physicians For? How Psychiatry's Competencies Endure Into the Age of AI.Academic psychiatry : the journal of the American Association of Directors of Psychiatric Residency Training and the Association for Academic Psychiatry · 2026Article
- Generative large language models in medicine: a scoping review of recent methodological advances.npj health systems · 2026Review
- Burn Extent and Fitzpatrick Skin Tone Assessment from Clinical Photographs: Systematic and Random Error in Multimodal Large Language Models.Bioengineering (Basel, Switzerland) · 2026Article
- Responsible and innovative AI for mental health care: five priority themes.NPP - digital psychiatry and neuroscience · 2026Review
- The Reliability of Human Evaluation of Large Language Models in Health Care Settings: Scoping Review.Journal of medical Internet research · 2026Article
- Artificial Intelligence as a Personal Coach: A Narrative Review of Benefits and Risks in Educational and Health Contexts.Behavioral sciences (Basel, Switzerland) · 2026Review
- Equitable digital health systems begin before technology is deployed.BMC global and public health · 2026Article
- The Influence of Patient Persona and Affective Framing on Management Recommendations of ChatGPT, Gemini and Claude for Unruptured Intracranial Aneurysms: A Comparative Benchmarking Study.Neurosurgical review · 2026Article
- Comparison of artificial intelligence-based chatbots and expert periodontists in responding to patient questions: a multi-dimensional analysis.BMC oral health · 2026Article
- A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded multi-stakeholder assessment and guideline comparison.Rheumatology international · 2026Observational
- Digital Mental Health Research Priorities, Revisited for the AI and Large Language Model Era.JMIR mental health · 2026Article
- [Tele-emergency physicians-a cost-benefit analysis].Medizinische Klinik, Intensivmedizin und Notfallmedizin · 2026Review
- How to benchmark medical AI agents.PLoS medicine · 2026Article
- Proposed Context-of-Use Evaluation Framework for Medication Management Tasks Completed by Generative Artificial Intelligence.medRxiv : the preprint server for health sciences · 2026Article
- AI and Its Shifting Roles in the Therapeutic Relationship: Implications for Precision Medicine.Journal of personalized medicine · 2026Review
Corrections and comments
- Erratum issued
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Global healthcare providers are exploring the use of large language models (LLMs) to provide medical advice to the public. LLMs now achieve nearly perfect scores on medical licensing exams, but this does not necessarily translate to accurate performance in real-world settings. We tested whether LLMs can assist members of the public in identifying underlying conditions and choosing a course of action (disposition) in ten medical scenarios in a controlled study with 1,298 participants. Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+) or a source of their choice (control). Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9% of cases and disposition in 56.3% on average. However, participants using the same LLMs identified relevant conditions in fewer than 34.5% of cases and disposition in fewer than 44.2%, both no better than the control group. We identify user interactions as a challenge to the deployment of LLMs for medical advice. Standard benchmarks for medical knowledge and simulated patient interactions do not predict the failures we find with human participants. Moving forward, we recommend systematic human user testing to evaluate interactive capabilities before public deployments in healthcare.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.