ArticleNPJ digital medicine2025
When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior.
Article in NPJ digital medicine, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 41 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
41 citing papers in PubMed.
- From Innovation to Impact: The CREATE Framework as a Blueprint for Large Language Model Adoption in Opioid Treatment Programs.Journal of medical Internet research · 2026Article
- MedNLI-Pain benchmarks a minimally necessary requirement for large language model alignment.JAMIA open · 2026Article
- AI-Mediated Assessment of Continuing Medical Education: The Case-based Learning Intelligence Credit System (CLICS) Framework.JMIR medical education · 2026Article
- Validating LLM judges for automated oversight of patient communication.medRxiv : the preprint server for health sciences · 2026Article
- The Utility of Large Language Models to Assist With Emergency Triage Decisions Within Otolaryngology.Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2026Article
- New article types atBMJ global health · 2026Article
- Artificial Intelligence as a Personal Coach: A Narrative Review of Benefits and Risks in Educational and Health Contexts.Behavioral sciences (Basel, Switzerland) · 2026Review
- The Influence of Patient Persona and Affective Framing on Management Recommendations of ChatGPT, Gemini and Claude for Unruptured Intracranial Aneurysms: A Comparative Benchmarking Study.Neurosurgical review · 2026Article
- The Diagnostic Mirage: Path Forward for Artificial Intelligence in Clinical Diagnosis.Journal of evaluation in clinical practice · 2026Article
- Review
- When Artificial Intelligence Agrees Too Easily: The Emerging Risk of Artificial Intelligence Sycophancy in Dentistry.International dental journal · 2026Article
- Article
- FaithfulnessFindings of ACL. ACL · 2026Article
- Artificial Intelligence in Child and Adolescent Psychiatry: A Narrative Review of Recent Clinical Applications and Ethical Considerations.Current psychiatry reports · 2026Review
- Characterizing the spiral: potential mechanisms in AI-associated delusions.NPP - digital psychiatry and neuroscience · 2026Review
- Who bears liability when AI gives bad prescribing advice.NPJ digital medicine · 2026Article
- Substance-induced manic psychosis in which delusions were corroborated by a chatbot - case report.BMC psychiatry · 2026Article
- Large Language Model Hallucinations in Spine Surgery: A Comparative Analysis of Clinician vs Patient-Level Prompts.Neurosurgery practice · 2026Article
- How to meaningfully evaluate AI in clinical medicine.Nature medicine · 2026Article
- Evaluation of Large Language Model Chatbot Responses to Psychotic Prompts.JAMA psychiatry · 2026Article
Corrections and comments
- Update of
Authors and funding
9 authors.
Funding
Abstract
Large language models (LLMs) exhibit a vulnerability arising from being trained to be helpful: a tendency to comply with illogical requests that would generate false information, even when they have the knowledge to identify the request as illogical. This study investigated this vulnerability in the medical domain, evaluating five frontier LLMs using prompts that misrepresent equivalent drug relationships. We tested baseline sycophancy, the impact of prompts allowing rejection and emphasizing factual recall, and the effects of fine-tuning on a dataset of illogical requests, including out-of-distribution generalization. Results showed high initial compliance (up to 100%) across all models, prioritizing helpfulness over logical consistency. Prompt engineering and fine-tuning improved performance, improving rejection rates on illogical requests while maintaining general benchmark performance. This demonstrates that prioritizing logical consistency through targeted training and prompting is crucial for mitigating the risk of generating false medical information and ensuring the safe deployment of LLMs in healthcare.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.