ArticleNPJ digital medicine2024
Evaluation and mitigation of cognitive biases in medical language models.
Article in NPJ digital medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 40 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
40 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks.Journal of medical Internet research · 2025Pooled it
- Assessing and mitigating demographic bias in large language models for diagnostic radiology.Japanese journal of radiology · 2026Article
- Large Language Models for Distress Rating in Korean Psycho-Oncology Interviews: Exploratory Clinician-Benchmarked Evaluation Study.Journal of medical Internet research · 2026Article
- Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers.British journal of clinical pharmacology · 2026Article
- Article
- Patient Cognitive Bias in Large Language Model-Supported Health Consultations: Simulation-Based Comparative Study.Journal of medical Internet research · 2026Article
- Susceptibility of Large Language Models to User-Driven Factors in Medical Queries.Journal of healthcare informatics research · 2026Article
- Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review.Journal of medical systems · 2026Article
- Comparing the performance of four mainstream large language models on medical literature review generation: a human expert evaluation in SMILE surgery.Graefe's archive for clinical and experimental ophthalmology = Albrecht von Graefes Archiv fur klinische und experimentelle Ophthalmologie · 2026Article
- AgentClinic: a multimodal benchmark for tool-using clinical AI agents.NPJ digital medicine · 2026Article
- Multidisciplinary perspectives on artificial intelligence in aging research and education: evolving uses, ethics, and equity considerations in gerontology.The Gerontologist · 2026Article
- Article
- LLMs in Patient-Centric Program Evaluation: Addressing Concerns and Further Considerations.Pain reports · 2026Article
- Generative AI in simulation debriefings: an exploratory study using the Team-FIRST framework and qualitative feedback from simulation experts and learners.Advances in simulation (London, England) · 2026Article
- Assessing the Quality of AI Responses to Patient Concerns About Axial Spondyloarthritis: Delphi-Based Evaluation.JMIR AI · 2026Article
- Implicit bias in safety-aligned large language models: A multi-faceted evaluation of clinical decision-making and health equity.PloS one · 2026Article
- Educational support and weekly telemonitoring during rapid maxillary expansion in children: a prospective preference-based comparative cohort study.Frontiers in dental medicine · 2026Article
- Cross-linguistic patterns of cognitive biases in large language models: a comparative study in English, Hebrew, and Russian.Frontiers in artificial intelligence · 2026Article
- Same child, different risk: demographic bias in childhood obesity attribution by large language models.Frontiers in public health · 2026Article
- Cross-modal bias in medical vision-language models: a pipeline-aware framework for mechanisms, evaluation, and mitigation.Frontiers in digital health · 2026Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
Abstract
Increasing interest in applying large language models (LLMs) to medicine is due in part to their impressive performance on medical exam questions. However, these exams do not capture the complexity of real patient-doctor interactions because of factors like patient compliance, experience, and cognitive bias. We hypothesized that LLMs would produce less accurate responses when faced with clinically biased questions as compared to unbiased ones. To test this, we developed the BiasMedQA dataset, which consists of 1273 USMLE questions modified to replicate common clinically relevant cognitive biases. We assessed six LLMs on BiasMedQA and found that GPT-4 stood out for its resilience to bias, in contrast to Llama 2 70B-chat and PMC Llama 13B, which showed large drops in performance. Additionally, we introduced three bias mitigation strategies, which improved but did not fully restore accuracy. Our findings highlight the need to improve LLMs' robustness to cognitive biases, in order to achieve more reliable applications of LLMs in healthcare.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.