ArticleNPJ digital medicine2025
Red teaming ChatGPT in medicine to yield real-world insights on model behavior.
Abstract read
In one paragraph
Article in NPJ digital medicine, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 25 papers, 3 of them syntheses that pooled it.
0numbers the graph read from it
0cells of the map it votes in
25citing papers in PubMed, 3 pooled it
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
25 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Mitigating hallucinations in healthcare AI: a systematic review of evidence-based strategies.BMC health services research · 2026Pooled it
- The potential of large language models in the field of infertility: a systematic review.Journal of assisted reproduction and genetics · 2025Pooled it
- Large language models in real-world clinical workflows: a systematic review of applications and implementation.Frontiers in digital health · 2025Pooled it
- Integrating Large Language Models Into Trauma Education for Medical Students: Randomized Controlled Pilot Trial.JMIR medical education · 2026Trial
- Responsible and innovative AI for mental health care: five priority themes.NPP - digital psychiatry and neuroscience · 2026Review
- Review
- An Acceptance Criteria Framework for Determining the Implementation Fit of Custom Large Language Models in Public Health Interventions.Journal of medical Internet research · 2026Article
- Evaluating the safety of large language models in healthcare and dentistry: adversarial testing approaches.BDJ open · 2026Review
- Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review.Journal of medical systems · 2026Article
- Agentic AI and the rise of in silico team science in biomedical research.Nature biotechnology · 2026Review
- Toward trustworthy chatbots: a protocol for red teaming for health related conversations.Scientific reports · 2026Article
- A scalable framework for evaluating health language models.NPJ digital medicine · 2026Article
- Regulation of clinical Artificial Intelligence (AI) in the Age of Agents: Unconfined Non-Deterministic Clinical Software (UNDCS) systems for healthcare.NPJ digital medicine · 2026Article
- A Survey on Medical Competence Evaluation Benchmarks for Large Language Models.Health care science · 2026Review
- Impact of authoritative and subjective cues on large language model reliability for clinical inquiries: an experimental study.Scientific reports · 2026Article
- Responsible use of large language models in gastroenterology and hepatology.Therapeutic advances in gastroenterology · 2026Review
- Do we need prequalification of AI as a medical device to drive equitable adoption.NPJ digital medicine · 2025Review
- Toward Trustworthy Chatbots: A Protocol for Red Teaming for Health Related Conversations.medRxiv : the preprint server for health sciences · 2025Article
- Moving the Fine Print to the Front Page: Transparent Communication of Facial Genetics Research.Advanced genetics (Hoboken, N.J.) · 2025Article
- Vulnerability of Large Language Models to Prompt Injection When Providing Medical Advice.JAMA network open · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
80 authors.
Crystal T Chang *Department of Dermatology, Stanford University, Stanford, USA.ORCID http://orcid.org/0000-0002-0647-7040
Hodan Farah *Department of Dermatology, Stanford University, Stanford, USA.
Haiwen Gui *Department of Dermatology, Stanford University, Stanford, USA.ORCID http://orcid.org/0000-0003-0564-940X
Shawheen Justin RezaeiSchool of Medicine, Stanford University, Stanford, CA, USA.
Charbel Bou-KhalilSchool of Medicine, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-5497-9037
Ye-Jean ParkTemerty Faculty of Medicine, Toronto, ON, Canada.
Akshay SwaminathanSchool of Medicine, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0003-3426-9289
Jesutofunmi A OmiyeDepartment of Dermatology, Stanford University, Stanford, USA.
Akaash KolluriStanford University, Stanford, CA, USA.
Akash ChaurasiaDepartment of Computer Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0003-3438-3313
Alejandro LozanoDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-6952-289X
Alice HeimanStanford University, Stanford, CA, USA.
Allison Sihan JiaStanford University, Stanford, CA, USA.
Amit KaushalDepartment of Bioengineering, Stanford University, Stanford, CA, USA.
Angela JiaStanford University, Stanford, CA, USA.
Angelica IacovelliDepartment of Pediatrics, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0009-0004-1327-2184
Archer YangDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Arghavan SallesStanford University, Stanford, CA, USA.
Arpita SinghalDepartment of Computer Science, Stanford University, Stanford, CA, USA.
Balasubramanian NarasimhanStanford University, Stanford, CA, USA.
Benjamin BelaiDepartment of Psychiatry, Stanford University, Stanford, CA, USA.
Benjamin H JacobsonSchool of Medicine, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-5550-3559
Binglan LiDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Celeste H PoeSchool of Medicine, Stanford University, Stanford, CA, USA.
Chandan SangheraStanford University, Stanford, CA, USA.
Chenming ZhengSchool of Medicine, Stanford University, Stanford, CA, USA.
Conor MesserStanford University, Stanford, CA, USA.
Damien Varid KettudStanford University, Stanford, CA, USA.
Deven PandyaStanford University, Stanford, CA, USA.
Dhamanpreet KaurSchool of Medicine, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-6123-7499
Diana HlaMayo Clinic Alix School of Medicine, Rochester, NY, USA.
Diba DindoustStanford University, Stanford, CA, USA.
Dominik MoehrleSchool of Medicine, Stanford University, Stanford, CA, USA.
Duncan RossDepartment of Statistics, Stanford University, Stanford, CA, USA.
Ellaine ChouDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Eric LinVeterans Affairs Medical Center, Palo Alto, CA, USA.
Fateme Nateghi HaredashtCenter for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.
Ge ChengDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Irena GaoStanford University, Stanford, CA, USA.
Jacob ChangDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Jake SilbergDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Jason A FriesCenter for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-9316-5768
Jiapeng XuDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Joe JamisonDepartment of Statistics, Stanford University, Stanford, CA, USA.
John S TamaresisDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-9868-2366
Jonathan H ChenClinical Excellence Research Center, School of Medicine, Stanford University, Palo Alto, CA, USA.ORCID http://orcid.org/0000-0002-4387-8740
Joshua LazaroDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Juan M BandaTechnology and Digital Solutions, Stanford Health Care, Palo Alto, CA, USA.
Julie J LeeDepartment of Pediatrics, Stanford University, Stanford, CA, USA.
Karen Ebert MatthysDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Kirsten R SteffnerDepartment of Anesthesiology, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-9759-616X
Lu TianStanford University, Stanford, CA, USA.
Luca PegolottiDepartment of Pediatrics, Stanford University, Stanford, CA, USA.
Malathi SrinivasanSchool of Medicine, Stanford University, Stanford, CA, USA.
Maniragav ManimaranGraduate School of Business, Stanford University, Stanford, CA, USA.
Matthew SchwedeDepartment of Medicine, Stanford University, Stanford, CA, USA.
Minghe ZhangDepartment of Statistics, Stanford University, Stanford, CA, USA.
Minh NguyenStanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-7149-849X
Mohsen FathzadehDepartment of Epidemiology and Population Health, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-7962-9148
Qian ZhaoDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Rika BajraSchool of Medicine, Stanford University, Stanford, CA, USA.
Rohit KhuranaDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Ruhana AzamStanford University, Stanford, CA, USA.
Rush BartlettStanford BioDesign, Stanford University, Stanford, CA, USA.
Sang T TruongDepartment of Computer Science, Stanford University, Stanford, CA, USA.
Scott L FlemingDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Shriti RajCenter for Biomedical Informatics Research, Stanford University, Stanford, CA, USA.
Solveig BehrDepartment of Education and Psychology, Freie Universität Berlin, Berlin, Germany.
Sonia OnyekaDepartment of Dermatology, Stanford University, Stanford, USA.
Sri MuppidiStanford University, Stanford, CA, USA.
Tarek BandaliStanford University, Stanford, CA, USA.
Tiffany Y EulalioDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-7084-9646
Wenyuan ChenDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.
Xuanyu ZhouDepartment of Epidemiology and Population Health, Stanford University, Stanford, CA, USA.
Yanan DingDepartment of Biomedical Data Science, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0009-0002-8967-2139
Ying CuiStanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-3697-5155
Yuqi TanDepartment of Pathology, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0002-7200-3222
Yutong LiuDepartment of Epidemiology and Population Health, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0009-0003-6695-8991
Nigam ShahSchool of Medicine, Stanford University, Stanford, CA, USA.ORCID http://orcid.org/0000-0001-9385-7158
Roxana DaneshjouDepartment of Dermatology, Stanford University, Stanford, USA. roxanad@stanford.edu.ORCID http://orcid.org/0000-0001-7988-9356
Funding
No grant is acknowledged in the PubMed record.
Abstract
Red teaming, the practice of adversarially exposing unexpected or undesired model behaviors, is critical towards improving equity and accuracy of large language models, but non-model creator-affiliated red teaming is scant in healthcare. We convened teams of clinicians, medical and engineering students, and technical professionals (80 participants total) to stress-test models with real-world clinical cases and categorize inappropriate responses along axes of safety, privacy, hallucinations/accuracy, and bias. Six medically-trained reviewers re-analyzed prompt-response pairs and added qualitative annotations. Of 376 unique prompts (1504 responses), 20.1% were inappropriate (GPT-3.5: 25.8%; GPT-4.0: 16%; GPT-4.0 with Internet: 17.8%). Subsequently, we show the utility of our benchmark by testing GPT-4o, a model released after our event (20.4% inappropriate). 21.5% of responses appropriate with GPT-3.5 were inappropriate in updated models. We share insights for constructing red teaming prompts, and present our benchmark for iterative model assessments.
Identifiers
What OpenQuestion holds
Textmetadata
LicenceCC BY
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.