Evidence map›Paper›PMID 41031074›Full record

ArticleArXiv2024

Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine.

Yifan Yang, Qiao Jin, Robert Leaman, Xiaoyu Liu, Guangzhi Xiong, Maame Sarfo-Gyamfi, Changlin Gong, Santiago Ferrière-Steinert, W John Wilbur, Xiaojun Li and 8 more

Abstract readPreprint
PubMed Full text at PMC
In one paragraph

Article in ArXiv, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.

0numbers the graph read from it
0cells of the map it votes in
0citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

0 citing papers in PubMed.

No citing paper in PubMed yet.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

18 authors.

Yifan YangDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.
Qiao JinDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.
Robert LeamanDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.
Xiaoyu LiuUniversity of Maryland at College Park, Department of Computer Science, College Park, MD 20742, USA.
Guangzhi XiongUniversity of Virginia, Department of Computer Science, Charlottesville, VA 22904, USA.
Maame Sarfo-GyamfiDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.
Changlin GongAlbert Einstein College of Medicine, Jacob Medical Center, New York City, NY 10461, USA.
Santiago Ferrière-SteinertUniversidad de Chile, School of Medicine, Santiago, Región Metropolitana, 8380453, Chile.
W John WilburDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.
Xiaojun LiUniversity of Florida, Department of Neurology, Gainesville, FL 32610.
Jiaxin YuanUniversity of Maryland at College Park, Department of Computer Science, College Park, MD 20742, USA.
Bang AnUniversity of Maryland at College Park, Department of Computer Science, College Park, MD 20742, USA.
Kelvin S CastroUniversidad de Chile, School of Medicine, Santiago, Región Metropolitana, 8380453, Chile.
Francisco Erramuspe ÁlvarezDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.
Matías StockleUniversidad de Chile, School of Medicine, Santiago, Región Metropolitana, 8380453, Chile.
Aidong ZhangUniversity of Virginia, Department of Computer Science, Charlottesville, VA 22904, USA.
Furong HuangUniversity of Maryland at College Park, Department of Computer Science, College Park, MD 20742, USA.
Zhiyong LuDivision of Intramural Research, National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD 20894, USA.

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

The remarkable capabilities of Large Language Models (LLMs) make them increasingly compelling for adoption in real-world healthcare applications. However, the risks associated with using LLMs in medical applications have not been systematically characterized. We propose using five key principles for safe and trustworthy medical AI - Truthfulness, Resilience, Fairness, Robustness, and Privacy - along with ten specific aspects. Under this comprehensive framework, we introduce a novel MedGuard benchmark with 1,000 expert-verified questions. Our evaluation of 11 commonly used LLMs shows that the current language models, regardless of their safety alignment mechanisms, generally perform poorly on most of our benchmarks, particularly when compared to the high performance of human physicians. Despite recent reports indicate that advanced LLMs like ChatGPT can match or even exceed human performance in various medical tasks, this study underscores a significant safety gap, highlighting the crucial need for human oversight and the implementation of AI safety guardrails.

Identifiers

PMID41031074
PMCPMC12478422

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.