ReviewRadiology2025
Pitfalls and Best Practices in Evaluation of AI Algorithmic Biases in Radiology.
Abstract readReview
In one paragraph
Review in Radiology, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 12 papers.
0numbers the graph read from it
0cells of the map it votes in
12citing papers in PubMed
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
12 citing papers in PubMed.
- Assessing and mitigating demographic bias in large language models for diagnostic radiology.Japanese journal of radiology · 2026Article
- Article
- Uncertainty quantification for artificial intelligence in medical imaging: what every radiologist needs to know.Abdominal radiology (New York) · 2026Review
- Language-Guided Segmentation of Medical Images: A Review of Foundation Models.Bioengineering (Basel, Switzerland) · 2026Review
- AI-Driven Design Platforms of Next-Generation Antibody Therapeutics.Topics in current chemistry (Cham) · 2026Review
- Multimodal Deep Learning Approaches for Lung Disease Detection: A Review.Medicina (Kaunas, Lithuania) · 2026Review
- Personalised Dosimetry in Nuclear Medicine: Bridging Physics, Biology and AI for Next Generation Radiopharmaceutical Therapy.Nuclear medicine and molecular imaging · 2026Review
- Radiomics and artificial intelligence in pancreatic cyst characterization: future or fiction?Abdominal radiology (New York) · 2026Review
- Equity and Generalizability of Radiomics in Orbital Disease: Challenges for Ophthalmology, Otolaryngology, and Plastic Surgery.Diagnostics (Basel, Switzerland) · 2026Review
- Artificial Intelligence in Radiology: Hidden Fragilities and the Path to Resilience.Saudi medical journal · 2026Review
- Anthropometry, sex, and age at diagnosis affect pulmonary blood volume quantification from computed tomography pulmonary angiography in pulmonary hypertension assessment.Frontiers in cardiovascular medicine · 2026Article
- The ethics of data mining in healthcare: challenges, frameworks, and future directions.BioData mining · 2025Review
Corrections and comments
- Commented on by
Authors and funding
11 authors.
Paul H YiFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-9433-8093
Preetham BachinaFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-9833-4487
Beepul BhartiFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0002-3204-5364
Sean P GarinFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0002-0445-6852
Adway KanhereFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-5295-2634
Pranav KulkarniFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-5291-2202
David LiFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-8422-421X
Vishwa S ParekhFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0009-0002-2299-597X
Samantha M SantomartinoFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-5229-2576
Linda MoyFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0001-9564-9360
Jeremias SulamFrom the Department of Radiology, St Jude Children's Research Hospital, 262 Danny Thomas Pl, Memphis, TN 38105-3678 (P.H.Y.); Johns Hopkins University School of Medicine, Baltimore, Md (P.B.); Department of Biomedical Engineering, Johns Hopkins University, Baltimore, Md (B.B., J.S.); Uniformed Services University of the Health Sciences, Bethesda, Md (S.P.G.); Institute for Health Computing, University of Maryland School of Medicine, Baltimore, Md (A.K., P.K.); Department of Medical Imaging, Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada (D.L.); Department of Diagnostic and Interventional Imaging, McGovern Medical School at The University of Texas Health Science Center at Houston (UTHealth Houston), Houston, Tex (V.S.P.); Drexel University School of Medicine, Philadelphia, Pa (S.M.S.); and Department of Radiology, New York University Grossman School of Medicine, New York, NY (L.M.).ORCID 0000-0003-0946-1957
Funding
SCH: Quantifying and mitigating demographic biases of machine learning in real world radiologyR01CA287422 · NCI · JOHNS HOPKINS UNIVERSITY · PI SULAM, JEREMIAS · 2023 to 2025
$889kNCI NIH HHS R01 CA287422
Abstract
Despite growing awareness of problems with fairness in artificial intelligence (AI) models in radiology, evaluation of algorithmic biases, or AI biases, remains challenging due to various complexities. These include incomplete reporting of demographic information in medical imaging datasets, variability in definitions of demographic categories, and inconsistent statistical definitions of bias. To guide the appropriate evaluation of AI biases in radiology, this article summarizes the pitfalls in the evaluation and measurement of algorithmic biases. These pitfalls span the spectrum from the technical (eg, how different statistical definitions of bias impact conclusions about whether an AI model is biased) to those associated with social context (eg, how different conventions of race and ethnicity impact identification or masking of biases). Actionable best practices and future directions to avoid these pitfalls are summarized across three key areas:
Indexed as
AlgorithmsArtificial IntelligenceDiagnostic ImagingRadiologyBiasHumans
Identifiers
What OpenQuestion holds
Textmetadata
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.