Gender and racial bias unveiled: clinical artificial intelligence (AI) and machine learning (ML) algorithms are fanning the flames of inequity.

This study aims to advocate for the continued evaluation of published clinical artificial intelligence (AI) and Machine Learning (ML) studies, including the reporting of demographic information, specifically gender, racial composition, and geographic location. Models are often trained on data lacking representation across basic demographics, potentially leading to biased outputs and exacerbating health disparities. Previous research in drug and device development has demonstrated the dangers of underrepresenting women and minority populations. This study aimed to assess the extent to which published clinical AI/ML studies report demographic information, specifically gender and racial composition, in their training datasets. A systematic review was conducted in accordance with PRISMA guidelines. The databases that were used in the study include Ovid MEDLINE, Embase, PsycINFO, Scopus, Web of Science and the Cochrane Library, for clinical AI/ML studies with direct implications for patient care. Inclusion criteria required models to be clinically actionable and not pre-clinical or administrative in scope. Two independent reviewers screened and extracted data using Covidence software, with conflicts resolved by a third reviewer. Out of 390 studies included, 84% of global models did not report the racial composition of their training data, while 31% lacked gender data. US-based models performed slightly better, with 56% reporting race and 77% reporting gender. Only 16% of all models utilized publicly available, non-proprietary datasets. The low frequency of demographic disclosure and limited use of open data raise serious concerns about the transparency, generalizability and fairness of clinical AI/ML models. Standardized reporting of gender and racial composition in training data is urgently needed to ensure ethical and equitable deployment of these technologies.