Factors underpinning the performance of implemented artificial intelligence-based patient deterioration prediction systems: reasons for selection and implications for hospitals and researchers.

objectiveThe degree to which deployed artificial intelligence-based deterioration prediction algorithms (AI-DPA) differ in their development, the reasons for these differences, and how this may impact their performance remains unclear. Our primary objective was to identify design factors and associated decisions related to the development of AI-DPA and highlight deficits that require further research. MATERIALS AND

methodsBased on a systematic review of 14 deployed AI-DPA and an updated systematic search, we identified studies of 12 eligible AI-DPA from which data were extracted independently by 2 investigators on all design factors, decisions, and justifications pertaining to 6 machine learning development stages: (1) model requirements, (2) data collection, (3) data cleaning, (4) data labeling, (5) feature engineering, and (6) model training.

resultsWe found 13 design factors and 315 decision alternatives likely to impact AI-DPA performance, all of which varied, together with their rationales, between all included AI-DPA. Variable selection, data imputation methods, training data exclusions, training sample definitions, length of lookback periods, and definition of outcome labels were key design factors accounting for most variation. In justifying decisions, most studies made no reference to prior research or compared with other state-of-the-art algorithms. DISCUSSION: Algorithm design decisions regarding factors impacting AI-DPA performance have little supporting evidence, are inconsistent, do not learn from prior work, and lack reference standards.

conclusionSeveral deficits in AI-DPA development that prevent implementers selecting the most accurate algorithm have been identified, and future research needs to address these deficits as a priority.