A validation of machine learning models for the identification of critically ill children presenting to the paediatric emergency room of a tertiary hospital in South Africa: A proof of concept.

Background: Machine learning (ML) refers to computational algorithms designed to learn from patterns in data to provide insights or predictions related to that data. Objectives: Multiple studies report the development of predictive models for triage or identification of critically ill children. In this study, we validate machine learning models developed in South Africa for the identification of critically ill children presenting to a tertiary hospital. Results: The validation sample comprised 267 patients. The event rate for the study outcome was 0.12. All models demonstrated good discrimination but weak calibration. Artificial neural network 1 (ANN1) had the highest area under the receiver operating characteristic curve (AUROC) with a value of 0.84. ANN2 had the highest area under the precision-recall curve (AUPRC) with a value of 0.65. Decision curve analysis demonstrated that all models were superior to standard strategies of treating all patients or treating no patients at a proposed threshold probability of 10%. Confidence intervals for model performance overlapped considerably. Post hoc model explanations demonstrated that models were logically coherent with clinical knowledge. Conclusions: Internal validation of the predictive models correlated with model performance in the development study. The models were able to discriminate between critically ill children and non-critically ill children; however, the superiority of one model over the others could not be demonstrated in this study. Therefore, models such as these still require further refinement and external validation before implementation in clinical practice. Indeed, successful implementation of machine learning in practice within the South African setting will require the development of regulatory and infrastructural frameworks in conjunction with the adoption of alternative approaches to electronic data capture, such as the use of mobile devices.