What Is the Accuracy of Three Different Machine Learning Techniques to Predict Clinical Outcomes After Shoulder Arthroplasty?

backgroundMachine learning techniques can identify complex relationships in large healthcare datasets and build prediction models that better inform physicians in ways that can assist in patient treatment decision-making. In the domain of shoulder arthroplasty, machine learning appears to have the potential to anticipate patients' results after surgery, but this has not been well explored. QUESTIONS/PURPOSES: (1) What is the accuracy of machine learning to predict the American Shoulder and Elbow Surgery (ASES), University of California Los Angeles (UCLA), Constant, global shoulder function, and VAS pain scores, as well as active abduction, forward flexion, and external rotation at 1 year, 2 to 3 years, 3 to 5 years, and more than 5 years after anatomic total shoulder arthroplasty (aTSA) or reverse total shoulder arthroplasty (rTSA)? (2) What is the accuracy of machine learning to identify whether a patient will achieve clinical improvement that exceeds the minimum clinically important difference (MCID) threshold for each outcome measure? (3) What is the accuracy of machine learning to identify whether a patient will achieve clinical improvement that exceeds the substantial clinical benefit threshold for each outcome measure?

methodsA machine learning analysis was conducted on a database of 7811 patients undergoing shoulder arthroplasty of one prosthesis design to create predictive models for multiple clinical outcome measures. Excluding patients with revisions, fracture indications, and hemiarthroplasty resulted in 6210 eligible primary aTSA and rTSA patients, of whom 4782 patients with 11,198 postoperative follow-up visits had sufficient preoperative, intraoperative, and postoperative data to train and test the predictive models. Preoperative clinical data from 1895 primary aTSA patients and 2887 primary rTSA patients were analyzed using three commercially available supervised machine learning techniques: linear regression, XGBoost, and Wide and Deep, to train and test predictive models for the ASES, UCLA, Constant, global shoulder function, and VAS pain scores, as well as active abduction, forward flexion, and external rotation. Our primary study goal was to quantify the accuracy of three machine learning techniques to predict each outcome measure at multiple postoperative timepoints after aTSA and rTSA using the mean absolute error between the actual and predicted values. Our secondary study goals were to identify whether a patient would experience clinical improvement greater than the MCID and substantial clinical benefit anchor-based thresholds of patient satisfaction for each outcome measure as quantified by the model classification parameters of precision, recall, accuracy, and area under the receiver operating curve.

resultsEach machine learning technique demonstrated similar accuracy to predict each outcome measure at each postoperative point for both aTSA and rTSA, though small differences in prediction accuracy were observed between techniques. Across all postsurgical timepoints, the Wide and Deep technique was associated with the smallest mean absolute error and predicted the postoperative ASES score to ± 10.1 to 11.3 points, the UCLA score to ± 2.5 to 3.4, the Constant score to ± 7.3 to 7.9, the global shoulder function score to ± 1.0 to 1.4, the VAS pain score to ± 1.2 to 1.4, active abduction to ± 18 to 21°, forward elevation to ± 15 to 17°, and external rotation to ± 10 to 12°. These models also accurately identified the patients who did and did not achieve clinical improvement that exceeded the MCID (93% to 99% accuracy for patient-reported outcome measures (PROMs) and 85% to 94% for pain, function, and ROM measures) and substantial clinical benefit (82% to 93% accuracy for PROMs and 78% to 90% for pain, function, and ROM measures) thresholds.

conclusionsMachine learning techniques can use preoperative data to accurately predict clinical outcomes at multiple postoperative points after shoulder arthroplasty and accurately risk-stratify patients by preoperatively identifying who may and who may not achieve MCID and substantial clinical benefit improvement thresholds for each outcome measure. CLINICAL RELEVANCE: Three different commercially available machine learning techniques were used to train and test models that predicted clinical outcomes after aTSA and rTSA; this device-type comparison was performed to demonstrate how predictive modeling techniques can be used in the near future to help answer unsolved clinical questions and augment decision-making to improve outcomes after shoulder arthroplasty.