Benchmarking of large language models to determine candidacy for spine surgery and comparison with conventional machine learning.

backgroundSymptomatic lumbar spinal stenosis (LSS) is a disabling condition with a substantial economic impact. The determination of surgical candidacy for LSS relies on a subjective assessment of multiple clinical and imaging factors, leading to variability in recommendations. Artificial intelligence (AI), including traditional machine learning (ML) and emerging large language models (LLMs), holds promise for enhancing the accuracy and consistency of surgical decision-making. However, the comparative performance of LLMs against established ML methods for this specific task remains unclear. This study therefore benchmarks various LLM configurations against a conventional random forest model in predicting LSS surgery recommendation using structured clinical vignettes.

methodsVarious LLMs, including zero-shot ChatGPT3.5, fine-tuned GPT-NEO (2.7B and 125 M parameters), and BioMedLM, were benchmarked against a random forest model. Using a ground truth established by five spine surgeons from 500 synthetic medical vignettes (36 variables each), models provided surgical recommendations. LLMs received these vignette data in text format; the random forest received structured tabular format, to simulate realistic LSS surgical candidates. Performance was evaluated using root mean square error (RMSE) for regression and area under the receiver operating curve (AUROC) for binary classification.

resultsFor surgical recommendation, zero-shot ChatGPT3.5 achieved an RMSE of 0.37 (AUROC = 0.79). The random forest model, trained on the structured data, achieved superior performance (RMSE = 0.12, AUROC = 0.96). Fine-tuned LLMs showed improved accuracy: GPT-NEO 125 m achieved an RMSE of 0.224 (AUROC = 0.81), BioMedLM an RMSE of 0.258 (AUROC = 0.86), and GPT-NEO 2.7b achieved an RMSE of 0.275 (AUROC = 0.90).

conclusionsWhen benchmarked on structured clinical vignettes, LLMs demonstrate credible but inferior predictive accuracy for LSS surgery recommendation compared to a conventional random forest model. These findings highlight that for well-structured data, specialized ML models retain an advantage. The critical next step is to evaluate LLMs on unstructured, real-world clinical data where their natural language processing strengths may be decisive.