Rigorous validation of machine learning models for orthodontic extraction prediction using nested cross-validation and smotenc-integrated pipelines.

Background/purpose: Machine learning (ML) in orthodontics often suffers from optimistic bias due to data leakage and improper handling of class imbalance. This study implemented and evaluated a production-grade validation framework - nested cross-validation (CV) - for three distinct algorithm classes (logistic regression, random forest, and gradient boosting) using a consensus-labeled orthodontic dataset. Materials and methods: A retrospective dataset of 500 patients (334 non-extraction/166 extraction) was curated with 41 processed features. Two stratified split protocols (70/30 and 80/20) were evaluated to assess model robustness. To ensure methodological rigor, a 5-fold outer/3-fold inner nested cross-validation was implemented. Crucially, SMOTENC (synthetic minority over-sampling technique for nominal and continuous) was integrated inside the training pipeline to prevent data leakage. Results: All models achieved high discrimination (ROC-AUC Conclusion: The use of nested CV combined with pipeline-integrated SMOTENC provides a conservative and reliable estimate of model performance. While gradient boosting (XGBoost) offers superior predictive power, the framework highlights the trade-off between stability (linear models) and performance (boosting) in dentistry.