Enhancing cancer prognostics with group penalty models: a comparative study on radiomics feature selection in lung adenocarcinomas and meningiomas.

Background: Radiomics, the extraction of features from medical images, has shown promise in cancer prognostics. However, its high-dimensional nature poses challenges for feature selection and model stability. Traditional least absolute shrinkage and selection operator (Lasso) tends to select only one feature from correlated groups, leading to unstable feature selection. This study aimed to compare group penalty models with Lasso in selecting radiomic features for cancer prognosis. Methods: We analyzed 590 lung adenocarcinoma lesions for predicting early-stage spread through air spaces (STAS) and 194 meningioma cases for tumor grade prediction. Various group penalty models, including group Lasso, group minimax concave penalty (MCP), group smoothly clipped absolute deviation (SCAD), and adaptive penalization regression with external covariates using variational Bayes (graper), were employed alongside Lasso. Features were organized into natural groups based on segmentation regions and mathematical properties. Model performance was assessed using 10-fold cross-validation, evaluating area under the curve (AUC) and feature selection stability through Jaccard Index. The best-performing models were validated on independent test sets. Results: For lung adenocarcinomas, group SCAD achieved the highest cross-validation AUC of 0.804 [standard deviation (SD) =0.056] with Jaccard Index of 0.613, compared to Lasso's AUC of 0.776 (SD =0.059) and Jaccard Index of 0.503. In the test set, both models showed comparable performance (group SCAD: AUC =0.874; Lasso: AUC =0.877; P=0.896). For meningiomas, group Lasso achieved the highest cross-validation AUC of 0.816 (SD =0.143) with Jaccard Index of 0.7, versus Lasso's AUC of 0.743 (SD =0.223) and Jaccard Index of 0.41. Test set validation showed no significant difference (group Lasso: AUC =0.877; Lasso: AUC =0.835; P=0.391). Conclusions: Group penalty models demonstrated superior feature selection stability while maintaining comparable predictive performance to Lasso. By selecting biologically meaningful feature groups rather than individual features, these models enhance interpretability and align better with clinical reasoning, offering a robust framework for radiomics-based cancer prognostics.