Recommendations for Validating Omics Prediction Models: Insights from a Lung Cancer RNA Biomarker Study.

backgroundExternal validation of predictive models in medical research is crucial to ensure their generalizability and applicability across diverse populations. However, validation often reveals discrepancies in model performance due to cohort differences, sample collection and storage, overfitting, and inconsistencies in data handling. This study investigates the challenges encountered during external validation of predictive models for early lung cancer detection using small RNA biomarkers, tying these challenges to specific validation outcomes and deriving recommendations.

methodsPredictive models based on the XGBoost algorithm, developed from serum samples in the JanusRNA cohort, were externally validated in two independent Norwegian cohorts: HUNT and Norwegian Women and Cancer Study. These cohorts differed in sample types, RNA abundance, library preparation protocols, and lung cancer histologic classification. Strategies to harmonize data processing and address these discrepancies were used to ensure a robust validation process.

resultsValidation revealed significant challenges due to cohort heterogeneity. Median AUC values ranged from 0.50 to 0.66 in validation cohorts, compared with 0.62 to 0.76 in the original models. Models performed worse in the female-only Norwegian Women and Cancer Study cohort, in which plasma was used, highlighting the impact of sample type and cohort characteristics on predictive accuracy.

conclusionBased on the challenges encountered during validation, we propose seven recommendations to guide robust external validation of omics-based predictive models, including harmonizing data processing across cohorts, reevaluating overfitting, and critically assessing model performance for clinical applications. IMPACT: By highlighting practical issues in model validation and providing recommendations, this study supports more reliable and clinically applicable biomarker-based prediction models, ultimately aiding cancer screening and prevention efforts.