CRC-TrustBench: benchmarking cross-study and cross-country generalization in gut metagenomic colorectal cancer prediction.

Background: Gut metagenomic classifiers for colorectal cancer (CRC) are often evaluated with random splits that place samples from the same studies, countries, or processing workflows in both training and test sets, potentially inflating apparent performance. We developed CRC-TrustBench, a leakage-aware framework for evaluating cross-study and cross-country generalization and probability reliability. Methods: From 2,340 MetaGenoPolis V16 source samples, we curated 1,953 CRC or healthy samples from 15 studies and 10 countries; 185 adenoma samples were retained outside binary model development as a challenge population. Subject-aware random, leave-one-study-out (LOSO), and leave-one-country-out (LOCO) protocols were applied to taxonomic profiles with 2,543 features and functional profiles with 378 features. We compared Elastic Net, LightGBM, cohort-balanced Elastic Net, and pre-specified neural domain-robust methods using fold-specific pre-processing. The Microbiome Leakage and Confounding Audit (MiLCA) quantified study and country shortcuts, while Platt calibration, conformal-style sets, and selective abstention were evaluated without outer-test labels. Results: For taxonomic profiles, LightGBM achieved an area under the receiver operating characteristic curve (AUROC) of 0.874 under random splitting, 0.827 under LOSO, and 0.821 under LOCO; the single pre-specified RANDOM estimate was numerically higher than the held-out-domain summaries. Microbiome profiles predicted study and country identity, and a descriptive feature-level audit indicated pervasive study-associated structure. Cohort balancing yielded limited gains, and neural domain-robust methods did not consistently outperform LightGBM. Platt scaling reduced the mean LOCO Brier score from 0.180 to 0.169 and log loss from 0.630 to 0.521. LightGBM conformal-style coverage fell below the pre-specified 90 and 95% targets, while uncertainty-based abstention reduced error relative to matched-random rejection in most recorded repeats. Functional profiles were less predictive, although more functional features met the pre-specified high-stability tiers. Discussion: The tiered features remain stable candidate hypotheses, and raw counts are not directly comparable across the differently sized feature spaces. Adenoma predictions were heterogeneous, and no reviewed external resource met all criteria for formal independent evaluation. CRC-TrustBench separates in-distribution discrimination from cohort generalization and probability reliability without implying clinical deployment or independent external validation.