ArticleBriefings in bioinformatics2011
An empirical assessment of validation practices for molecular classifiers.
Article in Briefings in bioinformatics, 2011. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 43 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
43 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Clinical lipidomics in bacterial sepsis: a systematic review of serum and plasma evaluations.Metabolomics : Official journal of the Metabolomic Society · 2026Pooled it
- Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges.Molecular psychiatry · 2022Pooled it
- Clinical outcome prediction by microRNAs in human cancer: a systematic review.Journal of the National Cancer Institute · 2012Pooled it
- Robustness Analyses Substantially Narrow in silico Network-Toxicology Signals Linking Selected Pharmaceuticals and Caffeine to Vitiligo.Clinical, cosmetic and investigational dermatology · 2026Article
- Mass spectrometry-based proteomics of FFPE tissues: progress, limitations, and clinical translation barriers.Clinical proteomics · 2025Review
- microRNA Biomarkers in Paediatric Infection Diagnostics-Bridging the Gap Between Evidence and Clinical Application: A Scoping Review.Non-coding RNA · 2025Review
- Large language models enable tumor-type classification and localization of cancers of unknown primary from genomic data.Cell reports. Medicine · 2025Article
- Prediction of Deleterious Single Amino Acid Polymorphisms with a Consensus Holdout Sampler.Current genomics · 2024Article
- Review
- Blood-based biomarkers and novel technologies for the diagnosis of colorectal cancer and adenomas: a narrative review.Biomarkers in medicine · 2024Review
- Multilingual RECIST classification of radiology reports using supervised learning.Frontiers in digital health · 2023Article
- Access to ground truth at unconstrained size makes simulated data as indispensable as experimental data for bioinformatics methods development and benchmarking.Bioinformatics (Oxford, England) · 2022Article
- Addressing Noise and Estimating Uncertainty in Biomedical Data through the Exploration of Chemical Space.International journal of molecular sciences · 2022Review
- Article
- Generalizing predictions to unseen sequencing profiles via deep generative models.Scientific reports · 2022Article
- PREDICTION OF HEREDITARY CANCERS USING NEURAL NETWORKS.The annals of applied statistics · 2022Article
- Predicting genotoxicity of viral vectors for stem cell gene therapy using gene expression-based machine learning.Molecular therapy : the journal of the American Society of Gene Therapy · 2021Article
- Large-scale benchmark study of survival prediction methods using multi-omics data.Briefings in bioinformatics · 2021Review
- The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models.Biostatistics (Oxford, England) · 2020Article
- Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness.BMJ (Clinical research ed.) · 2020Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
Abstract
Proposed molecular classifiers may be overfit to idiosyncrasies of noisy genomic and proteomic data. Cross-validation methods are often used to obtain estimates of classification accuracy, but both simulations and case studies suggest that, when inappropriate methods are used, bias may ensue. Bias can be bypassed and generalizability can be tested by external (independent) validation. We evaluated 35 studies that have reported on external validation of a molecular classifier. We extracted information on study design and methodological features, and compared the performance of molecular classifiers in internal cross-validation versus external validation for 28 studies where both had been performed. We demonstrate that the majority of studies pursued cross-validation practices that are likely to overestimate classifier performance. Most studies were markedly underpowered to detect a 20% decrease in sensitivity or specificity between internal cross-validation and external validation [median power was 36% (IQR, 21-61%) and 29% (IQR, 15-65%), respectively]. The median reported classification performance for sensitivity and specificity was 94% and 98%, respectively, in cross-validation and 88% and 81% for independent validation. The relative diagnostic odds ratio was 3.26 (95% CI 2.04-5.21) for cross-validation versus independent validation. Finally, we reviewed all studies (n = 758) which cited those in our study sample, and identified only one instance of additional subsequent independent validation of these classifiers. In conclusion, these results document that many cross-validation practices employed in the literature are potentially biased and genuine progress in this field will require adoption of routine external validation of molecular classifiers, preferably in much larger studies than in current practice.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.