SynthesisPloS one2020
Inconsistency in the use of the term "validation" in studies reporting the performance of deep learning algorithms in providing diagnosis from medical imaging.
Synthesis in PloS one, 2020. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 16 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
16 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Deep learning for cardiac CT segmentation for congenital heart disease: A systematic review.PloS one · 2026Pooled it
- Performance of Radiomics-based machine learning and deep learning-based methods in the prediction of tumor grade in meningioma: a systematic review and meta-analysis.Neurosurgical review · 2025Pooled it
- Perspectives on the Limits and Clinical Alignment of Medical AI from Population Statistics to Individual Care.Bioengineering (Basel, Switzerland) · 2026Article
- Clarifying validation terminologies in healthcare.NPJ digital medicine · 2026Article
- Differences in technical and clinical perspectives on AI validation in cancer imaging: mind the gap!European radiology experimental · 2025Article
- The Role of Machine Learning in the Detection of Cardiac Fibrosis in Electrocardiograms: Scoping Review.JMIR cardio · 2024Article
- Reporting Guidelines for Artificial Intelligence Studies in Healthcare (for Both Conventional and Large Language Models): What's New in 2024.Korean journal of radiology · 2024Article
- Position Statements of the Emerging Trends Committee of the Asian Oceanian Society of Radiology on the Adoption and Implementation of Artificial Intelligence for Radiology.Korean journal of radiology · 2024Review
- Evaluation of clinical prediction models (part 1): from development to external validation.BMJ (Clinical research ed.) · 2024Article
- Generalizability of prostate MRI deep learning: does one size fit all data?European radiology · 2023Article
- Performance of deep learning algorithms to distinguish high-grade glioma from low-grade glioma: A systematic review and meta-analysis.iScience · 2023Article
- Trustworthy AI: Closing the gap between development and integration of AI systems in ophthalmic practice.Progress in retinal and eye research · 2022Review
- Deep learning in image-based breast and cervical cancer detection: a systematic review and meta-analysis.NPJ digital medicine · 2022Article
- Article
- Advancing diagnostic performance and clinical applicability of deep learning-driven generative adversarial networks for Alzheimer's disease.Psychoradiology · 2021Article
- Key Principles of Clinical Validation, Device Approval, and Insurance Coverage Decisions of Artificial Intelligence.Korean journal of radiology · 2021Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThe development of deep learning (DL) algorithms is a three-step process-training, tuning, and testing. Studies are inconsistent in the use of the term "validation", with some using it to refer to tuning and others testing, which hinders accurate delivery of information and may inadvertently exaggerate the performance of DL algorithms. We investigated the extent of inconsistency in usage of the term "validation" in studies on the accuracy of DL algorithms in providing diagnosis from medical imaging. METHODS AND
findingsWe analyzed the full texts of research papers cited in two recent systematic reviews. The papers were categorized according to whether the term "validation" was used to refer to tuning alone, both tuning and testing, or testing alone. We analyzed whether paper characteristics (i.e., journal category, field of study, year of print publication, journal impact factor [JIF], and nature of test data) were associated with the usage of the terminology using multivariable logistic regression analysis with generalized estimating equations. Of 201 papers published in 125 journals, 118 (58.7%), 9 (4.5%), and 74 (36.8%) used the term to refer to tuning alone, both tuning and testing, and testing alone, respectively. A weak association was noted between higher JIF and using the term to refer to testing (i.e., testing alone or both tuning and testing) instead of tuning alone (vs. JIF <5; JIF 5 to 10: adjusted odds ratio 2.11, P = 0.042; JIF >10: adjusted odds ratio 2.41, P = 0.089). Journal category, field of study, year of print publication, and nature of test data were not significantly associated with the terminology usage.
conclusionsExisting literature has a significant degree of inconsistency in using the term "validation" when referring to the steps in DL algorithm development. Efforts are needed to improve the accuracy and clarity in the terminology usage.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.