What Do Persistent Misclassifications Tell Us About Alzheimer's Disease Detection using Structural MRI?

Deep learning classifiers applied to structural MRI (sMRI) have achieved high performance in detecting Alzheimer's Disease (AD), yet systematic investigation of their failure modes remains limited. In this study, we trained two deep learning architectures to classify AD from cognitively normal (CN) participants using sMRI data from the ADNI dataset, and examined whether misclassifications persist across models and training configurations. We identified a subgroup of subjects who were persistently misclassified across 100 model instances, and found that these subjects exhibited a markedly different atrophy subtype distribution compared to correctly classified AD cases, with substantial enrichment of hippocampal-sparing and minimal atrophy subtypes. To disentangle whether persistent false negatives (FN) reflect earlier disease stage or atypically presenting disease, we analyzed longitudinal follow-up scans and tested whether model predictions changed as neurodegeneration progressed. A change in prediction (from FN to true positive (TP)) was observed in only a subgroup of subjects and required intervals of up to five years, suggesting that persistent misclassification may not always be explained by disease staging alone. Although the sample size is small, these findings underscore the importance of accounting for disease heterogeneity in the development and evaluation of clinical AI models for AD detection.