How well do frozen foundation models transfer? A calibration-focused benchmark for diabetic retinopathy grading.

Background: Pretrained foundation models are increasingly adopted for diabetic retinopathy (DR) screening, yet it remains unclear how much of their performance derives from the learned representations versus the adaptation procedure. Most benchmarks report discrimination metrics alone, neglecting probability calibration. Methods: We compared the frozen representations of three pretrained encoders: MedSigLIP (medical vision-language; ViT-B/16, 448 × 448), RETFound (retinal self-supervised; ViT-L/16, 224 × 224), and EfficientNet-B0 (ImageNet-supervised; 224 × 224). All encoder weights were frozen; only an identical lightweight multilayer perceptron head was trained. Models were developed on APTOS 2019 (3,662 fundus images; five-fold cross-validation) and externally validated on MESSIDOR-2 (1,744 images). Binary referable DR detection and five-class severity grading were evaluated. AUC, expected calibration error (ECE), and Brier score served as co-primary endpoints. External-set tests used patient-level cluster-robust bootstrap to account for bilateral correlation. Results: On the development set, all three encoders achieved near-identical binary AUC (0.980-0.985). MedSigLIP showed superior calibration, with a lower Brier score than RETFound (0.044 vs. 0.049; Conclusion: Under frozen transfer, the MedSigLIP encoder package produced more generalisable and better calibrated representations than both retinal self-supervised (RETFound) and ImageNet-supervised (EfficientNet-B0) encoders. Domain-specific pretraining did not guarantee domain-general frozen representations. These findings demonstrate that development-set discrimination alone is insufficient for encoder evaluation and that calibration metrics-particularly the Brier score-should be reported as standard practice.