RepFed-Net: privacy-preserving federated learning with reputation-aware aggregation for retinal vessel segmentation using enhanced U-Net.

Accurate retinal blood vessel segmentation is essential for the early diagnosis of vision-threatening diseases such as diabetic retinopathy, glaucoma, and retinal vein occlusion. Although deep learning–based segmentation models have achieved promising performance, centralized training approaches can compromise data privacy and may risk violating medical data protection regulations such as HIPAA and GDPR. Federated Learning (FL) enables collaborative model training without sharing raw data; however, existing FL-based segmentation methods still face challenges related to privacy leakage during parameter exchange, robustness to unreliable client updates, and limited architectural capability for capturing fine vascular structures. To address these challenges, we propose RepFed-Net, a privacy-preserving federated learning framework with reputation-aware aggregation for retinal vessel segmentation. RepFed-Net is built upon an enhanced U-Net architecture that integrates Inception modules for multi-scale feature extraction, Residual connections for stable gradient propagation, Squeeze-and-Excitation blocks for adaptive channel recalibration, and Pyramid Attention for modeling long-range contextual dependencies. This unified architectural design improves the continuity and recovery of thin and low-contrast vascular structures, leading to more accurate segmentation outcomes. To preserve privacy during collaborative training, RepFed-Net incorporates the CKKS homomorphic encryption scheme, enabling secure transmission and aggregation of model parameters directly in the encrypted domain under an honest-but-curious server. In addition, a reputation-aware aggregation strategy adaptively weights encrypted client updates based on smoothed validation F1-scores, improving robustness against noisy or low-quality client contributions without exposing raw data or plaintext model parameters. Experimental results demonstrate that RepFed-Net achieves 95.65% segmentation accuracy on the Retinal Blood Vessel dataset and 96.35% accuracy on the DRIVE dataset, consistently outperforming existing U-Net variants and conventional federated learning approaches. Overall, RepFed-Net provides a unified framework that enhances segmentation performance while offering design-level privacy preservation and robustness suitable for collaborative medical image analysis.