A novel hierarchical transfer learning to enhance breast cancer detection on screening mammography using deep learning.

Accurate and early detection of breast cancer in screening mammography remains a critical challenge in medical imaging, particularly due to variations in image quality, tissue density, and annotation standards across datasets. This study aims to design and validate a Hierarchical Transfer Learning (HTL) framework that leverages heterogeneous mammography datasets and multi‑stage pre-processing to achieve more accurate and generalizable deep learning-based detection of breast lesions in screening mammography. Our approach integrates five image enhancement techniques, CLAHE, Gamma Correction, Adaptive Histogram Equalization, Global Histogram Equalization, and Unsharp Masking into the learning process, allowing the model to adapt progressively to enhancement specific features. We evaluate the framework using two state-of-the-art object detection architectures, YOLOv8 and Faster R-CNN, across three publicly available datasets (INbreast, CBIS-DDSM, and MIAS) and a locally collected dataset. Our HTL model achieves a peak mean Average Precision (mAP) of 0.981 and accuracy of 0.998, outperforming previous methods in both generalizability and precision. Moreover, the model maintains robust performance on real-world local data, demonstrating its practical utility for deployment in diverse clinical environments. These results confirm that hierarchical learning combined with strategic image enhancement and pre-processing significantly improves detection performance and supports large scale breast cancer screening applications with robust cross-dataset generalization, enabling consistent performance across heterogeneous imaging domains with varying acquisition protocols, image quality, and annotation standards.