Performance trade-offs between dense prediction and sparse query mechanisms for brain tumor MRI detection: a comparative study of YOLOv8 and RT-DETR.

Objective: This study investigates optimal training strategies for YOLOv8s in brain tumor MRI detection, systematically evaluates the effects of key hyperparameters on model performance, and compares the adaptability and trade-offs between convolution-based dense prediction and Transformer-based sparse query mechanisms in medical image detection. Methods: Experiments were conducted on publicly available Kaggle MRI datasets of meningioma and glioma. YOLOv8s was adopted as the baseline model, and a systematic hyperparameter search was performed over learning rates (0.01, 0.001, and 0.0001) and bounding box regression loss weights (5.0, 7.5, and 10.0), forming nine configurations evaluated via five-fold cross-validation. Based on the optimal setting (learning rate = 0.01, loss weight = 7.5), the original YOLOv8s detection head was replaced with an RT-DETR Transformer-based decoder head. Performance and generalization were assessed on both internal and external test sets, and Score-CAM was employed for model interpretability analysis. Results: The hyperparameter analysis demonstrated that the learning rate was the primary factor affecting detection performance, whereas the influence of the bounding box regression loss weight was relatively limited within a reasonable range. Using the optimal hyperparameter combination (learning rate = 0.01 and box loss weight = 7.5), the native YOLOv8s detection head achieved the best overall performance, with a precision of 91.7%, a recall of 89.2%, and mAP@50 of 92.6%, outperforming the RT-DETR detection head (mAP@50 = 91.1%). On the external test set, YOLOv8s also demonstrated superior cross-domain generalization (mAP@50 = 79.0%), exceeding RT-DETR by 10.1%. Further NMS IoU sensitivity analysis showed that the RT-DETR detection head maintained more stable performance across different NMS IoU thresholds, whereas YOLOv8s was more sensitive to variations in NMS post-processing. Score-CAM visualization further revealed distinct attention patterns, with YOLOv8s exhibiting broader lesion responses and RT-DETR focusing more selectively on lesion regions. Conclusion: Appropriate hyperparameter optimization can substantially improve the performance of brain tumor MRI detection models. The dense prediction mechanism of YOLOv8s provides superior lesion coverage, resulting in better detection performance and cross-domain generalization, whereas the sparse query mechanism of RT-DETR exhibits greater robustness to NMS parameter variations during inference. These findings highlight the complementary characteristics of the two detection paradigms and provide practical guidance for detection head selection and inference strategy optimization in medical object detection.