Hyperparameter optimization of YOLO using differential evolution, multi-fidelity optimization, and Bayesian optimization.

Object detection in aerial imagery faces significant challenges from small, randomly oriented, and crowded targets across large frames, where default hyperparameter settings consistently underperform. This paper presents a systematic methodology for hyperparameter optimization of YOLO (You Only Look Once) through a novel integration of Differential Evolution, Multi-fidelity Optimization, and Bayesian Optimization (DE-MFO-BO). Our approach optimizes four critical hyperparameters; learning rate, batch size, momentum, and weight decay for both Adam and SGD (Stochastic Gradient Descent) optimizers. The methodology employs a two-stage strategy, Differential Evolution performs rapid exploration using low-fidelity training (5 epochs) to identify promising hyperparameter regions, followed by Bayesian Optimization with high-fidelity training (up to 800 epochs) for precise refinement. A composite fitness function combining precision, recall, and mean Average Precision (IoU 50–95) (mAP(50–95)) guides the optimization process. We validate this framework on VisDrone2019 and our indigenous Realm dataset using YOLO-11n model. For VisDrone2019, Adam optimizer achieves 18.42% recall improvement and 13.5% mAP(50–95) enhancement, while SGD shows 5.07% precision increase. On the Realm dataset, Adam optimizer demonstrates remarkable gains with 20.96% mAP(50–95) improvement, 10.22% precision increase, and 7.17% recall enhancement, whereas SGD achieves 2.78% precision improvement. These substantial performance improvements, achieved without architectural modifications, establish the effectiveness of our systematic hyperparameter optimization approach for aerial object detection applications.