Feed-forward neural networks for genomic selection: insights into varying levels of dominance heritability.

Genomic selection enables the prediction of genetic and breeding values from high-dimensional marker data, improving selection efficiency in breeding programs. Feed-forward neural networks (FFNN) are a type of deep learning model that approximates complex non-linear relationships between markers and traits through interconnected layers of neurons. The purpose of this study was evaluating the predictive performance of FFNN relative to parametric methods (GBLUP, BayesB, BayesL) and other machine learning (ML) approaches, including kernel-based (Support Vector Machine, SVM) and tree-based ensemble methods (Random Forest, RF), across traits with varying levels of narrow-sense (additive) heritability ([Formula: see text]= 0.05, 0.20, 0.50) and dominance heritability ([Formula: see text] = 0.00, 0.05, 0.10, 0.20). Prediction accuracy was evaluated using Pearson correlation coefficients (PCC), mean absolute error (MAE), and mean squared error (MSE), while model stability was assessed via the coefficient of variation across 100 Monte Carlo replicates. Additionally, Cohen’s d effect size was used to assess the magnitude of differences among methods. The results indicate that in the absence of dominance, parametric methods showed higher PCC, but this advantage disappeared in the presence of dominance, such that in traits with low to medium additive heritability, ML methods showed higher PCC. Among ML methods, FFNN, RF and SVM reported higher PCC at low, medium and high additive heritabilities, respectively, in the absence of dominance. Also, method FFNN was able to perform better in the presence of dominance, especially when the additive heritability was high. An increase in dominance heritability at medium to high levels of additive heritability led to a decrease in PCC in all methods, which was not observed in low additive heritability. The FFNN method showed the highest MAE and MSE in the absence of dominance, which with increasing dominance heritability, this method was able to have the best performance among all methods. The highest stability of estimates was observed in traits with high additive heritability across 100 Monte Carlo replicates. Dominance variance substantially influenced predictive patterns, underscoring the importance of accounting for non-additive genetic effects in model selection. These findings highlight the effectiveness of FFNN for accurate and robust genomic prediction across complex genetic architectures and suggest that incorporating dominance heritability can optimize model performance in breeding programs.