EMD-HVG: a normalization-independent method for highly variable gene selection based on Earth mover's distance.

backgroundIdentifying highly variable genes (HVGs) is a critical step in single-cell RNA sequencing (scRNA-seq) and spatial transcriptomics (ST) analyses. Conventional approaches typically rely on normalization to adjust for library size differences and estimate variability under predefined distributional assumptions. However, normalization procedures can inadvertently mask true biological variability, and the assumed distributions often fail to adequately capture the sparsity and noise inherent in scRNA-seq and ST data. To address these limitations, this study aims to develop a normalization-independent and nonparametric method for robust HVG identification in scRNA-seq and ST data.

resultsWe propose Earth Mover's Distance based Highly Variable Gene identification method (EMD-HVG), a normalization-free and nonparametric framework for HVG identification. EMD-HVG models gene-specific expression patterns using a mixture distribution across cells or spatial locations, thereby preserving native biological heterogeneity without requiring library size normalization. To measure expression variability, EMD-HVG employs Earth Mover's Distance (EMD), a robust, nonparametric metric that avoids reliance on any specific distributional form.

conclusionExtensive benchmarking on both real and simulated datasets demonstrates that EMD-HVG consistently outperforms existing HVG detection methods. It achieved top performance in 26 out of 30 evaluation scenarios across various accuracy metrics. Furthermore, HVGs identified by EMD-HVG significantly enhance the quality of downstream clustering, facilitating more precise cell-type delineation across diverse biological systems. Together, these results highlight the robustness and broad applicability of EMD-HVG in both single-cell and spatial transcriptomic analyses.