NicheDeSig: niche-aware deconvolution and adaptive signature analysis for spatial transcriptomics.

motivationFor spot-based spatial transcriptomics (ST), accurate cell-type deconvolution is essential for downstream analysis since each spot captures mixtures of multiple cell types. Meanwhile, spatial niches define distinct micro-environmental contexts, also salient for biological interpretation. However, existing deconvolution methods usually rely on fixed reference signatures or mapping single cells onto ST spots, without incorporating niche priors or modeling niche-dependent shifts. Consequently, existing methods remain focused on spot-level proportion estimation, with limited ability to support functional analysis of niche-associated molecular programs.

resultsWe present NicheDeSig for niche-aware deconvolution. NicheDeSig models each cell type through adaptive signatures, enabling spot deconvolution under context-dependent signatures and supporting niche-aware analysis of cell-state variation across spatial micro-environments. Our method achieves strong deconvolution performance across the simulated benchmark datasets and improves spatial fidelity in the simulated colon dataset. The learned signatures recover laminar and white-matter-associated programs in the human dorsolateral prefrontal cortex (DLPFC), domain-stratified tumor microenvironment patterns in breast cancer (BRCA), and region-associated signatures in pancreatic ductal adenocarcinoma sample A (PDAC-A) and colorectal liver metastasis analyses. AVAILABILITY AND IMPLEMENTATION: Source code and the archived code snapshot are available at https://github.com/Davidcoach/NicheDeSig and https://doi.org/10.5281/zenodo.20685597.