stHGC: a self-supervised graph representation learning for spatial domain recognition with hybrid graph and spatial regularization.

Advancements in spatial transcriptomics (ST) technology have enabled the analysis of gene expression while preserving cellular spatial information, greatly enhancing our understanding of cellular interactions within tissues. Accurate identification of spatial domains is crucial for comprehending tissue organization. However, the effective integration of spatial location and gene expression still faces significant challenges. To address this challenge, we propose a novel self-supervised graph representation learning framework named stHGC for identifying spatial domains. Firstly, a hybrid neighbor graph is constructed by integrating different similarity metrics to represent spatial proximity and high-dimensional gene expression features. Secondly, a self-supervised graph representation learning framework is introduced to learn the representation of spots in ST data. Within this framework, the graph attention mechanism is utilized to characterize relationships between adjacent spots, and the self-supervised method ensures distinct representations for non-neighboring spots. Lastly, a spatial regularization constraint is employed to enable the model to retain the structural information of spatial neighbors. Experimental results demonstrate that stHGC outperforms state-of-the-art methods in identifying spatial domains across ST datasets with different resolutions. Furthermore, stHGC has been proven to be beneficial for downstream tasks such as denoising and trajectory inference, showcasing its scalability in handling ST data.