Mood states recognition based on Mandarin speech and deep learning in patients with bipolar disorder.

backgroundCurrent assessments of bipolar disorder mainly rely on clinician’s experience and patients’ self-reporting, which has the risk of observer bias and subjective cognitive bias. There is an urgent need for more objective indicators to improve the accuracy of recognizing and monitoring mood states in patients with bipolar disorder. Speech might be a promising marker for recognizing different mood states of bipolar disorder, because of its accessible, objective, and non-invasive features. This study aims to construct a deep learning model based on Mandarin speech to recognize depressive episode, (hypo)manic episode, and euthymic state in bipolar disorder.

methodsFifty-three patients with bipolar disorder were recruited from the outpatient clinic in Tianjin Anding Hospital from March 2023 to April 2024. Speech recordings were collected in a controlled acoustic environment and were pre-processed. Three speech features including the Mel spectrogram, HuBERT and WavLM self-supervised feature were extracted. The speech feature with the highest recognition rate was selected as the basic speech feature for the mood states recognition. Three mood states recognition models were constructed based on deep learning algorithms: (1) the LSTM model, (2) the ECAPA-TDNN model, and (3) our proposed model (Ours). The Ours model enhances the ECAPA-TDNN architecture by integrating an LSTM layer to capture contextual information, followed by processing through a TFA mechanism. The recognition performance of the constructed models is evaluated by WA, UA, and Macro_F1 value.

results19 patients with bipolar disorder were in a (hypo)manic episode, 15 in a depressive episode, and 19 in a euthymic state. The WavLM self-supervised feature had the highest speech recognition rate (90.13%), followed by the HuBERT self-supervised feature (67.56%) and the Mel Spectrum feature (58.09%). The Ours model achieved the best overall performance, with UA, WA, and Macro_F1 of 85.94%, 85.79%, and 85.78%, respectively, which is 6–8% higher than those of LSTM and ECAPA-TDNN.

conclusionsThis study validates the effectiveness of the fusion model of self-supervised features and deep learning in Mandarin speech recognition of bipolar disorder patients’ mood states. The findings provide evidence for the use of speech features as biomarkers to accurately distinguish mood states in bipolar disorder. CLINICAL TRIAL NUMBER: Not applicable.