An effective two-stage key frame extraction method for speech-visual emotion recognition.

Speech-visual emotion recognition plays a vital role in human-computer interaction applications. However, it typically confronts several challenges, such as: (1) conventional speech-visual key frame (SVKF) extraction methods are susceptible to redundancy and emotional information loss; (2) widely adopted attention-based speech-visual feature fusion approaches often compute weights with limited interpretability. To address these challenges, this paper proposes an effective two-stage key frame extraction method for speech-visual emotion recognition. Specifically, in the first stage, visual key frames (VKFs) are extracted by employing information entropy (IE) to model the continuous process of emotion generation, thereby decreasing visual frame redundancy. Corresponding speech key frames (SKFs) are obtained simultaneously by eliminating silent segments to reduce redundancy in the speech modality. Subsequently, by leveraging the complementarity characteristics of speech and visual modalities, the first-stage SKFs and VKFs are aligned to produce the ultimate second-stage SVKFs for preserving important emotional information, in which a simple and interpretable weighted fusion is also proposed to focus on processing important emotional information. The experimental results on the RML, eNTERFACE05, MEAD and BAUM-1s datasets demonstrate that the proposed effective two-stage key frame extraction has better inference and generalization performance.