基于一致性图卷积模型的多模态对话情绪识别

多模态对话情绪识别是一项根据对话中话语的文本、语音、图像模态预测其情绪类别的任务。针对现有研究主要关注话语上下文的多模态特征提取和融合，而没有充分考虑每个说话人情绪特征利用的问题，提出一种基于一致性图卷积网络的多模态对话情绪识别模型。该模型首先构建了多模态特征学习和融合的图卷积网络，获得每条话语的上下文特征；在此基础上，以说话人在完整对话中的平均特征为一致性约束，使模型学习到更合理的话语特征，从而提高预测情绪类别的性能。在两个基准数据集IEMOCAP和MELD上与其他基线模型进行了比较，结果表明所提模型优于其他模型。此外，还通过消融实验验证了一致性约束和模型其他组成部分的有效性。

Consistency based graph convolution network for multimodal emotion recognition in conversation

Multimodal emotion recognition in conversations(MERC) is a task to predict the emotional category of the discourse in a dialogue based on its textual, audio, and visual modality. Existing studies focus on multimodal feature extraction and fusion of discourse context without fully considering the utilization of emotional features of different speakers. Therefore, this paper proposed a model of multimodal dialogue emotion recognition based on a consistent graph convolutional network. The model first constructed a graph convolutional network of multimodal feature learning and fusion, and obtained the context features of each discourse. On this basis, the average features of the speaker in the complete dialogue as the constraint to make the model learn more reasonable discourse features, so as to improve the performance of predicting emotion class. The paper compared with other baseline models on two benchmark datasets IEMOCAP and MELD and show that this proposed model is superior to the other models. In addition, the paper verifies the consistency constraints and other components of the model through ablation experiments.