基于注意力的多层次混合融合的多任务多模态情感分析

针对多模态情感分析中的模态内部特征表示和模态间的特征融合问题,结合注意力机制和多任务学习,提出了一种基于注意力的多层次混合融合的多任务多模态情感分析模型MAM(multi-level attention and multi-task)。首先,利用卷积神经网络和双向门控循环单元来实现单模态内部特征的提取;其次,利用跨模态注意力机制实现模态间的两两特征融合;再次,在不同层次使用自注意力机制实现模态贡献度选择;最后,结合多任务学习获得情感和情绪的分类结果。在公开的CMU-MOSEI数据集上的实验结果表明,情感和情绪分类的准确率和F;值均有所提升。

Multimodal sentiment analysis based on hybrid feature fusion of multi-level attention mechanism and multi-task learning

Aiming at the problem of intra-modality feature representation and inter modality feature fusion in multimodal sentiment analysis, this paper proposed a multi-level hybrid fusion multi-modal sentiment analysis model based on attention mechanism and multi-task learning.Firstly, the model used convolution neural network and bi-directional gated unit to extract the single-modality internal feature.Secondly, it used the cross-modality attention mechanism to realize the pairwise feature fusion between modalities.Thirdly, it used the self-attention mechanism to select the modality contribution at different levels.Finally, combining with multi-task learning, the model obtained both sentiment and emotion classification results.The experimental results on CMU-MOSEI dataset show that this method can improve the accuracy and F;-score of sentiment and emotion classification.