联合多头数据增强与多粒度语义挖掘的图像情感分析

图像情感分析是机器视觉领域热点问题,然而情感判断主观性较强,仅分析完整图像难以准确刻画图像中情感语义,且高质量图像情感数据不足.为此,提出联合多头数据增强与多粒度语义挖掘的图像情感分析模型M².首先,设计多头数据增强方法,基于自动数据增强与主动样本精选策略构建递进式数据增强模型,从“质”与“量”两个角度提升数据集;其次,引入情感区域检测模型完成情感区域增强,深入挖掘图像中情感语义强烈的局部区域,进而联合局部区域与整幅图像构建多粒度图像;然后,基于深度互学习框架及局部区域完成模型预训练,充分挖掘异构SENet网络之间互补的情感语义,并以迁移学习方式指导多粒度图像情感分析;最后,设计自适应特征融合模块,融合异构SENet特征以完成多粒度语义挖掘,实现图像情感分析.在Twitter I和FI数据集上验证M²模型,其准确率分别达到90.97%和81.14%,优于主流基线. M²拥有泛化性更强的数据增强策略,可以为其训练提供坚实的数据基础,且对应的实证分析效果较好,模型具备一定的实用价值.

Image sentiment analysis via multi-head data augmentation and multi-granularity semantics mining

Image sentiment analysis is a hot topic in the computer vision field. However, it is hard to accurately characterize the sentiment semantics by only analyzing the whole image since sentiment judgment is usually subjective. And the image samples with high-quality are scarce. To alleviate the two issues, we propose a multi-head data augment and multi-granularity semantics mining(M²) model. First, a progressive data augmentation model is constructed based on automatic data augmentation and active sample refinement. We improve datasets from the perspectives of quality and quantity. Second, an affective region detection model is introduced for sentiment region augmentation. Intense sentiment semantics is deeply mined from these affective local regions. Then we combine local regions with the whole images to create multi-granularity image data. Third, we pretrain the model through the deep mutual learning framework and affective local regions. The complementary sentiment semantics between heterogeneous SENet networks are fully mined, which is transferred in turn to guide the sentiment analysis of multi-granularity images. Finally, an adaptive feature fusion module is proposed to fuse heterogeneous SENet features to complete multi-granularity semantics mining as well as realize image sentiment analysis. The accuracies of M² are 90.97% and 81.14% on TwitterI and FI, respectively, which outperform mainstream baselines. The M² contains a data augmentation strategy with powerful generalization ability, which builds a firm data basis for training. Meanwhile, the corresponding empirical analysis is satisfactory, indicating a certain practicality.