基于视觉信息补偿的多流音视显著性检测

音视显著性检测方法采用的双流网络结构，在音视信号不一致时，双流网络的音频信息对视频信息产生负面影响，削弱物体的视觉特征；另外，传统融合方式忽视了特征属性的重要程度。针对双流网络的问题进行研究，提出了一种基于视觉信息补偿的多流音视显著性算法（MSAVIC）。首先，在双流网络的基础上增加单独的视频编码分支，保留视频信号中完整的物体外观和运动信息。其次，利用特征融合策略将视频编码特征与音视频显著性特征相结合，增强视觉信息的表达，实现音视不一致情况下对视觉信息的补偿。理论分析和实验结果表明，MSAVIC在四个数据集上超过其他方法2%左右，在显著性检测方面具有较好的效果。

Multi-stream audio-visual saliency detection of visual information compensation

Audio-visual saliency detection methods use dual-stream network structure, when audio-visual signals is inconsistent, its'' audio information has negative impact on video and weakens visual features of objects. In addition, the traditional fusion approaches ignore the importance of feature attributes. Study on dual-stream networks, this paper proposed a multi-stream audio-visual of visual information compensation(MSAVIC) saliency algorithm. Firstly, adding independent video encoding branch based on dual-stream held the appearance and motion information of the object in the audio-visual inconsistent case. Secondly, utilizing feature fusion strategy combined visual encoding feature and audio-visual saliency feature to enhance the expression of visual information and realize the compensation of visual information in the case of sound and visual inconsistency. Theoretical analysis and experimental results show that MSAVIC outperforms the others about 2% on four datasets and has a clear effect on saliency detection.