一种基于多模态感知的双声道音频生成方法

现有多数视频只包含单声道音频，缺乏双声道音频所带来的立体感。针对这一问题，本文提出了一种基于多模态感知的双声道音频生成方法，其在分析视频中视觉信息的基础上，将视频的空间信息与音频内容融合，自动为原始单声道音频添加空间化特征，生成更接近真实听觉体验的双声道音频。我们首先采用一种改进的音频视频融合分析网络，以编码器-解码器的结构，对单声道视频进行编码，接着对视频特征和音频特征进行多尺度融合，并对视频及音频信息进行协同分析，使得双声道音频拥有了原始单声道音频所没有的空间信息，最终生成得到视频对应的双声道音频。在公开数据集上的实验结果表明，本方法取得了优于现有模型的双声道音频生成效果，在STFT距离以及ENV距离两项指标上均取得提升。

A Dual-Channel Audio Generation Method Based on Multimodal Perception

Most existing videos only contain mono audio and lack the stereoscopic sense by dual-channel audio. To address this issue, this paper proposes a method for generating dual-channel audio based on multimodal perception. Based on the analysis of visual information in the video, it fuses the spatial information and the audio content of the video, and generates dual-channel audio that is closer to the real auditory experience. We first encode the mono video via an improved audio-video fusion analysis network with an encoder-decoder structure. Then we fuse the video features and audio features in multiple perspectives. Subsequently, we co-analyze the video and audio information, so that the dual-channel audio has spatial information that the original mono audio does not have. Finally, the corresponding dual-channel audio is generated by the audio-video fusion analysis network. Experimental results demonstrate that our method achieves better performance than existing models in the generation of two-channel audio, with improvements in both STFT distance and ENV distance.