基于复合图文特征的视觉问答模型研究

针对当前主流视觉问答(visual question answering,VQA)任务使用区域特征作为图像表示而面临的训练复杂度高、推理速度慢等问题,提出一种基于复合视觉语言的卷积网络(composite visionlinguistic convnet,CVlCN)来对视觉问答任务中的图像进行表征.该方法将图像特征和问题语义通过复合学习表示成复合图文特征,然后从空间和通道上计算复合图文特征的注意力分布,以选择性地保留与问题语义相关的视觉信息.在VQA-v2数据集上的测试结果表明,该方法在视觉问答任务上的准确率有明显的提升,整体准确率达到64.4％.模型的计算复杂度较低且推理速度更快.

Research on visual question answering model based on composite graphic features

In view of the problems of high training complexity and slow inference speed involved by the current mainstream visual question answering task which uses regional features as image representations, this paper proposed a convolutional network(composite visionlinguistic ConvNet, CVlCN) based on composite visual language to extract the image features in visual question answering tasks. The proposed method represented image features and problem semantics into composite picture-text features through composite learning, and then calculated the attention distribution of composite picture-text features from space and channels to selectively retain visual information related to problem semantics. The experimental results show that, on the VQA-v2 dataset, the test accuracy of the proposed method on the visual question answering task is obviously improved, and the overall accuracy is 64.4%. And the model has low computational complexity and fast inference speed.