面向视觉问答的多模块协同注意模型

视觉问答（VQA）是计算机视觉和自然语言处理领域中典型的多模态问题,然而传统VQA模型忽略了双模态中语义信息的动态关系和不同区域间丰富的空间结构。提出一种新的多模块协同注意力模型,对视觉场景中对象间关系的动态交互和文本上下文表示进行充分理解,根据图注意力机制建模不同类型对象间关系,学习问题的自适应关系表示,将问题特征和带关系属性的视觉关系通过协同注意编码,加强问题词与对应图像区域间的依赖性,通过注意力增强模块提升模型的拟合能力。在开放数据集VQA 2.0和VQA-CP v2上的实验结果表明,该模型在“总体”、“是/否”、“计数”和“其他”类别问题上的精确度明显优于DA-NTN、ReGAT和ODA-GCN等对比方法,可有效提升视觉问答的准确率。

Multi-Module Co-Attention Model for Visual Question Answering

Visual Question Answering(VQA) is a typical multi-modal problem in computer vision and natural language processing.Most of the existing VQA models ignore the dynamic relationships of semantic information between two modes and the rich spatial structure of an image.For this reason, the paper proposes a novel multi-module Co-Attention Network named MMCAN, which can fully understand the dynamic interactions between objects and contextual text representation in a visual scenario.Based on the graph attention mechanism, relations between different types of objects are modeled.The adaptive relation representation of the problem is learnt, and the visual object relations as well as the problem features are encoded through co-attention to strengthen the dependence between the words and corresponding image areas.Finally, the enhancement module is used to improve the fitting ability of the model. Experimental results on the open data set VQA 2.0 and VQA-CP V2 show that the accuracy of the proposed model is significantly better than that of DA-NTN, ReGAT and ODA-GCN for "total", "yes/no", "count" and "other" categories of questions.It can effectively improve the accuracy of visual question answering.