基于多模态神经网络的图像中文摘要生成方法

图像的自然语言描述(image captioning)是一个融合计算机视觉、自然语言处理和机器学习的跨领域课题。它作为多模态处理的关键技术,近年来取得了显著成果。当前研究大多针对图像生成英文摘要,而对于中文摘要的生成方法研究较少。该文提出了一种基于多模态神经网络的图像中文摘要生成方法。该方法由编码器和解码器组成,编码器基于卷积神经网络,包括单标签视觉特征提取网络和多标签关键词特征预测网络,解码器基于长短时记忆网络,由多模态摘要生成网络构成。在解码过程中,该文针对长短时记忆网络的特点提出了四种多模态摘要生成方法CNIC-X、CNIC-H、CNIC-C和CNIC-HC。在中文摘要数据集Flickr8k-CN上实验,结果表明该文提出的方法优于现有的中文摘要生成模型。

Chinese Image Captioning Method Based on Multimodal Neural Network

Image captioning is a cross-domain task which connects computer vision, natural language processing and machine learning. As a key technology of multimodal processing, it has made remarkable progress in the recent years. Research on image caption generation has typically focused on generating a caption in English for an image, but generating Chinese caption is lack of research. In this paper, we propose a method generating Chinese image caption based on multimodal neural network. This method belongs to the family of encoder-decoder. Encoder based on convolutional neural network, consists of single-label visual feature extraction network and multi-label keyword prediction network. Decoder based on long short-term memory, consists of multimodal caption generation network. During the process of decoding, we propose four multimodal caption generation methods: CNIC-X, CNIC-H, CNIC-C and CNIC-HC. Experimental results on Chinese multimodal dataset Flickr8k-CN show that the proposed method outperforms state-of-the-art Chinese image captioning methods.