融合BERT和卷积门控的生成式文本摘要方法

结合注意力机制的循环神经网络(RNN)模型是目前主流的生成式文本摘要方法,采用基于深度学习的序列到序列框架,但存在并行能力不足或效率低的缺陷,并且在生成摘要的过程中存在准确率低和重复率高的问题.为解决上述问题,提出一种融合BERT预训练模型和卷积门控单元的生成式摘要方法.该方法基于改进Transformer模型,在编码器阶段充分利用BERT预先训练的大规模语料,代替RNN提取文本的上下文表征,结合卷积门控单元对编码器输出进行信息筛选,筛选出源文本的关键内容;在解码器阶段,设计3种不同的Transformer,旨在探讨BERT预训练模型和卷积门控单元更为有效的融合方式,以此提升文本摘要生成性能.实验采用ROUGE值作为评价指标,在LCSTS中文数据集和CNN/Daily Mail英文数据集上与目前主流的生成式摘要方法进行对比的实验,结果表明所提出方法能够提高摘要的准确性和可读性.

An abstractive text summarization method combining BERT and convolutional gating unit

The recurrent neural network(RNN) model combined with the attention mechanism is the current mainstream abstractive text summarization method, which uses a sequence-to-sequence framework based on deep learning. However, the abstractive summarization model based on the RNN has insufficient parallel ability or performance defects of long-term dependence, and the problem of low accuracy and high repetition rate in the process of generating summary. In order to overcome these problems, an abstractive summarization model method combining the BERT pre-training model and the convolutional gating unit is proposed based on the improved Transformer model. In the encoder stage, it makes full use of the large-scale corpus pre-trained by the BERT to replace the RNN to extract the contextual representation of the text, and then combines the convolutional gating unit to filter the output of the encoder to filter out the source text. In the decoder stage, three different Transformers are designed, for exploring a more effective fusion method of the BERT pre-training model and convolutional gating unit to improve the performance of text summarization. The ROUGE value is used as the evaluation index in the experiments. The experimental results on the LCSTS Chinese dataset and CNN/Daily Mail dataset show that the proposed method improves the accuracy and readability of the abstract.