A Survey of Text Summarization Approaches Based on Deep Learning

Automatic text summarization (ATS) has achieved impressive performance thanks to recent advances in deep learning (DL) and the availability of large-scale corpora.The key points in ATS are to estimate the salience of information and to generate coherent results.Recently,a variety of DL-based approaches have been developed for better considering these two aspects.However,there is still a lack of comprehensive literature review for DL-based ATS approaches.The aim of this paper is to comprehensively review significant DL-based approaches that have been proposed in the literature with respect to the notion of generic ATS tasks and provide a walk-through of their evolution.We first give an overview of ATS and DL.The comparisons of the datasets are also given,which are commonly used for model training,validation,and evaluation.Then we summarize single-document summarization approaches.After that,an overview of multi-document summarization approaches is given.We further analyze the performance of the popular ATS models on common datasets.Various popular approaches can be employed for different ATS tasks.Finally,we propose potential research directions in this fast-growing field.We hope this exploration can provide new insights into future research of DL-based ATS.     

Improving the readability and saliency of abstractive text summarization using combination of deep neural networks equipped with auxiliary attention mechanism

Rapid and exponential development of textual data in recent years has yielded to the need for automatic text summarization models which aim to automatically condense a piece of text into a shorter version. Although various unsupervised and machine learning-based approaches have been introduced for text summarization during the last decades, the emergence of deep learning has made remarkable progress in this field. However, deep learning-based text summarization models are still in their early steps of development and their potential has yet to be fully explored. Accordingly, a novel abstractive summarization model is proposed in this paper which utilized the combination of convolutional neural network and long short-term memory integrated with auxiliary attention in its encoder to increase the saliency and coherency of generated summaries. The proposed model was validated on CNN\Daily Mail and DUC-2004 datasets and empirical results indicated that not only the proposed model outperformed existing models in terms of ROUGE metric but also its generated summaries had higher saliency and readability compared to the baseline model according to human evaluation.

     

项目上下文增强的自动代码摘要

代码摘要通过生成源代码片段的简短自然语言描述, 可帮助开发人员理解代码并减少文档工作. 近期, 关于代码摘要的研究工作主要采用深度学习模型, 这些模型中的大多数都在由独立代码摘要对组成的大型数据集上进行训练. 尽管取得了良好的效果, 这些工作普遍忽略了代码片段和摘要的项目级上下文信息, 而开发人员在编写文档时往往高度依赖这些信息. 针对该问题, 研究了一种与开发者行为和代码摘要工具实现更加一致的代码摘要场景——项目级代码摘要, 其中, 创建了用于项目特定代码摘要的数据集, 该数据集包含800k方法摘要对及其生命周期信息, 用于构建特定时刻准确的项目项目上下文; 提出了一种新颖的深度学习方法, 利用高度相关的代码片段及其相应的摘要来表征上下文语义, 并通过迁移学习整合从大规模跨项目数据集中学到的常识. 实验结果表明: 基于项目级上下文的代码摘要模型不仅能够比通用代码摘要模型获得显著的性能提升, 同时, 针对特定项目能够生成更一致的摘要.     

融合BERT的多层次语义协同模型情感分析研究

由于基于变换器的双向编码器表征技术（Bidirectional Encoder Representations from Transformers,BERT）的提出,改变了传统神经网络解决句子级文本情感分析问题的方法。目前的深度学习模型BERT本身学习模式为无监督学习,其需要依赖后续任务补全推理和决策环节,故存在缺乏目标领域知识的问题。提出一种多层协同卷积神经网络模型（Multi-level Convolutional Neural Network,MCNN）,该模型能学习到不同层次的情感特征来补充领域知识,并且使用BERT预训练模型提供词向量,通过BERT学习能力的动态调整将句子真实的情感倾向嵌入模型,最后将不同层次模型输出的特征信息同双向长短期记忆网络输出信息进行特征融合后计算出最终的文本情感性向。实验结果表明即使在不同语种的语料中,该模型对比传统神经网络和近期提出的基于BERT深度学习的模型,情感极性分类的能力有明显提升。     

MCMR: Maximum coverage and minimum redundant text summarization model

In paper, we propose an unsupervised text summarization model which generates a summary by extracting salient sentences in given document(s). In particular, we model text summarization as an integer linear programming problem. One of the advantages of this model is that it can directly discover key sentences in the given document(s) and cover the main content of the original document(s). This model also guarantees that in the summary can not be multiple sentences that convey the same information. The proposed model is quite general and can also be used for single- and multi-document summarization. We implemented our model on multi-document summarization task. Experimental results on DUC2005 and DUC2007 datasets showed that our proposed approach outperforms the baseline systems.     

面向Web图像检索的基于语义迁移的无监督深度哈希

当前主流的Web图像检索方法仅考虑了视觉特征,没有充分利用Web图像附带的文本信息,并忽略了相关文本中涉及的有价值的语义,从而导致其图像表达能力不强。针对这一问题,提出了一种新的无监督图像哈希方法——基于语义迁移的深度图像哈希（semantic transfer deep visual hashing,STDVH）。该方法首先利用谱聚类挖掘训练文本的语义信息;然后构建深度卷积神经网络将文本语义信息迁移到图像哈希码的学习中;最后在统一框架中训练得到图像的哈希码和哈希函数,在低维汉明空间中完成对大规模Web图像数据的有效检索。通过在Wiki和MIR Flickr这两个公开的Web图像集上进行实验,证明了该方法相比其他先进的哈希算法的优越性。     

基于深度迁移学习的网络入侵检测

为解决网络入侵检测问题,提高检测准确率和降低误报率,提出一种基于深度迁移学习的网络入侵检测方法,该方法使用非监督学习的深度自编码器来进行迁移学习,实现网络的入侵检测。首先对深度迁移学习问题进行建模,然后对深度模型进行迁移学习。迁移学习框架由嵌入层和标签层实现编/解码,编码和解码权重由源域和目标域共享,用于知识的迁移。嵌入层中,通过最小化域之间的嵌入实例的KL散度来强制源域和目标域数据的分布相似;在标签编码层中,使用softmax回归模型对源域的标签信息进行编码分类。实验结果表明,该方法能够实现网络入侵检测,且性能优于其他入侵检测方法。     

Semantic separator learning and its applications in unsupervised Chinese text parsing

Grammar learning has been a bottleneck problem for a long time. In this paper, we propose a method of semantic separator learning, a special case of grammar learning. The method is based on the hypothesis that some classes of words, called semantic separators, split a sentence into several constituents. The semantic separators are represented by words together with their part-of-speech tags and other information so that rich semantic information can be involved. In the method, we first identify the semantic separators with the help of noun phrase boundaries, called subseparators. Next, the argument classes of the separators are learned from corpus by generalizing argument instances in a hypernym space. Finally, in order to evaluate the learned semantic separators, we use them in unsupervised Chinese text parsing. The experiments on a manually labeled test set show that the proposed method outperforms previous methods of unsupervised text parsing.     

融合BERT和卷积门控的生成式文本摘要方法

结合注意力机制的循环神经网络(RNN)模型是目前主流的生成式文本摘要方法,采用基于深度学习的序列到序列框架,但存在并行能力不足或效率低的缺陷,并且在生成摘要的过程中存在准确率低和重复率高的问题.为解决上述问题,提出一种融合BERT预训练模型和卷积门控单元的生成式摘要方法.该方法基于改进Transformer模型,在编码器阶段充分利用BERT预先训练的大规模语料,代替RNN提取文本的上下文表征,结合卷积门控单元对编码器输出进行信息筛选,筛选出源文本的关键内容;在解码器阶段,设计3种不同的Transformer,旨在探讨BERT预训练模型和卷积门控单元更为有效的融合方式,以此提升文本摘要生成性能.实验采用ROUGE值作为评价指标,在LCSTS中文数据集和CNN/Daily Mail英文数据集上与目前主流的生成式摘要方法进行对比的实验,结果表明所提出方法能够提高摘要的准确性和可读性.     

The nature of unsupervised learning in deep neural networks: A new understanding and novel approach

Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural network due to its deep architecture. In this paper we investigate the nature of unsupervised learning in restricted Boltzmann machine. We have proved that maximization of the log-likelihood input data distribution of restricted Boltzmann machine is equivalent to minimizing the cross-entropy and to special case of minimizing the mean squared error. Thus the nature of unsupervised learning is invariant to different training criteria. As a result we propose a new technique called “REBA” for the unsupervised training of deep neural networks. In contrast to Hinton’s conventional approach to the learning of restricted Boltzmann machine, which is based on linear nature of training rule, the proposed technique is founded on nonlinear training rule. We have shown that the classical equations for RBM learning are a special case of the proposed technique. As a result the proposed approach is more universal in contrast to the traditional energy-based model. We demonstrate the performance of the REBA technique using wellknown benchmark problem. The main contribution of this paper is a novel view and new understanding of an unsupervised learning in deep neural networks.     

Digital document analytics using logistic regressive and deep transition-based dependency parsing

The selection of text features is a fundamental task and plays an important role in digital document analysis. Conventional methods in text feature extraction necessitate indigenous features. Obtaining an efficient feature is an extensive process, but a new and real-time representation of features in text data is a challenging task. Deep learning is making inroads in digital document mining. A significant distinction between deep learning and traditional methods is that deep learning learns features in a digital document in an automatic manner. In this paper, logistic regression and deep dependency parsing (LR-DDP) methods are proposed. The logistic regression token generation model generates robust tokens by means of Napierian grammar. With the robust generated tokens, a deep transition-based dependency parsing using duplex long short-term memory is designed. Experimental results demonstrate that our dependency parser achieves comparable performance in terms of digital document parsing accuracy, parsing time and overhead when compared to existing methods. Hence, these methods are found to be computationally efficient and accurate.

     

基于知识块摘要和词转移距离的高效司法文档分类

随着全国司法机关智能化建设的深入推进,通过信息化建设应用所积累的海量司法文书为开展司法智能服务提供了司法数据分析基础。通过司法文书的相似性分析实现类案推送,可以为司法人员提供智能辅助办案决策支持,从而提高办案的质量和效率。针对面向通用领域的文本分类方法因没有考虑特定司法领域文本的复杂结构和知识语义而导致司法文本分类的效能低问题,提出一种基于司法知识块摘要和词转移距离（WMD）的高效司法文档分类方法。首先为司法文书构建领域本体知识模型,进而基于领域本体,利用信息抽取技术获取司法文档中核心知识块摘要;然后基于司法文本的知识块摘要利用WMD进行司法文档相似度计算;最后利用K最近邻算法进行司法文本分类。以两个典型罪名的案件文档集作为实验数据,与传统的WMD文档相似度计算方法进行对比,实验结果表明,所提方法能明显提高司法文本分类的正确率（分别有5.5和9.9个百分点的提升）,同时也降低了文档分类所需的时间（速度分别提升到原来的52.4和89.1倍）。     

基于域自适应的云南重彩画无监督情感识别

计算机视觉中,深度学习之所以取得如此巨大的突破,均得益于可获得的大规模标记数据集,而在图像情感分析中,由于情感语义的模糊性,导致图像情感标注困难,公开可获得图像情感数据集较少,其规模也小,制约了基于深度学习的图像情感分析性能。情感语义具有其特有的有序性和极性分组特征,而已有的图像情感分析方法中很少关注到情感语义的这些本质特征。基于域自适方法,考虑情感语义的本质特征,提出一种基于推土机距离的情感语义对齐方式,将带标记的情感数据集的训练模型更好地迁移到无标记的情感数据集上,实现无监督的图像情感分析,解决情感数据集标注困难的问题。该方法应用于创建的云南重彩画数据集,实验结果表明,其能有效地对齐源域和目标域数据,实现无监督的情感数据集自动标注,有利于扩充图像情感数据集规模。     

An approach to online identification of Takagi-Sugeno fuzzy models.

An approach to the online learning of Takagi-Sugeno (TS) type models is proposed in the paper. It is based on a novel learning algorithm that recursively updates TS model structure and parameters by combining supervised and unsupervised learning. The rule-base and parameters of the TS model continually evolve by adding new rules with more summarization power and by modifying existing rules and parameters. In this way, the rule-base structure is inherited and up-dated when new data become available. By applying this learning concept to the TS model we arrive at a new type adaptive model called the Evolving Takagi-Sugeno model (ETS). The adaptive nature of these evolving TS models in combination with the highly transparent and compact form of fuzzy rules makes them a promising candidate for online modeling and control of complex processes, competitive to neural networks. The approach has been tested on data from an air-conditioning installation serving a real building. The results illustrate the viability and efficiency of the approach. The proposed concept, however, has significantly wider implications in a number of fields, including adaptive nonlinear control, fault detection and diagnostics, performance analysis, forecasting, knowledge extraction, robotics, behavior modeling.     

基于Siamese LSTM的中文多文档自动文摘模型

在文本信息数量迅速增长的环境下,为提升阅读效率,提出一种基于深度学习的多文档自动文本摘要模型。在传统文摘模型的基础上将Siamese LSTM深度学习网络应用到文本相似度计算中,计算曼哈顿距离来表征文本相似度,并采用去除停用词的方法改进该网络模型以提升计算效率。实验结果表明,使用Siamese LSTM与传统余弦相似度等方法相比,生成的文摘在语义方面更贴近主题,质量更高,整个文摘系统的工作效率也显著提升。     

多层校正的无监督领域自适应方法

目的 目前深度神经网络已成功应用于众多机器学习任务,并展现出惊人的性能提升效果。然而传统的深度网络和机器学习算法都假定训练数据和测试数据服从的是同一分布,而这种假设在实际应用中往往是不成立的。如果训练数据和测试数据的分布差异很大,那么由传统机器学习算法训练出来的分类器的性能将会大大降低。为了解决此类问题,提出了一种基于多层校正的无监督领域自适应方法。方法 首先利用多层校正来调整现有的深度网络,利用加法叠加来完美对齐源域和目标域的数据表示;然后采用多层权值最大均值差异来适应目标域,增加网络的表示能力;最后提取学习获得的域不变特征来进行分类,得到目标图像的识别效果。结果 本文算法在Office-31图像数据集等4个数字数据集上分别进行了测试实验,以对比不同算法在图像识别和分类方面的性能差异,并进行准确度测量。测试结果显示,与同领域算法相比,本文算法在准确率上至少提高了5%,在应对照明变化、复杂背景和图像质量不佳等干扰情况时,亦能获得较好的分类效果,体现出更强的鲁棒性。结论 在领域自适应相关数据集上的实验结果表明,本文方法具备一定的泛化能力,可以实现较高的分类性能,并且优于其他现有的无监督领域自适应方法。     

结合预训练模型和语言知识库的文本匹配方法

针对文本匹配任务,该文提出一种大规模预训练模型融合外部语言知识库的方法。该方法在大规模预训练模型的基础上,通过生成基于WordNet的同义—反义词汇知识学习任务和词组—搭配知识学习任务引入外部语言学知识。进而,与MT-DNN多任务学习模型进行联合训练,以进一步提高模型性能。最后利用文本匹配标注数据进行微调。在MRPC和QQP两个公开数据集的实验结果显示,该方法可以在大规模预训练模型和微调的框架基础上,通过引入外部语言知识进行联合训练有效提升文本匹配性能。     

HAUSS: Incrementally building a summarizer combining multiple techniques

The idea of automatic summarization dates back to 1958, when Luhn invented the “auto abstract” (Luhn, 1958). Since then, many diverse automatic summarization approaches have been proposed, but no single technique has solved the increasingly urgent need for automatic summarization. Rather than proposing one more such technique, we suggest that the best solution is likely a system able to combine multiple summarization techniques, as required by the type of documents being summarized. Thus, this paper presents HAUSS: a framework to quickly build specialized summarizers, integrating several base techniques into a single approach. To recognize relevant text fragments, rules are created that combine frequency, centrality, citation and linguistic information in a context-dependent way. An incremental knowledge acquisition framework strongly supports the creation of these rules, using a training corpus to guide rule acquisition, and produce a powerful knowledge base specific to the domain. Using HAUSS, we created a knowledge base for catchphrase extraction in legal text. The system outperforms existing state-of-the-art general-purpose summarizers and machine learning approaches. Legal experts rated the extracted summaries similar to the original catchphrases given by the court. Our investigation of knowledge acquisition methods for summarization therefore demonstrates that it is possible to quickly create effective special-purpose summarizers, which combine multiple techniques, into a single context-aware approach.     

Making understanders out of parsers: Semantically driven parsing as a key concept for realistic text understanding applications

Semantically driven natural language parsers have found wide-spread application as a text processing methodology for knowledge-based information retrieval systems. It is argued that this parsing technique particularly corresponds to the requirements inherent to large-scale text analysis. Unfortunately, this approach suffers from several shortcomings which demand a thorough reformulation of its paradigm. Incorporating principles from conceptual analysis and word expert parsing in a model of lexically distributed text parsing, the focus of the modifications proposed in this article, is on a clean declarative separation of linguistio and other knowledge representation levels, abstraction mechanisms leading to a small collection of specification primitives for the parser, and an attempt to incorporate linguistic generalizations and modularization principles into the design of a semantic text grammar. A sample parse illustrates the operation and linguistic coverage of a lexically distributed text parser based on these theoretical considerations with respect to the semantic analysis of noun groups, simple assertional sentences, nominal anaphora, and textual ellipsis.