基于非平衡数据分类的单文档自动文摘方法

自动文摘是自然语言处理领域的一个重要研究话题,基于机器学习的自动文摘方法则是该项研究中的一个热点。然而,自动文摘问题中的数据分布有一个重要现象,即文摘句子与非文摘句子的数量相差非常悬殊,该现象将给传统机器学习算法的应用效果带来负面影响。为此,本文针对自动文摘中句子类别分布严重不平衡这一现象,以支持向量机算法为基础,设计了两种有效的处理非平衡自动文摘数据的分类方法。在第一种方法中,将传统支持向量机中正负类平衡的分类间隔转换为不平衡的分类间隔;在第二种方法中,通过将数据集进行切分,设计了一种支持向量机集成学习算法。通过在DUC2001数据集上的实验证明,本文设计的两种基于非平衡数据分类的单文档自动文摘方法显著优于基于传统分类算法的自动文摘方法。

Imbalanced Classification Approaches to Automatic Single-Document Summarization

Machine learning based automatic document summarization approaches have drawn increasing attentions in the natural language processing literature.However,neither of them takes the imbalanced class distribution in automatic document summarization into account,i.e.,the number of the sentences in summary is much fewer than that of in the whole document.It is obvious that the highly imbalanced data distribution will degrade the effectiveness of the conventional machine learning algorithms.This paper addresses the problem of automatic document summarization from a perspective of imbalanced classification and proposes two learning strategies to deal with the highly imbalanced distributed data in automatic single-document summarization effectively.The experimental results on the DUC 2001 data set show the significant performance improvements of our approaches in terms of F1 and ROUGH-2.