基于关键短语的文本分类研究

文本分类的进一步改进除了算法方面,应该还立足于影响文本分类最底层、最根本的因素: 文本表示中的特征项,提高特征项的完整独立程度。关键短语是具有强文本表示功能的特征短语,在表示文本时,能将文本的内容特征(如主题类别)鲜明地表示出来。关键短语具有结构稳定、语义完整和较强统计意义的特点,能克服向量空间模型和贝叶斯假设的缺点,更适合作为文本表示的特征,有利于提高文本分类的效果。本文从语言学、认知心理学和言语习得、计算语言学等方面寻求关键短语优势的理论依据,对关键短语进行了界定,通过抽取网页上专家标引的关键词获得关键短语。在约3万篇测试集上(共15个大类,244个小类),与以词为特征的文本分类相比,以关键短语为特征的文本分类的大类微平均提高了3.1％,小类微平均提高了15％。

Text Categorization Based on Key Phrases

Improvement in text categorization lies not on algorithm of classing model,but on the fundamental element: integrated and independent feature of text representation.Key phrases are phrase that have strong text representation function,can characterize text content such as subject and kind.With steady structure,integrated meaning and statistical significance,Key phrases can overcome the limitation of VSM(Vector Space Model) and NB(Naive-Bayes),are fit for feature of text representation,and are propitious to improving effect of text categorization.From linguistics,cognitive psychology and computational linguistics,we searched the base of theory of Key phrases' advantage, defined Key phrases,and acquired them by extracting keywords labeled by specialist in web pages.The experiment proved that Key phrases are fitter for feature of text representation than words: Micro F1 increase of 3.1 percent of parent-category,Micro F1 increase of 15 percent of sub-category.