中文文本分类中特征选择方法的研究

本文研究了文档频率DF、信息增益IG、互信息MI、x2分布(CHI)、期望交叉熵、优势率、文本证据权七种不同的特征选取方法.针对DF对高频词过于依赖,以及MI,IG和CHI对低频词过于依赖这一特点,试验了将它们组合起来形成DF-MI,DF-IG两种组合式特征选择方法-同时针对DF的特点提出了新的特征选取方法DFR-用KNN分类器试验了几种组合方法和DFR方法-实验结果表明DFR较DF-MI、DF-IG对分类效果有明显的提高,而组合特征选取方法较单个特征选取方法对分类器的分类效果有了很大的提高.

A Study on Feature Selection in Chinese Text Categorization

This paper is a study of feature selection methods in text categorization. Seven methods ere evaluated, including document frequency (DF), information gain (IG), mutual information (MI), x2-test(CH I), Expected Cross Entropy(CE), Weight of Evidence for Text and Odds Ratio. DF relies on the high frequency word and MI, IG and CHI rely on the low frequency word. So feature selection method of a combined type is used and suppress effectively the lack of the high or low frequency word. Meanwhile we introduce a new feature selection method DFR. A furthermore experiment proved that the combined feature selection method is effective.