基于类信息的文本特征选择与加权算法研究

文本自动分类中特征选择和加权的目的是为了降低文本特征空间维数、去除噪音和提高分类精度。传统的特征选择方案筛选出的特征往往偏爱类分布不均匀文档集中的大类,而常用的TF·IDF特征加权方案仅考虑了特征与文档的关系,缺乏对特征与类别关系的考虑。针对上述问题,提出了基于类别信息的特征选择与加权方法,在两个不同的语料集上进行比较和分析实验,结果显示基于类别信息的特征选择与加权方法比传统方法在处理类分布不均匀的文档集时能有效提高分类精度,并且降维程度有所提高。

Research on algorithm of text feature selection and weighting based on category

The aim of feature selection and weighting in automatic text categorization is to reduce the dimension of feature space,remove noise features and improve classification precision.The features selected by traditional feature selection methods always bias common category,and the commonly used weighting method TF*IDF only considers the relationship between features and documents and ignores the relationship between features and categories.According to the above problem,this paper presents a text feature selection and weighting method based on category.Experiments on skewed category distribution corpus of two different languages show that the method can improve categorization precision effectively,and comparing with traditional method,the feature space dimension is also reduced to a certain degree.