文本分类中词语权重计算方法的改进与应用

文本的形式化表示一直是信息检索领域关注的基础性问题。向量空间模型(Vector SpaceModel)中的tf.idf文本表示是该领域里得到广泛应用,并且取得较好效果的一种文本表示方法。词语在文本集合中的分布比例量上的差异是决定词语表达文本内容的重要因素之一。但是其IDF的计算,并没有考虑到特征项在类间的分布情况,也没有考虑到在类内分布相对均匀的特征项的权重应该比分布不均匀的要高,应该赋予其较高的权重。用改进的TFIDF选择特征词条、用KNN分类算法和遗传算法训练分类器来验证其有效性,实验表明改进的策略是可行的。

Improvement and application to weighting terms based on text classification

Text representation has been the fundamental problem in Information Retrieval.tf.idf（term frequency，inverse document frequency） as one of term weighting schemes in Vector Space Model is a good text representation，Which is popular and make good results in the field of Information Retrieval.The difference of the proportion of distribution of terms in text collection is one of the most important factors of expressing the content of text.But the calculation of IDF，don’t consider the information of distribution about terms among classes，and don’t consider the more term weighting for the terms of the relative distributed balance inner classes.The improved TFIDF are used to select feature，KNN algorithm and genetic algorithm are used to train the classifier.and proves that the improved TFIDF method is feasible.