基于文档类密度的特征权重算法*

面对海量数据的管理和分析,文本自动分类技术必不可少。特征权重计算是文本分类过程的基础,一个好的特征权重算法能够明显提升文本分类的性能。本文对比了多种不同的特征权重算法,并针对前人算法的不足,提出了基于文档类密度的特征权重算法(tf-idcd)。该算法不仅包括传统的词频度量,还提出了一个新的概念,文档类密度,它通过计算类内包含特征的文档数和类内总文档数的比值来度量。最后,本文在两个中文常见数据集上对五种算法进行实验对比。实验结果显示,本文提出的算法相比较其他特征权重算法在F1宏平均和F1微平均上都有较大的提升。

A feature weighting algorithm based on document class density

In the face of huge amounts of data management and analysis, automatic text classification technology is necessary. Feature weighting is the basis of text categorization process, a good feature weighting algorithm can significantly improve the performance of text categorization. This paper compares the different types of feature weighting algorithm, and aiming at the shortcomings of the previous algorithm, a feature weighting algorithm based on document class density (tf-idcd) is proposed. The algorithm not only includes traditional measure of word frequency, but also puts forward a new concept, the document class density, it contains characteristics within the class by calculation the number of documents and the ratio of the total number of documents to measure in the class. Finally, Based on the common data set on two Chinese experimental comparison of five kinds of algorithm. The experimental results show that the proposed algorithm compared with other feature weighting algorithm on average F1 macro and micro F1 average has larger ascension.