文档中词语权重计算方法的改进

文本的形式化表示一直是文本检索、自动文摘和搜索引擎等信息检索领域关注的基础性问题。向量空间模型(Vector Space Model) 中的tf.idf文本表示是该领域里得到广泛应用并且取得较好效果的一种文本表示方法。词语在文本集合中的分布比例量上的差异是决定词语表达文本内容的重要因素之一,但现在tf.idf方法无法把握这一因素。针对这个问题,本文引入信息论中信息增益的概念,提出一种对tf.idf的改进方法tf.idf.IG文本表示方法。该方法将词语的信息增益作为一个文本表示的一个因子,来衡量词语在文本集合中分布比例在量上的差异。在文本分类实验中,tf.idf.IG文本表示的向量空间模型的分类效果要好于tf.idf方法,验证了改进方法tf.idf.IG的有效性和可行性。

An Improved Approach to Weighting Terms in Text

Text Representation has been the fundamental problem in Information Retrieval,such as text retrieval,automatic summary and search engine.tf.idf(term frequency,inverse document frequency)as one of term weighting schemes in Vector Space Model is a good text representation which is popular and make good results in the field of Information Retrieval.The proportion of distribution of terms in text collection is one of the most important factors of expressing the content of text, but it is beyond tf.idf's power.Because of this,this paper provides an improved approach named tf.idf.IG to remedy this defect by Information Gain from Information Theory.The Information Gain of terms as one factor for term weighting schemes can effectively weight the proportion of distribution of terms.In text classification,tf.idf.IG in this paper overcomes old tf.idf.