用于聊天词汇的权重计算方法CDTF*IDF

随着聊天室的广泛使用,对聊天内容监控也变成亟待解决的问题.在聊天室监控中,为衡量聊天数据中词汇对聊天内容的描述能力,现在一般直接采用文本词汇的权重计算方法;然而,这种方法忽视了聊天数据与静态文本结构上的差异,导致计算出的权重并不能准确反应词汇描述聊天内容的能力.针对聊天数据的固有特点,提出了一种专门针对聊天数据的词汇权重计算方法CDTF*IDF.该方法通过分别计算词汇在不同数据源中的权值并汇总、并对重点词汇提高权重等方式来计算聊天数据的词汇权重.基于IRC聊天室内容监控的实验表明:该方法能较好地衡量聊天词汇的权重,同时基于该方法的监控系统能够准确地识别出聊天数据中的话题.

A CDTF*IDF Algorithm for Calculating Term Weight of Chat Data

Chat room monitoring becomes an urgent task with its wide use.In the process of chat room monitoring,in order to scale the ability of terms describing the contents of chat data,chat room monitoring systems at present generally use the text terms weight calculating method.However,this method neglects the difference between chat data and text in structure aspect;hence the weight calculated can not response the feature of chat data accurately.The paper presents a new method to calculate the term weight for chat data named CDTF*IDF.CDTF*IDF considers the special features of chat data.It calculates each term weight in different resources,and then gets the final weight by increasing the weight of key terms and some other means.Experiments based on IRC show that this method can calculate the terms weight accurately;at the same time,the chat room monitoring system based on the proposed method has a good performance in topic detection.