汉语短文话题提取系统中SDTF*PDF算法的研究

互联网技术得到迅速发展以来,大量信息尤其是文本信息在网上传播。文中面向海量汉语短文话题提取系统中多信源、短文篇幅小的特点,结合词汇语义相似性度量,提出了一个词汇权重计算算法---SDTF PDF(ShortDocumentTermFrequency ProportionalDocumentFrequency),测试表明,基于该算法的汉语短文话题识别系统能够较准确地在海量中文文本信息中自动提取一段时间内(一天或一周,可以指定)的主要话题。

Study on SDTF*PDF algorithm implemented in system of topic retrieval from short Chinese passages

More and more information, especially text information,has spread widely on Internet. To detect hot topics from plenty of Chinese text information,a term weight counting algorithm SDTF*PDF(Short Document Term Frequency * Proportional Document Frequency)was discussed. There were lots of channels in the system implementing this algorithm of detecting topics from short Chinese passages, and the passages in channels were usually short. Results worked out by it indicate that the system of detecting topic from short Chinese passages based on this algorithm can accurately extract the hot topics in a period of time, a day or a week, from enormous Chinese text information.