无词典高频字串快速提取和统计算法研究

本文提出了一种快速的高频字串提取和统计方法。使用Hash技术,该方法不需要词典,也不需要语料库的训练,不进行分词操作,依靠统计信息,提取高频字串。用语言学知识进行前缀后缀等处理后,得到的高频字串可以作为未登录词处理、歧义消解和加权处理等的辅助信息。实验显示了该方法速度较快且不受文章本身的限制,在处理小说等真实文本时体现了较高的可用性。

Research on Fast High-frequency Strings Extracting and Statistics Algorithm with no Thesaurus

In this paper we describe a fast high frequency strings extracting algorithm. Our approach uses HASH technology to avoid relying on corpus and word segmentation. To extract the high frequency strings, we only use statistics information. After processing the prefixes and suffixes, the high frequency strings we get can be the supplement knowledge for the un-login words processing, word disambiguation and word weighting. The experimental results show that it has a high speed and can work on arbitrary texts. Our method has good effect when processing novels and other real texts.