基于论坛语料识别中文未登录词的方法

为解决中文分词中未登录词识别效率低的问题,提出了基于论坛语料识别中文未登录词的新方法.利用网络蜘蛛下载论坛网页构建一个语料库,并对该语料库进行周期性的更新以获取具备较强时效性的语料;利用构造出的新统计量MD(由Mutual Information函数和Duplicated Combination Frequency函数构造)对语料库进行分词产生候选词表;最后通过对比候选词表与原始词表发现未登录词,并将识别出的未登陆词扩充到词库中.实验结果表明,该方法可以有效提高未登录词的识别效率.

Algorithm to recognize unknown Chinese words based on BBS corpus

To deal with the problem of low efficiency of Chinese unknown word segmentation,a new method based on BBS corpus is presented.Network spider is used to download BBS web pages to build a corpus and this corpus is updated periodically in order to obtain a strong limitation.The new statistic MD(constructed by the mutual information function and duplicated combination frequency function) is used to segment the corpus to generate a candidate word list.By comparing candidate words list and the previous lexicon to...