一种新的基于统计的词典扩展方法

在建立统计语言模型时,往往会遇到词典的词汇量不够的问题。对于医学等专业领域的语料,这一问题尤为严重。针对这一问题,本文提出了一种新的基于统计的识别新词方法——右边缘扩展法。该方法对分词后的语料中产生的连续单字词进行关联范数估计,利用右边缘扩展的方法判断词的边界。在实验中,我们将右边缘扩展法与基于Witten-Bell back off方法的两两合并法相结合,循环地调整词典,优化语言模型。实验结果表明,该算法具有很高的识别正确率与检出率,可以有效地识别出语料中出现的新词汇,尤其是专业术语。

A New Statistical Method of Automatic Lexicon Augmentation

The out of vocabulary problem is one of the bottlenecks in Chinese Language Modeling.The problem is especially serious for domain specific training data set.This paper presents a new statistical method to extract new words from the training data.This new method is based on association norm estimation,and searches for the word boundaries by right boundary expanding.Combining the new method with another word merging method,we can iteratively optimize the lexicon,segmentation and language model.And very encouraging results are reported in our experiments.