基于“固结词串”实例的中文分词研究

近几年的中文分词研究中,基于条件随机场(CRF)模型的中文分词方法得到了广泛的关注。但是这种分词方法在处理歧义切分方面存在一定的问题。CRF虽然可以消除大部分原有的分词歧义,却会带来更多新的错误切分。该文尝试找到一种简单的、基于“固结词串”实例的机器学习方法解决分词歧义问题。实验结果表明,该方法可以简单有效的解决原有的分词歧义问题,并且不会产生更多新的歧义切分。

Research on Stable String Example-Based Chinese Word Segmentation

Chinese word segmentation based on CRF(Conditional Random Field) has attracted the most attention in recent research.But this method has certain defects in handling the ambiguity of word segmentation: eliminating most original ambiguity errors at the cost of more new errors.In this paper,we attempt on a simple and example-based machine learning method to deal with the problem of word segmentation ambiguity: the method based on stable string.The experiment results indicate that stable string based method can solve the ambiguity simple and effective.And it will not introduce more new errors.