汉语分词中组合歧义字段的研究

汉语自动分词中组合歧义是难点问题,难在两点: 组合歧义字段的发现和歧义的消解。本文研究了组合歧义字段在切开与不切时的词性变化规律,提出了一种新的组合歧义字段自动采集方法,实验结果表明该方法可以有效地自动发现组合歧义字段,在1998年1月《人民日报》中就检测到400多个组合歧义字段,远大于常规方法检测到的歧义字段数目。之后利用最大熵模型对60个组合歧义字段进行消歧,考察了六种特征及其组合对消歧性能的影响,消歧的平均准确度达88.05%。

Research on Combinational Ambiguity in Chinese Word Segmentation

One of challenges in Chinese Word Segmentation is the combinational ambiguity problem with two main obstacles: the detection of combinational ambiguities and ambiguity resolution.This paper investigate the structures of combinational ambiguities and proposes a new approach for automatically detecting this type of ambiguities.The experimental result reveals the approach is effective in the tagged corpus of 1998-01 People Daily with about 1 million words,we have detected more than 400 combinational ambiguities,far more than that detected by common approaches.Then the resolutions of 60 combinational ambiguities are carried out using the maximum entropy model.The effect of six kinds of features,as well as their combination,on the performance of disambiguation is further studies.The average accuracy of disambiguation reaches 88.05%.