基于字的词位标注汉语分词

近年来基于字的词位标注方法极大地提高了汉语分词的性能,该方法将汉语分词转化为字的词位标注问题,借助于优秀的序列标注模型,基于字的词位标注汉语分词方法逐渐成为汉语分词的主要技术路线。该方法中特征模板选择至关重要,采用四词位标注集,使用条件随机场模型进一步研究基于字的词位标注汉语分词技术,在第三届和第四届国际汉语分词评测Bakeoff语料上进行封闭测试,并对比了不同特征模板集对分词性能的影响。实验表明采用的特征模板集:TMPT-10′较传统的特征模板集分词性能更好。

Word-position-based tagging for Chinese word segmentation

 The performance of Chinese word segmentation has been greatly improved by word-position-based approaches in recent years. This approach treats Chinese word segmentation as a word position tagging problem. With the help of powerful sequence tagging model, word-position-based method quickly rose as a mainstream technique in this field. Feature template selection is crucial in this method. We further studied this technique via using four word positions and conditional random fields. Closed evaluations are performed on corpus from the third and the fourth international Chinese word segmentation Bakeoff, and comparative experiments are performed on different feature templates. Experimental results show that the feature template set: TMPT-10′  is much better performance than the traditional template set.