词位标注汉语分词中特征模板定量研究

基于字的词位标注的方法能极大地提高汉语分词的性能,该方法将汉语分词转化为字的词位标注问题,词位标注汉语分词中特征模板的设定至关重要,为了更加准确地设定特征模板,从多个角度进行了定量分析,并在国际汉语分词评测Bakeoff2005的PKU和MSRA两种语料上进行了封闭测试,得到如下结论:同等条件下,训练出的模型大小与扩展出的特征数成正比；不同的单字特征模板在同一语料中扩展出的特征数基本相同,单字特征模板对分词性能的贡献要比双字特征模板小得多；增加B特征模板之后,训练时间大大增加,模型大小基本不变,对分词性能都是正增长.

Quantitative research on feature templates for word-position-based tagging Chinese word segmentation

The performance of Chinese word segmentation is greatly improved by word-position-based approaches in recent years.This approach treats Chinese word segmentation as a word-position tagging problem.Feature template selection is crucial in this method,in order to do better,quantitative analysis on feature templates for word-position-based tagging Chinese word segmentation is implemented.Closed evaluations are performed on PKU and MSRA corpus from the second international Chinese word segmentation Bakeoff-2005,and comparative experiments are performed on different feature templates.Experimental results show the following conclusions: under the same conditions,the trained model size is proportional with the number of features.The number of features by different single character feature template is same.These templates’ contribution is much smaller than the double-character feature template.With increasing the B feature template,the training time greatly increased,and size of the model remain unchanged,the performance for Chinese word segmentation is positive growth.