基于Multigram语言模型的主动学习中文分词

分词是中文处理中的重要基础问题。为了克服Web文本分析中传统方法在适应繁杂的专业领域和多变的语言现象时存在的困难,本文以无督导分词方法为基本框架,使用EM算法建立n元multigram语言模型,提出了一种基于置信度的主动学习分词算法,使得系统在主要利用大量未标注数据的同时,还能够主动选择少量最有价值的数据提交人工标注。实验结果表明算法性能优于相关的几种无督导分词算法。

Active Learning in Chinese Word Segmentation Based on Multigram Language Model

Word segmentation is a fundamental task in Chinese processing.To solve the difficulties of traditional methods in coping with various application domains and evolutive language phenomena,this paper adopts an unsupervised learning framework,using EM algorithm to train the n-multigram language model.A new certainty-based active learning segmentation algorithm is proposed,which combine labeled data with unlabeled data together to optimize language model.In experiments it outperforms other unsupervised word segmentation algorithms.