基于改进最大匹配算法的中文分词粗分方法

中文粗分和歧义消解是中文分词的两大基本过程。通过引入广义词条和诱导词集,在最大匹配算法基础上提出一种中文分词的粗分方法,以最长广义词匹配为原则进行中文分词,利用诱导词集实现交叉型歧义识别。在保证快速准确切分无歧义汉语语句的同时,100%检测并标记有歧义汉语语句中的交叉型歧义,最大程度上简化后续歧义消解过程。通过对含有160万汉字1998年1月人民日报语料测试的结果证明了算法速度、歧义词准确率以及粗分召回率的有效性。

Method of Chinese words rough segmentation based on improvingmaximum match algorithm

Chinese words rough segmentation and ambiguity resolution are two fundamental processes of Chinese word segmentation. Under the introduction of generalized term and induced word set, a method used for Chinese words rough segmentation is proposed based on maximum matching method. It executes Chinese word segmentation under the principle of the longest generalized term matching, and recognizes the overlapping ambiguities by utilizing induced word set. It seg- ments Chinese sentences without any ambiguity rapidly and accurately, detects and marks ambiguities by 100 percent in those sentences which have ambiguities, which will simplify the process of ambiguity resolution to the maximum extent. The result of the experiment on People＇ s Daily corpus in January 1998 which contains 1.6 million Chinese characters shows the method is effective both in speed and accuracy.