首页 | 本学科首页   官方微博 | 高级检索  
     

一种基于EM非监督训练的自组织分词歧义解决方案
引用本文:王伟,钟义信,等.一种基于EM非监督训练的自组织分词歧义解决方案[J].中文信息学报,2001,15(2):39-45.
作者姓名:王伟  钟义信
作者单位:北京邮电大学智能中心
基金项目:国家自然科学基金资助! (6 9982 0 1)
摘    要:本文旨在提供一种基于非监督训练的分词歧义解决方案和一种分词算法。基于EM的思想,每个句子所对应的所有(或一定范围内)的分词结果构成训练集,通过这个训练集和初始的语言模型可以估计出一个新的语言模型。最终的语言模型通过多次迭代而得到。通过一种基于该最终语言模型的统计分词算法,对于每个句子至少带有一个歧义的测试集的正确切分精度达到85.36%(以句子为单位) 。

关 键 词:EM算法  分词歧义  非监督  

A Self-organized Scheme for Word Segmentation Ambiguity Resolution Based on EM Training Algorithm
WANG Wei,ZHONG Yi,xin,SUN Jian,YANG Li.A Self-organized Scheme for Word Segmentation Ambiguity Resolution Based on EM Training Algorithm[J].Journal of Chinese Information Processing,2001,15(2):39-45.
Authors:WANG Wei  ZHONG Yi  xin  SUN Jian  YANG Li
Affiliation:Research Center of Intelligence of Beijing University of Posts and Telecommunications
Abstract:This paper is mainly to present a word segmentation ambiguity resolution scheme based on unsupervised training.According to the idea of EM,a language model is built increasingly by collection the fractional counts of patterns (such as bigram pair)from the augmentations of all the segmentation candidates of a sentence.The learned language model is incorporated into a statistical segmentor.Experiments show that this scheme can resolve 85.36% ambiguity on test set each sentence of which has at least one ambiguous part(and the accuracy rate is based on sentence).
Keywords:EM algorithm  segmentation ambiguity  unsupervised learning
本文献已被 CNKI 维普 等数据库收录!
点击此处可从《中文信息学报》浏览原始摘要信息
点击此处可从《中文信息学报》下载全文
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号