基于维基百科的双语可比语料的句子对齐

该文提出了一种从维基百科的可比语料中抽取对齐句子的方法。在获取了维基百科中英文数据库备份并进行一定处理后,重构成本地维基语料数据库。在此基础上,统计了词汇数据、构建了命名实体词典,并通过维基百科本身的对齐机制获得了双语可比语料文本。然后,该文在标注的过程中分析了维基百科语料的特点,以此为指导设计了一系列的特征,并确定了“对齐”、“部分对齐”、“不对齐”三分类体系,最终采用SVM分类器对维基百科语料和来自第三方的平行语料进行了句子对齐实验。实验表明:对于语言较规范的可比语料,分类器对对齐句的分类正确率可达到82%,对于平行语料,可以达到92%,这说明该方法是可行且有效的。

Sentence Alignment for Bilingual Comparable Corpus from Wikipedia

We propose a method that can extract aligned sentences for comparable corpus derived from Wikipedia. First, we retrieve Wikipedia data dump of English and Chinese and re-construct it into a local Wikipedia corpra database. Second, we extract the lexicon, esp. the build named entity lexicon, and obtain bilingual comparable corpus through alignment mechanism provided by Wikipedia. Third, we analyze the characteristics of Wikipedia corpus manually, and designe a series of features. Adopting a taxonomic of alignment/partial alignment/non-alignment, we finally apply SVM classifier to identify the sentences alignment. Experimented on the Wikipedia corpus and a third-party parallel corpus, the proposed method achieves the precision of 0.82 and 0.92, respectively.