基于word2vec的大中华区词对齐库的构建

该文针对大陆、香港和台湾地区(简称大中华区)存在同一种语义但采用不同词语进行表达的语言现象进行分析。首先,我们抓取了维基百科以及简繁体新闻网站上的3 200 000万组大中华区平行句对,手工标注了一致性程度达到95%以上的10 000组大中华区平行词对齐语料库。同时,我们提出了一个基于word2vec的两阶段大中华区词对齐模型,该模型采用word2vec获取大中华区词语的向量表示形式,并融合了有效的余弦相似度计算方法以及后处理技术。实验结果表明我们提出的大中华区词对齐模型在以上两种不同文体的词对齐语料库上的F1值显著优于现有的GIZA++和基于HMM的基准模型。此外,我们在维基百科上利用该词对齐模型进一步生成了90 029组准确率达82.66%的大中华区词语三元组。

Word2vec Based Word Alignment Corpus for the Greater China Region

We deal with the linguistic phenomenon that different expressions to the same semantic meaning among the Mainland China, Hong Kong and Taiwan, a.k.a., the greater China region(GRC). Firstly, we automatically crawl 3.2 million GCR parallel sentences from the wikipedia and the news website with simplified and traditional encoding, and then manually annotate 10 000 GCR parallel word alignment corpora with an annotation agreement of more than 95%. Meanwhile, we present a 2-phase GCR word alignment model based on word2vec representation of the GCR words the cosine similarity measure and other post-processing techniquest. Experiment results on the proposed 2 different word alignment corpus demenstrate the effectiveness of our GCR model which significantly outperforms the current GIZA++ and HMM-based models. Furthermore, we generate 90,029 triples from wikipedia with accuracy over 82.66%.