跨语言相似文档检索

检索一篇文档在其他语言中的译文对于双语平行语料库的建立是一件很有意义的工作。本文提出一种改进的跨语言相似文档检索算法,该算法使用双语词典或统计翻译模型作为双语知识库,查找两篇文档的共同翻译词对,把翻译词对的权重作为一种特征来进行相似度计算,用Dice方法的改进算法计算双语文档的相似度。在实验中,统计检索文档的译文排在检索结果前 N位的总次数来评价算法的性能,并使用了两个噪音数据集来评价算法的有效性。实验表明,在噪音数据干扰比较大的情况下,译文排在检索结果前5位的译文结果接近90%。实验证明,翻译词对的权重对于相似度计算有很大帮助,本算法可以有效地发现一种语言书写的文档在另一种语言中的译稿。

Cross-Language Similar Document Retrieval

To retrieve translations of a document is very helpful for bilingual parallel corpora construction.This paper proposes an improved approach for this purpose,which uses statistical translation model to match bilingual word-pairs,uses weights of word-pairs as features for computing similarity and uses a new Dice-based method to compute Cross-Language document similarity.The approach was evaluated by measuring the numbers of how many times the translation of a given document was identified in the top N similar documents. Although two noisy datasets were used in the experiment,about 90% translations were identified in the top 5 similar documents.The experimental results show that the weighs of bilingual words-pairs are good features for similarity computing and this approaqch can effectively find translation equivalent of a document in other languages.