藏语句子相似度算法的研究

该文提出了一种藏语句子相似度的计算方法,即采用散列单词倒排索引和基于句长相似度粗选的算法,快速从语料库中筛选出候选句子的集合,散列单词倒排索引能够有效提高算法的查找速度;再采用基于词形和连续单词序列相似度的多策略精选算法,可以有效衡量两个藏语句子的相似程度。实验结果证明算法是有效的。

Research on Similarity Algorithm Tibetan Sentences

A method to compute the similarity of Tibetan sentences is proposed in this paper. This method takes advantage of the reverse index of a hashed vocabulary and the sentence length based coarse-selection algorithm toextract candidate sentences from the corpus rapidly. The reverse index of the hashed vocabulary promotes the searching speed effectively. The multi-strategy delicate selection algorithm adopting word shape based similarity and the continuous word sequence based similarity, which could effectively assess the similarity extent of two Tibetan sentences. The method is validated by the experiments.
Key wordsnatural language processing;corpus; continuous word series; Tibetan language;sentence similarity