一种改善的基于语言模型的中文检索系统研究

最近几年提出的语言模型检索系统将语音识别领域的语言模型技术引入信息检索领域并改善了检索系统的性能,但是其隐含的词汇间相互独立的假设并不符合实际情况。尽管统计翻译模型考虑了词汇间的同义词因素,但是由于它没有考虑词汇上下文信息,所以对于解决多义词词义的区分并无帮助。我们提出了触发语言模型检索方法来改善这一状况,通过训练语料得到词汇在一定上下文中的相关比率,同时利用查询条件所含词汇计算触发词汇集合来区别查询条件词汇的具体含义并将相关参数引入文档语言模型形成触发语言模型。实验结果表明我们提出的这个方法显著改善了检索系统的性能,与经典语言模型方法相比,触发语言模型方法的平均查准率提高了约12%,召回率提高了10.8%。

An Improved Language Model-based Chinese IR System

Language model based IR system proposed in recent 5 years has introduced the language model approach in the speech recognition area into the IR community and improves the performance of the IR system effectively. However, the assumption that all the indexed words are irrelative behind the method is not the truth. Though statistical MT approach alleviates the situation by taking the synonymy factor into account, it never helps to judge the different meanings of the same word in varied context. In this paper we propose the trigger language model based IR system to resolve the problem. Firstly we compute the association ratio of the words from training corpus and then get the triggered words collection of the query words to find the real meaning of the word in specific text context. We introduce the relative parameters into the document language model to form the trigger language model based IR system. Experiments have shown that the performance of trigger language model based IR system has been improved greatly. Compared with classical language model IR system, Precision of the trigger language model based IR system increased almost 12% and recall of the system increased 10.8%.