基于主题模型的深层网数据源选择算法

联邦搜索是从大规模深层网上获取信息的一种重要技术。给定一个用户查询,联邦搜索系统需要解决的一个主要问题是数据源选择问题,即从海量数据源中选出一组最有可能返回相关结果的数据源。现有的数据源选择算法大多基于数据源的样本文档集和查询之间的关键词匹配,通常无法很好地解决少量样本文档的信息缺失问题。针对这一问题,提出了基于隐含狄利克雷分布(LDA)主题模型进行数据源选择的方法。首先,使用LDA主题模型获得数据源和查询的主题概率分布;然后,通过比较两者主题概率分布的相近性来对所有数据源进行排序。通过将数据源和查询映射到低维的主题空间来解决高维词条空间稀疏性所带来的信息缺失问题。在TREC FedWeb 2013和2014 Track的测试集上分别进行了实验,并和其他参赛方法的结果进行了比较。在FedWeb 2013测试集上的实验结果显示比其他参赛方法的最好结果提高了24%;在FedWeb 2014测试集上的实验结果显示比传统的基于小文档和大文档的关键词匹配方法分别提高了22%和43%。另外,使用文档片段来代替文档还可以大幅提升系统的效率,更增加了此方法的实用性和可行性。

Deep Web resource selection using topic model

Federated search is a widely-used technique to find information on Deep Web. Given a user query, one of the challenges for a federated search system is to select a set of resources that are most likely to return relevant results for the query. Most existing resource selection methods are based on text-matching between the sample documents of the resource and the query, which typically suffer the problem of missing vocabulary or incomplete information. To alleviate the problem of incomplete information, Latent Dirichlet Allocation (LDA) topic model approach for resource selection was proposed. First, topic probability distributions for resources and query were inferred using LDA topic model approach. Then the similarities between the topic distributions of resources and query were calculated to rank the resources. By mapping both resources and the query into the low dimensional topic space, the problem of missing information caused by the sparsity of high dimensional word space was alleviated. Experiments were conducted on the test sets of TREC FedWeb 2013 and 2014 Tracks, and the results were compared with that of other participants in the Tracks. The experimental results on the TREC FedWeb 2013 Track show that the LDA based approach outperforms the best result of other participants by 24%; and the results on the TREC FedWeb 2014 Track show that it outperforms the best results of the traditional text-matching-based resource selection methods using either small-or big-document strategies by 22% for small-document methods and 43% for big-document methods respectively. In addition, using sampled snippets rather than documents to generate big-document representation for resources can significantly improve the efficiency of the system, thus enables the proposed approach more feasible and applicable in practice.