一种基于HITS的主题敏感爬行方法

基于主题的信息采集是信息检索领域内一个新兴且实用的方法,通过将下载页面限定在特定的主题领域，来提高搜索引擎的效率和提供信息的质量。其思想是在爬行过程中按预先定义好的主题有选择地收集相关网页，避免下载主题不相关的网页，其目标是更准确地找到对用户有用的信息。探讨了主题爬虫的一些关键问题,通过改进主题模型、链接分类模型的学习方法及链接分析方法来提高下载网页的主题相关度及质量。在此基础上设计并实现了一个主题爬虫系统，该系统利用主题敏感HITS来计算网页优先级。实验表明效果良好。

HITS-based topic sensitive crawling method

Topic crawler is a new and practical application in the field of information retrieval. The main idea is to selectively collect Web pages on a predefined topic and avoid downloading irrelative Web pages in order to find more accurate and useful information for the user. Several key issues of topic crawler were discussed and corresponding new approaches were proposed. Then a topic crawler system was designed and implemented, employing topic sensitive Hyperlink-Induced Topic Search (HITS) to predict the priority of fetched Web pages. The experiments show our system performs well.