摘要： 分析传统搜索引擎分配任务的方式及存在的问题，根据垂直搜索引擎的特点，提出一种比传统方法粒度更细的任务分配方式——网站划分。该分配方式将较大规模的网站切分为若干较小规模的子集，并将子集交给若干爬虫节点并行抓取，以加快爬虫系统的整体获取速率，作为对传统方法的有效优化。将网站划分算法应用于样本数据集，验证其有效性。

关键词: 垂直搜索引擎, 任务分配, 网站划分, 爬虫

Abstract: In allusion to the problem of traditional search engines’ task allocating methods, a new fine-grained method called Web site partition is presented, which is as an effective optimization of the traditional method adopted by vertical search engines. This method divides large-scale Web sites into a number of smaller subsets, so that several crawlers can parallel crawl each subset in order to accelerate the overall downloading progress. The proposed algorithm is proved to be effective against the sample data sets.

Key words: vertical search engine, task allocation, Web site partition, crawler

中图分类号: 

• N945