| 分布式网络爬虫的设计与实现 |
| |
| 引用本文： | 吴黎兵,柯亚林,何炎祥,刘楠.分布式网络爬虫的设计与实现[J].计算机应用与软件,2011,28(11). |
| |
| 作者姓名： | 吴黎兵 柯亚林 何炎祥 刘楠 |
| |
| 作者单位： | 1. 武汉大学计算机学院 湖北武汉430072;信息网络安全公安部重点实验室 上海201204
2. 信息网络安全公安部重点实验室 上海201204 |
| |
| 基金项目： | 信息网络安全公安部重点实验室开放课题(201007001); 湖北省自然科学基金(ZRY1496); 中央高校基本科研业务费专项资金(6081013) |
| |
| 摘 要： | 提出一种可部署于单一网域及多网域间的分布式爬虫DSpider。DSpider能够通过调整节点规模和连接超时阈值,有效部署于LAN和WAN两种网络环境中。首先简要介绍了DSpider的系统结构,然后详细分析了DSpider的任务调度策略,并且在实验中将DSpider爬虫部署在LAN和WAN两种环境中的不同性能作了详细的分析。
|
| 关 键 词： | DSpider 系统架构 节点协同 Web划分 |
DESIGN AND IMPLEMENTATION OF A NOVEL DISTRIBUTED WEB CRAWLER |
| |
| Wu Libing,Ke Yalin,He Yanxiang,Liu Nan.DESIGN AND IMPLEMENTATION OF A NOVEL DISTRIBUTED WEB CRAWLER[J].Computer Applications and Software,2011,28(11). |
| |
| Authors: | Wu Libing Ke Yalin He Yanxiang Liu Nan |
| |
| Affiliation: | Wu Libing~(1,2) Ke Yalin~(1,2*) He Yanxiang~(1,2) Liu Nan~2 1(School of Computer,Wuhan University,Wuhan 430072,Hubei,China) 2(Key Lab of Information Network Security,Ministry of Public Security,Shanghai 201204,China) |
| |
| Abstract: | In this article,a novel distributed web crawler DSpider is presented.DSpider can be deployed in single network domain and among multiple network domains,by adjusting its number of nodes and the threshold of connection timeout,it can also be effectively deployed in two network environments of both LAN and WAN.In the article,firstly the system architecture of DSpider is introduced briefly.Then the task scheduling strategy of DSpider is elaborately analysed.The article also gives a report of the experiment in ... |
| |
| Keywords: | DSpider System architecture Node collaboration Web partition |
| 本文献已被 CNKI 万方数据 等数据库收录！ |
|
