| 采用多种策略的分布式Web Spider |
| |
| 引用本文： | 陈炎龙,段红玉.采用多种策略的分布式Web Spider[J].计算机与数字工程,2012,40(9):63-65. |
| |
| 作者姓名： | 陈炎龙 段红玉 |
| |
| 作者单位： | 郑州牧业工程高等专科学校信息工程系 郑州450011 |
| |
| 摘 要： | 针对日益突出的网页访问问题,设计了一种新型分布式Web Spider.该分布式Web Spider采用中央控制节点来协调各个web spider的行为,利用宽度优先搜索来获得高质量的网页,通过对DNS缓存来提高访问web server的速度,以增加并行线程数量的方式来增加网页下载速度,并能动态地加入web spider节点和子中央控制节点,具有很强的灵活性和扩张能力.实验结果表明该分布式Web Spider作为搜索引擎的前端能够快速有效地下载网页,具有较好的性能.
|
| 关 键 词： | 中央控制节点 宽度优先搜索 线程 搜索引擎 |
New Distributed Web Spider by Applying Many Optimization Strategies |
| |
| CHEN Yanlong
,
DUAN Hongyu.New Distributed Web Spider by Applying Many Optimization Strategies[J].Computer and Digital Engineering,2012,40(9):63-65. |
| |
| Authors: | CHEN Yanlong DUAN Hongyu |
| |
| Affiliation: | (Department of Information Engineering,Zhengzhou College of Animal Husbandry Engineering,Zhengzhou 450011) |
| |
| Abstract: | For the increasingly prominent web access problems,A New Distributed Web Spider(NDWS) was proposed.NDWS uses central control node to coordinate actions of all web spiders,employs Breadth-First search to obtain high-quality web pages,caches DNS to improve speed of access to web server,increases number of concurrent threads to increase download speed of web pages.Meanwhile,NDWS also can dynamically add web spider node and sub-central-control-node so that NDWS has strong flexibility and expansion capability.Experimental results show that as a front-end of search engine,NDWS can quickly and efficiently download web pages,and has better performance. |
| |
| Keywords: | central control node breadth-first search thread search engine |
| 本文献已被 CNKI 维普 万方数据 等数据库收录！ |
|
