| 网络爬虫效率瓶颈的分析与解决方案 |
| |
| 引用本文： | 尹江,尹治本,黄洪.网络爬虫效率瓶颈的分析与解决方案[J].计算机应用,2008,28(5):1114-1116. |
| |
| 作者姓名： | 尹江 尹治本 黄洪 |
| |
| 作者单位： | 西南交通大学,信息科学与技术学院,成都,610031 |
| |
| 摘 要： | 网络爬虫的效率，直接关系到搜索引擎系统为用户提的供服务质量。如何设计高效、快速的网络爬虫，成为目前网络爬虫研究的热点。要提高网络爬虫的爬行效率，除了需要改进网络爬虫的爬行策略之外，还需要优化网络爬自身的设计，改进网络爬虫自身的结构，消除效率瓶颈。通过对网络爬虫结构、应用环境以及用户要求的分析，提出一个通用网络爬虫的改进设计方案，并通过实验得到较好的测试结果。
|
| 关 键 词： | 爬行策略 套接字 多线程 网络爬虫 |
| 文章编号： | 1001-9081(2008)05-1114-03 |
| 收稿时间： | 2007-11-12 |
| 修稿时间： | 2007年11月12 |
Efficiency bottlenecks analysis and solution of Web crawler |
| |
| YIN Jiang,YIN Zhi-ben,HUANG Hong.Efficiency bottlenecks analysis and solution of Web crawler[J].journal of Computer Applications,2008,28(5):1114-1116. |
| |
| Authors: | YIN Jiang YIN Zhi-ben HUANG Hong |
| |
| Affiliation: | YIN Jiang,YIN Zhi-ben,HUANG Hong(School of Information Science , Technology,Southwest Jiaotong University,Chengdu Sichuan 610031,China) |
| |
| Abstract: | The efficiency of a web crawler determines the quality of services a web searching system offers to its users. How to design a more efficient and faster web crawler is becoming a hot issue in the research of web crawler. In order to raise the crawling efficiency of a web crawler, the crawling strategy needs to be reformed. Besides, the design of the web crawler system has to be optimized and its structure also needs to be improved to eliminate bottlenecks. In this paper, an improved scheme of designing a general web crawler was presented through analyzing crawler's structure, application environment and user requirement, and the preferable testing result has proven better efficiency it has. |
| |
| Keywords: | crawl strategy socket multi-threaded web crawler |
| 本文献已被 CNKI 维普 万方数据 等数据库收录！ |
| 点击此处可从《计算机应用》浏览原始摘要信息 |
|
点击此处可从《计算机应用》下载全文 |
