面向Web的信息收集工具的设计与开发

随着互联网的发展以及网上信息的日益丰富 ,传统的信息处理已经延伸到互联网领域。在对互联网上的信息进行处理时 ,常常要将分布在互联网各处的Web页面下载到本地供进一步处理 ;这便是所讨论的Web页面收集工具的核心功能。该页面收集系统在综合使用Web页面间的链接关系和页面内容的基础上 ,增加了多层次的页面过滤模块 ,可用来收集特定领域内的Web页面 ;同时可采用多机并行收集的方法提高页面收集的效率 ;采用大型数据库存放元收集信息 ,并对收集到的页面进行压缩 ,能够支持海量数据的收集 ;动态更新机制的实施使得下载到本地的页面信息能够得到及时的更新。

Design and Implementation of a Web Page-gathering Tool

With the growth of Internet and the fact that information on Web are becoming abundant , Internet has become new stage of traditional information processing. Before processing these web information,people often download the distributed web information to local storage for additional processing,which is the core function of the information-gathering system described in this paper. This system makes use of the links between pages and content of these pages to gather needed information. It can support specific information gathering using a multiple-grade filter. It can also use multiple machines to boost the gathering efficiency. It supports large-scale information gathering , using large-scale database to store the meta information during gathering process and compressing downloaded pages. It can timely update local web copies using dynamic updating mechanism.