基于领域知识抽样的深网资源采集方法

深网资源是指隐藏在HTML表单后端的Web数据库资源,这些资源主要通过表单查询的方式访问。然而,目前的网页采集技术由于采用页面超链接的方式采集资源,所以无法有效覆盖这些资源,为此,该文提出了一种基于领域知识抽样的深网资源采集方法,该方法首先利用开源目录服务创建领域属性集合,接着基于置信度函数对属性进行赋值,然后利用领域属性集合选择查询接口并生成查询接口赋值集合,最后基于贪心选择策略选择置信度最高的查询接口赋值生成查询实例进行深网采集。实验表明,该方法能够有效地实现深网资源的采集。

An Approach to Crawling the Deep Web Based on Domain Knowledge Sampling

The Deep Web refers to the Web databases content hidden behind HTML forms, which can only be accessed by performing form submissions. The current web page collection technologies can not cover these resources effectively by employing only hyperlinks. For this purpose, this paper proposes an approach to crawling the deep web based on domain knowledge sampling. Firstly, it creates a domain attributes set using open source directory services and assigns the attributes based on a confidence function; Secondly, it uses the domain attributes set to select query interface and generate assignments, and finally, it selects the assignment with the highest confidence as a query instance for deep web crawling based on greedy algorithm. Experiments show that our approach can effectively collect the deep web resources.