基于结果模式的Deep Web数据标注方法

全面准确地标注Deep Web查询结果是Deep Web数据集成的关键问题,但现有的Web数据库标注方法还不能较好地解决该问题,为此提出一种基于结果模式的Deep Web数据标注方法。首先通过结果页面解析和抽取结构化数据来完成数据预处理的工作,并在集成结果模式和待标注数据之间建立正确的语义映射,进而确定Deep Web数据的标注信息。通过对4个领域Web数据库进行实验测试,结果表明所提方法能有效地标注Deep Web查询结果数据。     

基于本体的Deep Web数据标注

借鉴语义Web领域中深度标注的思想,提出了一种对Web数据库查询结果进行语义标注的方法.为了获得完整且一致的标注结果,将领域本体作为Web数据库遵循的全局模式引入到查询结果语义标注过程中.对查询接口及查询结果特征进行详细分析,并采用查询条件重置的策略,从而确定查询结果数据的语义标记.通过对多个不同领域Web数据库的测试,在具有领域本体支持的条件下,该方法能够对Web数据库查询结果添加正确的语义标记,从而验证了该方法的有效性.     

基于语义相似度计算的Deep Web数据库查询

Deep Web蕴藏着海量信息,现有的搜索引擎很难挖掘到其中的内容。如何充分地获取Deep Web中有价值的信息成为一个难题。提出了基于语义相似度计算的Deep Web数据查询方法,该方法通过语义相似度计算作为中间件,计算出关键词和数据库属性词典对应列的相似度,从而将关键词的搜索范围限制在一个(或多个)相关领域,最后生成相应的SQL查询语句。试验证明,该方法能够有效地提高基于Deep Web的数据查询效率。     

基于最优查询的多领域deep Web爬虫*

Deep Web信息通过在网页搜索接口提交查询词获得。通用搜索引擎使用超链接爬取网页,无法索引deep Web数据。为解决此问题,介绍一种基于最优查询的deep Web爬虫,通过从聚类网页中生成最优查询,自动提交查询,最后索引查询结果。实验表明系统能自动、高效地完成多领域deep Web数据爬取。     

关于中国Deep Web的规模、分布和结构

随着Web数据库的广泛应用,Web正在加速的"深化",大量的高质量的信息隐藏在Deep Web中.基于IP采样的方法,对1,000,000个IP样本进行了分析统计,揭示了2006年初中国Deep Web的规模、分布和结构.主要结论包括有:1中国Deep Web查询接口往往位于站点浅层,94.6%的Deep Web查询接口出现在站点的前3层;2中国Deep Web大约有24,000个站点,28,000个Web数据库和74,000个查询接口;3中国Deep Web大部分是非结构化的,其中64%是非结构化的Web数据库;4中国Deep Web分布于多种不同的主题领域;5目前主要的中文搜索引擎已覆盖国内Deep Web大约二分之一的页面.     

基于搜索引擎的Deep Web数据源发现技术

随着Web数据库的广泛应用,Web正在不断“深化”。传统搜索引擎只能检索浅层网络,却不能直接索引到深层网络（Deep Web）的资源。为了有效地利用Deep Web资源,必须要对Deep Web数据进行大规模集成。其中,数据源发现是整合Deep Web资源的首要工作,能否高效地发现Deep Web站点是Deep Web数据获取的关键。提出了一种基于传统搜索引擎的Deep Web数据源发现方法,该方法通过分析返回结果来扩展查询,从而进一步提高了数据源发现的效率。实验证明该方法能得到较好的结果。     

基于本体的Deep Web查询接口集成

Deep Web中的海量信息只能通过查询接口访问获得,为了能够同时访问同一领域多个Web数据库,需要对多个Web数据库的查询接口进行集成.因此,引入本体技术,提出基于本体的Deep Web查询接口集成方法.Deep Web查询接口集成主要完成两个方面的工作:模式匹配与模式融合.模式匹配采用本体的"Bridge(桥接)"效应建立不同接口模式间的属性映射关系,以准确发现不同接口属性间的语义关联.模式融合根据模式匹配的结果,合并DeepWeb数据库查询接口集合中表示同一语义的属性,并尽可能地保持该领域查询接口的结构特征和属性顺序,以获得集成查询接口.通过实验分析,基于本体的DeepWeb查询接口集成方法不仅简化了模式匹配的复杂过程,而且很大程度上提高了模式集成的精度.因此,基于本体的Deep Web查询接口集成方法是高效可行的.     

Deep Web数据源发现与分类模型

随着Internet的发展,Web正在不断深人人们的生活.传统搜索引擎只能检索浅层网络(Surface Web),不能直接索引到深层网络(Deep Web)的资源.为了有效利用Deep Web资源,对Deep Web数据源发现并进行领域类别的划分,已成为一个非常迫切的问题.该模型首先抽取Deep Web页面查询接口的特征,构造了一个Deep Web页面过滤器,从而能够发现Deep Web的数据源,其次在对查询接12特征进行分析后,构建了一个基于KNN的分类器,并通过该分类器对新产生的Deep Web数据源进行领域分类.试验结果表明,这种模型的平均分类准确率达到86.9%,具有良好的分类效果.     

Deep Web查询接口的自动判定

传统搜索引擎仅可以索引浅层Web页面,然而在网络深处隐含着大量、高质量的信息,传统搜索引擎由于技术原因不能索引这些被称之为Deep Web的页面。由于查询接口是Deep Web的唯一入口,因此要获取Deep Web信息就需判定哪些网页表单是Deep Web查询接口。文中介绍了一种利用朴素贝叶斯分类算法自动判定网页表单是否为Deep Web查询接口的方法,并实验验证了该方法的有效性。     

基于搜索引擎的Deep Web数据源发现技术

随着Web数据库的广泛应用,Web正在不断"深化".传统搜索引擎只能检索浅层网络,却不能直接索引到深层网络(Deep Web)的资源.为了有效地利用Deep Web资源,必须要对Deep Web数据进行大规模集成.其中,数据源发现是整合DeepWeb资源的首要工作,能否高效地发现DeepWeb站点是DeepWeb数据获取的关键.提出了一种基于传统搜索引擎的Deep Web数据源发现方法,该方法通过分析返回结果来扩展查询,从而进一步提高了数据源发现的效率.实验证明该方法能得到较好的结果.     

DeepWeb信息集成系统中查询转换

随着Internet信息的迅速增长,许多Web信息已经被各种各样的可搜索在线数据库所深化,并被隐藏在Web查询接口下面。传统的搜索引擎由于技术原因不能索引这些信息——DeepWeb信息。由于DeepWeb惟一“入口点”是查询接口,为使查询接口自动产生有意义有查询,给出了DeepWeb信息集成系统框架,提出了基于数据类型的搜索驱动的用户查询转换方法,基于此设计并实现了一个针对中文DeepWeb信息集成原型系统。通过在实际DeepWeb站点上的实验证明了此方法是非常有效的。     

An efficient approach to suggesting topically related web queries using hidden topic model

Keyword-based Web search is a widely used approach for locating information on the Web. However, Web users usually suffer from the difficulties of organizing and formulating appropriate input queries due to the lack of sufficient domain knowledge, which greatly affects the search performance. An effective tool to meet the information needs of a search engine user is to suggest Web queries that are topically related to their initial inquiry. Accurately computing query-to-query similarity scores is a key to improve the quality of these suggestions. Because of the short lengths of queries, traditional pseudo-relevance or implicit-relevance based approaches expand the expression of the queries for the similarity computation. They explicitly use a search engine as a complementary source and directly extract additional features (such as terms or URLs) from the top-listed or clicked search results. In this paper, we propose a novel approach by utilizing the hidden topic as an expandable feature. This has two steps. In the offline model-learning step, a hidden topic model is trained, and for each candidate query, its posterior distribution over the hidden topic space is determined to re-express the query instead of the lexical expression. In the online query suggestion step, after inferring the topic distribution for an input query in a similar way, we then calculate the similarity between candidate queries and the input query in terms of their corresponding topic distributions; and produce a suggestion list of candidate queries based on the similarity scores. Our experimental results on two real data sets show that the hidden topic based suggestion is much more efficient than the traditional term or URL based approach, and is effective in finding topically related queries for suggestion.     

Databases deepen the Web

The Web has become the preferred medium for many database applications, such as e-commerce and digital libraries. These applications store information in huge databases that users access, query, and update through the Web. Database-driven Web sites have their own interfaces and access forms for creating HTML pages on the fly. Web database technologies define the way that these forms can connect to and retrieve data from database servers. The number of database-driven Web sites is increasing exponentially, and each site is creating pages dynamically-pages that are hard for traditional search engines to reach. Such search engines crawl and index static HTML pages; they do not send queries to Web databases. The information hidden inside Web databases is called the "deep Web" in contrast to the "surface Web" that traditional search engines access easily. We expect deep Web search engines and technologies to improve rapidly and to dramatically affect how the Web is used by providing easy access to many more information resources.     

Searching for multimedia: analysis of audio,video and image Web queries

The development of digital libraries has enhanced the integration of textual and multimedia information in many document collections. The World Wide Web provides the connectivity for many digital library users. Studies exploring the searching characteristics of Web users are an important and a growing area of research. Most Web user studies have focused on general Web searching, regardless of subject matter or format. Little research has examined how Web users search for multimedia information. Our study examines users' multimedia searching on a major Web search service. The data set examined consisted of 1,025,908 queries from 211,058 users of Excite ®, a major Web search service. From this data set, we identified and analyzed queries for audio, image, and video queries. Our findings were compared to results from general Web searching studies. Implications for the design of Web searching services and interfaces are discussed.     

Towards Deeper Understanding of the Search Interfaces of the Deep Web

Many databases have become Web-accessible through form-based search interfaces (i.e., HTML forms) that allow users to specify complex and precise queries to access the underlying databases. In general, such a Web search interface can be considered as containing an interface schema with multiple attributes and rich semantic/meta-information; however, the schema is not formally defined in HTML. Many Web applications, such as Web database integration and deep Web crawling, require the construction of the schemas. In this paper, we first propose a schema model for representing complex search interfaces, and then present a layout-expression based approach to automatically extract the logical attributes from search interfaces. We also rephrase the identification of different types of semantic information as a classification problem, and design several Bayesian classifiers to help derive semantic information from extracted attributes. A system, WISE-iExtractor, has been implemented to automatically construct the schema from any Web search interfaces. Our experimental results on real search interfaces indicate that this system is highly effective.     

Answering form-based web queries using the data-mining approach

Web users often post queries through form-based interfaces on the Web to retrieve data from the Web; however, answers to these queries are mostly computed according to keywords entered into different fields specified in a query interface, and their precision and recall could be low. The precision and recall ratios in answering this type of query can be improved by considering closely related previous queries submitted through the same interface, along with their answers. In this paper, we present an approach for enhancing the retrieval of relevant answers to a form-based Web query by adopting the data-mining approach using previous, relevant queries and their answers. Experimental results on a randomly selected set of 3,800 documents retrieved from various Web sites show that our data-mining, query-rewriting approach achieves average precision and true positive ratios on rewritten queries in the upper 80% range, whereas the average false positive ratio is less than 2.0%. Work partially done during a visit to BYU and partially supported by National Natural Science Foundation of China No. 60503036 and Fok YingTong Education Foundation No. 104027.     

Crawling ranked deep Web data sources

In the era of big data, the vast majority of the data are not from the surface Web, the Web that is interconnected by hyperlinks and indexed by most general purpose search engines. Instead, the trove of valuable data often reside in the deep Web, the Web that is hidden behind query interfaces. Since numerous applications, like data integration and vertical portals, require deep Web data, various crawling methods were developed for exhaustively harvesting a deep Web data source with the minimal (or near-minimal) cost. Most existing crawling methods assume that all the documents matched by queries are returned. In practice, data sources often return the top k matches. This makes exhaustive data harvesting difficult: highly ranked documents will be returned multiple times, while documents ranked low have small chance being returned. In this paper, we decompose this problem into two orthogonal sub-problems, i.e., query and ranking bias problems, and propose a document frequency based crawling method to overcome the ranking bias problem. The rational of our method is to use the queries whose document frequencies are within the specified range to avoid the effect of search ranking plus return limit and significantly reduce the difficulty of crawling ranked data source. The method is extensively tested on a variety of datasets and compared with two existing methods. The experimental result demonstrates that our method outperforms the two algorithms by 58 % and 90 % on average respectively.     

频繁项集在Deep Web数据源聚类中的应用

在Deep Web页面的背后隐藏着海量的可以通过结构化的查询接口进行访问的数据源。将这些数据源按所属领域进行组织划分,是DeepWeb数据集成中的一个关键步骤。已有的划分方法主要是基于查询接口模式和提交查询返回结果,存在查询接口特征难以完全抽取和提交数据库查询效率不高等问题。提出了一种结合网页文本信息,基于频繁项集的聚类方法,根据数据源查询接口所在页面的标题、关键词和提示文本,将数据源按照领域进行聚类,有效解决了传统方法中依赖查询接口特征以及文本模型的高维性问题。实验结果表明该方法是可行的,具有较高的效率。     

使用分类器自动发现特定领域的深度网入口

在深度网研究领域,通用搜索引擎(比如Google和Yahoo)具有许多不足之处:它们各自所能覆盖的数据量与整个深度网数据总量的比值小于1/3;与表层网中的情况不同,几个搜索引擎相结合所能覆盖的数据量基本没有发生变化.许多深度网站点能够提供大量高质量的信息,并且,深度网正在逐渐成为一个最重要的信息资源.提出了一个三分类器的框架,用于自动识别特定领域的深度网入口.查询接口得到以后,可以将它们进行集成,然后将一个统一的接口提交给用户以方便他们查询信息.通过8组大规模的实验,验证了所提出的方法可以准确高效地发现特定领域的深度网入口.     

基于贝叶斯网络的XML文档查询模型

目前XML查询语言及查询界面对Web用户过于复杂,该文描述了一种XML文档索引机制,在此基础上建立了一个通用的贝叶斯网络查询模型。用户只需在交互界面输入自然语言描述的查询,系统就能对其实现基于语义的构造,由它生成多个结构化查询;对这些查询建立贝叶斯网络,计算查询在给定文档下的概率,选择概率最大的前3个查询提交给系统执行。