基于Top-k查询约束的深网增量爬取*

深网数据源的动态性、自治性和体量使第三方应用难以完全爬取所有Web数据.文中研究查询类型(仅允许Top-k查询)和查询资源约束下深网数据源爬取问题,提出基于Top-k查询约束的深网增量爬取方法,结合历史数据和领域知识,优化总体数据质量.首先基于查询树获得有效查询,利用历史数据和领域知识估计查询变化和查询代价.然后,基于估计的查询代价和数据质量,近似选择最优的查询子集最大化总体数据质量.实验表明文中方法较好地提高动态Web数据库爬取的效率和数据质量.     

面向混合类型关键词查询的非合作结构化深网数据源选择

为有效地利用深网中的资源,深网集成应运而生.为了提高深网集成的效率和返回结果的质量,数据源选择成为深网集成的关键技术.深网数据源大多数是结构化和非合作型的.当前已有的非合作结构化深网数据源选择的研究分为2类:一类是面向离散型关键词查询的源选择;另一类是面向字符型关键词查询的源选择,而未见面向混合类型关键词查询的结构化数据源选择的相关研究.基于此,将用户查询关键词分为检索型关键词和约束型关键词,基于主题词与主题词、主题词与特征词和直方图与直方图的关联特征构建了面向检索型、约束型混合关键词查询的层次化数据源摘要,有效地反映了非合作结构化深网数据源选择中检索型关键词的检索意图和约束型关键词的约束相关性,并依据此摘要给出了相应的数据源选择策略.实验结果表明,该方法在面向混合类型关键词查询的非合作结构化深网数据源选择时具有较好的记录召回率及准确率.     

Web大数据系统数据源选择

如何从数量众多的Web数据源集合中选择数量合适的数据源,使得在满足特定查询需求的前提下尽可能地减少访问数据源的数量,是Web大数据系统集成中的关键问题之一。提出了一个两阶段数据源选择方案:第一阶段通过各个数据源模式与中间模式的相似度选择与查询相关度高的数据源,通过计算依赖数据源的质量来选取质量较好的数据源;第二阶段基于最大熵理论计算数据源之间的重复率,设计实现了一个查询最小代价模型动态选择数据源算法。最后在实验平台上对算法进行了评估,实验表明该算法具有较高的效率与扩展性。     

基于主题与概率模型的非合作深网数据源选择

在深网数据集成过程中,用户希望仅检索少量数据源便能获取高质量的检索结果,因而数据源选择成为其核心技术.为满足基于相关性和多样性的集成检索需求,提出一种适合小规模抽样文档摘要的深网数据源选择方法.该方法在数据源选择过程中首先度量数据源与用户查询的相关性,然后进一步考虑候选数据源提供数据的多样性.为提升数据源相关性判别的准确性,构建了基于层次主题的数据源摘要,并在其中引入了主题内容相关性偏差概率模型,且给出了基于人工反馈的偏差概率模型构建方法以及基于概率分析的数据源相关性度量方法.为提升数据源选择结果的多样性程度,在基于层次主题的数据源摘要中建立了多样性链接有向边,并给出了数据源多样性的评价方法.最后,将基于相关性和多样性的数据源选择问题转化为一个组合优化问题,提出了基于优化函数的数据源选择策略.实验结果表明：在基于少量抽样文档进行数据源选择时,该方法具有较高的选择准确率.     

面向医学领域实体关联检索的深网数据源选择

每个领域下的深网数据源众多,如果检索领域内所有深网以获取所需的集成信息,那么工作量将十分巨大,因而数据源选择技术应运而生。医学领域实体间存在着丰富的关联关系,把相关关联信息进行有效集成可以促进人们健康生活。为提升医学领域实体关联的信息集成效率,提出了一种基于实体关联特征的数据源选择方法。基于实体关联图中的实体权重以及链接信息,构建了实体关联矩阵摘要;基于实体关联查询意图提出了数据源相关性计算方法。利用领域数据集进行了大量的实验,结果表明所提出方法准确率和召回率较高,可以为医学领域信息集成提供有效支撑。     

基于概率统计及查询质量的数据源选择策略

数据源选择策略对提高异构数据集成效率具有重要意义。综合考虑概率统计信息和数据源相对执行质量，提出一种新的数据源选择策略。概率统计信息通过分析查询日志得到，包括针对频繁主题类的数据源覆盖度和数据源集合重叠度。数据源相对执行质量包括查询价格、执行时延、数据源可靠性和用户满意度。给出基于以上标准的数据源选择贪心算法，并通过实验原型验证算法有效性。     

基于用户反馈的深网数据源选择

在因特网上,每个主题往往拥有成百上千个相关的深网数据源,从众多的数据源中找到合适的源进行数据集成变得越来越重要.传统的考虑源质量的选择方法是不考虑源主题特性的,而是根据经验选取统一的质量维度,因而在不同主题下选择准确性有较大的差异.基于此,提出基于用户反馈的深网源选择方法,依据用户反馈获取特定主题源的核心质量维度从而建立质量评价模型.选取了三个不同主题下的数据源进行了相关的验证,实验结果表明,针对不同主题下的数据源选取,该方法均具有较高的准确性且计算量较少.     

基于Choquet积分的深层网数据源选择

提出一种基于Choquet模糊积分的深层网数据源选择方法。通过在1组关键字上的查询结果评价深层网数据源的质量,评价时不仅考虑到关键字间的权重,同时还考虑多个关键字之间存在的交互作用。实验结果表明,该方法有效,且对深层网数据源的评价更接近真实。     

高效的数据源选择方式

随着关键词查询技术的飞速发展和互联网数据的迅猛增长,高效、准确的数据源选择变得十分有意义。提出了一种基于倒排列表的数据源选择方式,通过这种方式,能够在短时间内选择出相关度高的数据源,在这些数据源中执行检索,从而减少查询时间,给用户带来了更好的查询体验。从实验结果可以看出,这种方法在实际系统(例如机票查询系统)中可以得到很好的效果。为了在大规模的数据集上高效地实现相关算法,将min-hash算法应用到相似度估计中来,减少了查询空间和时间的消耗。与传统算法的比较结果表明:min-hash算法能够得到较高的精确度,并且极大地节省了算法的运行时间。     

基于世界知识的深网数据源增强分类模型

针对传统词袋方法在深网(Deep Web)数据源分类应用中的局限性,提出一种基于世界知识的Deep Web数据源增强分类模型,通过对外部知识库的主题分析,建立特征映射,构造基于领域概念的辅助分类器,丰富Deep Web查询表单的特征集合。基于Wikipedia百科知识库对真实Web数据进行分类。实验结果证明该模型有效。     

Query Processing and Optimization on the Web

The advent of the Internet and the Web and their subsequent ubiquity have brought forth opportunities to connect information sources across all types of boundaries (local, regional, organizational, etc.). Examples of such information sources include databases, XML documents, and other unstructured sources. Uniformly querying those information sources has been extensively investigated. A major challenge relates to query optimization. Indeed, querying multiple information sources scattered on the Web raises several barriers for achieving efficiency. This is due to the characteristics of Web information sources that include volatility, heterogeneity, and autonomy. Those characteristics impede a straightforward application of classical query optimization techniques. They add new dimensions to the optimization problem such as the choice of objective function, selection of relevant information sources, limited query capabilities, and unpredictable events. In this paper, we survey the current research on fundamental problems to efficiently process queries over Web data integration systems. We also outline a classification for optimization techniques and a framework for evaluating them.     

基于树合并的Deep Web查询接口集成

随着在线数据库应用的流行,整个互联网已经被迅速"深化".对于某一特定领域的deep Web,不同的站点往往会提供不同查询能力的查询接口.为了能够集成同一领域内的各个数据源,首先要解决的问题就是查询接口的集成.但是面对数量众多和结构各异的查询接口,基于人工或规则的集成方案显然无法适用.针对这种状况,提出了基于树合并的查询接口全自动集成方法.树结构可以充分体现查询接口属性之间的结构关系,通过树的合并来实现查询接口的集成.该方法不但能保持各个原始接口的结构特征,还具备良好的扩展性,以适应不断频繁更新的查询接口.最后通过多个领域的测试数据证明了本方法的有效性.     

Web 数据源选择技术

在Web数据集成的过程中,如何从大量的Web数据源集合中选择合适数量的数据源,使得在满足特定查询需求的前提下尽可能地减少所需访问的数据源数量,同时保持返回数据结果的高质量,成为Web数据集成中的一个热点问题.以近十几年的研究实践为背景,介绍Web数据源选择的研究沿革及现状,并对Web数据源选择方法进行了归类.分别讨论了基于相关性的和基于质量的数据源选择的研究动机、研究方法和研究成果等,并对相关研究的目标、关键技术、优点和缺点进行了对比分析;最后展望了Web数据源选择未来的研究方向.     

Result Merging for Structured Queries on the Deep Web with Active Relevance Weight Estimation

Data integration systems on the Deep Web offer a transparent means to query multiple data sources at once. Result merging– the generation of an overall ranked list of results from different sources in response to a query– is a key component of a data integration system. In this work we present a result merging model, called Active Relevance Weight Estimation model. Different from the existing techniques for result merging, we estimate the relevance of a data source in answering a query at query time. The relevances for a set of data sources are expressed with a (normalized) weighting scheme: the larger the weight for a data source the more relevant the source is in answering a query. We estimate the weights of a data source in each subset of the data sources involved in a training query. Because an online query may not exactly match any training query, we devise methods to obtain a subset of training queries that are related to the online query. We estimate the relevance weights of the online query from the weights of this subset of training queries. Our experiments show that our method outperforms the leading merging algorithms with comparable response time.     

Deep Web数据源聚类与分类

随着Internet信息的迅速增长,许多Web信息已经被各种各样的可搜索在线数据库所深化,并被隐藏在Web查询接口下面.传统的搜索引擎由于技术原因不能索引这些信息--Deep Web信息.本文分析了Deep Web查询接口的各种类型,研究了基于查询接口特征的数据源聚类方法和基于聚类结果的数据源分类方法,讨论了从基于规则与线性文档分类器中抽取查询探测集的规则抽取算法和Web文档数据库分类的查询探测算法.     

Exploiting location information for Web search

Most Web pages contain location information, which are usually neglected by traditional search engines. Queries combining location and textual terms are called as spatial textual Web queries. Based on the fact that traditional search engines pay little attention in the location information in Web pages, in this paper we study a framework to utilize location information for Web search. The proposed framework consists of an offline stage to extract focused locations for crawled Web pages, as well as an online ranking stage to perform location-aware ranking for search results. The focused locations of a Web page refer to the most appropriate locations associated with the Web page. In the offline stage, we extract the focused locations and keywords from Web pages and map each keyword with specific focused locations, which forms a set of <keyword, location> pairs. In the second online query processing stage, we extract keywords from the query, and computer the ranking scores based on location relevance and the location-constrained scores for each querying keyword. The experiments on various real datasets crawled from nj.gov, BBC and New York Time show that the performance of our algorithm on focused location extraction is superior to previous methods and the proposed ranking algorithm has the best performance w.r.t different spatial textual queries.     

A mobile query service for integrated access to large numbers of online semantic web data sources

From the Semantic Web’s inception, a number of concurrent initiatives have given rise to multiple segments: large semantic datasets, exposed by query endpoints; online Semantic Web documents, in the form of RDF files; and semantically annotated web content (e.g., using RDFa), semantic sources in their own right. In various mobile application scenarios, online semantic data has proven to be useful. While query endpoints are most commonly exploited, they are mainly useful to expose large semantic datasets. Alternatively, mobile RDF stores are utilized to query local semantic data, but this requires the design-time identification and replication of relevant data. Instead, we present a mobile query service that supports on-the-fly and integrated querying of semantic data, originating from a largely unused portion of the Semantic Web, comprising online RDF files and semantics embedded in annotated webpages. To that end, our solution performs dynamic identification, retrieval and caching of query-relevant semantic data. We explore several data identification and caching alternatives, and investigate the utility of source metadata in optimizing these tasks. Further, we introduce a novel cache replacement strategy, fine-tuned to the described query dataset, and include explicit support for the Open World Assumption. An extensive experimental validation evaluates the query service and its alternative components.     

Query optimization in information integration

The problem of decentralized data sharing, which is relevant to a wide range of applications, is still a source of major theoretical and practical challenges, in spite of many years of sustained research. In this paper we focus on the challenge of efficiency of query evaluation in information integration systems that use the global-as-view approach, with the objective of developing query-processing strategies that would be widely applicable and easy to implement in real-life applications. Our algorithms take into account important features of today’s data sharing applications: XML as likely interface or representation for data sources; the potential for information overlap across data sources; and the need for inter-source processing, as in joins of data across sources. The focus of this paper is on performance-related characteristics of several alternative approaches that we propose for efficient query processing in information integration, including an approach that uses materialized restructured views. We use synthetic and real-life datasets in our implementation of an information integration system shell to provide experimental results that demonstrate that our algorithms are efficient and competitive in the information integration setting. In addition, our experimental results allow us to make context-specific recommendations on selecting query-processing approaches from our proposed alternatives. As such, our approaches could form a basis for scalable query processing in information integration and interoperability in many practical settings.