基于DHT的连续查询的设计与实现

DHT(分布式哈希表,Distributed Hashing Table)具有良好的可靠性和稳定性,Chord一种是基于DHT的典型路由管理策略,提供具有可扩展性和高效的服务。查询是P2P数据管理技术的一个重要研究内容,连续查询(continuous queues,简称CQ)用来监视数据流的变化,能够将被动的网络结构变成主动的网络结构,在信息监视等领域有广泛的应用前景。如何在P2P网络结构中开展有效的连续查询,实现高效的数据管理,是一个具有挑战性的问题。设计和分析一种经过改进的基于DHT的连续查询,采用super-peer管理查询,利用Chord组织节点和查询,备份节点能保证查询不丢失,负载平衡策略能很好保持系统的稳定性。     

基于语义查询的P2P路由模型研究

针对结构化P2P网络路由模型中存在难以有效支持内容定位的问题,提出基于语义查询的P2P路由模型.利用概念向量来描述文档、节点、簇和查询请求;通过节点概念向量间的相似度聚成不同层次的语义簇,并采用基于概念相似度计算进行路由选取和转发的路由策略;利用节点的异构性,让性能好的超级节点承担更多的工作量,并引入服务节点便于快速导航语义簇中的语义类别.实验结果表明,该路由模型能有效提高系统的搜索效率.     

TBSN:一种基于分类层次的P2P网络

构建语义覆盖网络是支持基于语义的搜索、提高P2P网络查找性能和可扩展性的重要手段.现有的基于分类层次的P2P语义覆盖网络并不能充分利用分类层次所包含的语义信息,为此提出了一种基于分类层次的P2P网络(TBSN),该网络充分考虑遵循某种分类层次的数据源的特点,能够利用分类层次所包含的语义信息,动态地将peer节点聚簇成不同的语义簇,语义簇之间根据语义关系建立相应的语义路由,从而形成了一种基于语义的P2P网络.每个语义簇由一个super-peer节点和多个peer节点组成,仅负责回答其语义子空间上的查询.查询根据语义首先被路由到可能有结果的语义簇中,然后转发给相关的peer节点,从而减少查询所涉及的节点和消息的数量,提高了网络的性能.实验表明,TBSN网络在查找性能和开销之间取得了良好的平衡,同时较好地保持了语义簇之间的负载平衡和簇中数据之间的语义关系.     

无线传感器网络中基于聚簇结构的Skyline查询方法

现有的基于单服务器的Skyline查询算法已经不能很好地应用于无线传感器网络这类分布式多跳自组织网络中。基于聚簇结构的Skyline查询算法就是针对 这类特定的网络结构而提出的。该算法采用基于聚簇的路由结构,为了减少Skyline查询处理过程中传感器节点的通信开销,挑选具有最大支配力的数据元组作为全局过滤元组来过滤不满足Skyline条件的数据。同时,在Skyline查询处理过程中引入滑动窗口机制,该机制也能有效地降低通信开销。大量的仿真实验结果显示,所提Skyline查询算法在确保能耗的基础上仍然具有很好的性能。     

Super-Peer网络中基于语义的分簇算法研究

在P2P网络中,利用共享数据的语义信息,将网络划分成不同的语义簇是提高网络查找性能、增强网络可扩展性的有效手段.然而现有的基于分类层次的语义分簇方法较少考虑簇之间的负载平衡问题,这必然会影响网络的性能.为此本文提出了两种针对分类层次语义空间的自组织语义分簇算法,即语义优先分簇算法SFCA和负载平衡优先分簇算法LBFCA,这两种算法能够根据网络的负载动态的将网络划分成不同的语义簇,并很好的保持了簇中数据的语义关系和簇之间的负载平衡.实验表明这两种分簇算法具有良好的性能和可扩展性.     

一种基于聚类的对等网络图像搜索机制

将基于内容的图像检索应用到P2P网络中,提出了一种基于“类簇”的P2P网络信息搜索机制。结合聚类方法和Small world原理,将相似的图像文件聚集到同一个“类簇”中,不同簇之间建立远距离连接;查询时先定位到所属的簇,在簇内广播查询,并根据查询记录更新节点路由表。仿真实验表明该搜索机制具有良好的搜索性能。     

基于拥塞控制的非结构化P2P网络的路由负载均衡策略

非结构化P2P(unstructured peer-to-peer network)对等网络中的节点资源定位的路由查询是对等网络研究中的一个主要难题,特别是当网络中客户端节点由于其频繁加入、离开导致网络结构动态变化所带来的资源查询难题.提出了一种新的基于拥塞控制的路由查询方法来实现动态网络下的资源查询.该方法分两部分实现:首先是网络资源的分组与节点重连策略.该策略使得具有同等资源的节点相互连接,并周期性地调整节点上的节点连接数量以减少同组资源节点上的负载.通过以上策略,使得网络的拓扑结构自动地从随机网络结构进化到以资源组为单位的聚类网络,从而使得网络中形成网络资源组间的查询负载均衡.另一方面,组内的节点之间的路由负载均衡是通过节点间协同学习实现的.采用协同Q-学习方法,所研究的方法不仅从节点上学习其处理能力、连接数和资源的个数等参数,还将节点的拥塞状态作为协同Q-学习的重要参数,并建立模型.通过这种技术,同一组节点上的资源查询被有目的地引导,以避开那些组内拥塞的节点,从而最终实现资源组内节点之间的查询均衡.仿真实验结果表明,相比常用的random walk资源查找方法,该研究所实现的资源定位方法能够更迅速地实现网络的资源查询.仿真结果还表明,相比random walk方法,所提出的方法在网络高强度查询和网络节点动态加入和退出的情况下进行查询具有更高的鲁棒性和适应性.     

面向无线自组网的交叠分簇网络结构及性能分析

无线自组网常常采用分簇网络结构来改善网络性能,并且大都是非交叠分簇结构.首先,比较了交叠分簇结构和非交叠分簇结构的优缺点.然后,重点对交叠分簇网络结构的特性进行了研究,特别考虑了相邻簇之间的高效通信问题.最后,通过模拟实验分析了交叠分簇策略下几种典型分簇算法的性能,验证了算法的有效性.     

一种P2P环境下基于查询日志的查询路由策略

在P2P文件共享系统里,通过文件描述可以描述文件的集合并且可以用它来决定查询的路由。但在这种模型下,会出现虚假匹配的现象,从而导致网络带宽和计算资源的浪费。文章提出了一种基于查询命中日志方法来较精确地划分网络结点文件描述的策略来减小虚假同现的发生,来提高查询路由的效率。     

采用元组聚类的增量式数据分区方法

数据分区是提升数据库可扩展能力的有效方法。在事务查询密集的系统中,合理的分区策略可减少分布式事务查询数量,并提高事务查询响应速度。提出了一种基于元组聚类的增量式分区方法,通过将元组聚簇和采用分区感知的数据筛选策略来降低算法的复杂度。首先依据时间窗口模型聚类元组,并构建簇节点图,然后利用分区感知策略对图进行删减,最后采用图划分算法对图进行子图划分来得到分区。与现有方法相比,该方法减少了分区响应时间,保证了较少的分布式事务数量,并提高了分区事务查询速度。     

A Framework for Supporting Tree-Like Indexes on the Chord Overlay

With the explosive growth of data, to support efficient data management including queries and updates, the database system is expected to provide tree-like indexes, such as R-tree, M-tree, B＋-tree, according to different types of data. In the distributed environment, the indexes have to be scattered across the compute nodes to improve reliability and scalability. Indexes can speed up queries, but they incur maintenance cost when updates occur. In the distributed environment, each compute node maintains a subset of an index tree, so keeping the communication cost small is more crucial, or else it occupies lots of network bandwidth and the scalability and availability of the database system are affected. Further, to achieve the reliability and scalability for queries, several replicas of the index are needed, but keeping the replicas consistent is not straightforward. In this paper, we propose a framework supporting tree-like indexes, based on Chord overlay, which is a popular P2P structure. The framework dynamically tunes the number of replicas of index to balance the query cost and the update cost. Several techniques are designed to improve the efficiency of updates without the cost of performance of the queries. We implement M-tree and R-tree in our framework, and extensive experiments on real- life and synthetic datasets verify the efficiency and scalability of our framework.     

A flabellate overlay network for multi-attribute search

Peer-to-peer (P2P) technology provides a popular way of distributing resources, sharing, and locating in a large-scale distributed environment. However, most of the current existing P2P systems only support queries over a single resource attribute, such as file name. The current multiple resource attribute search methods often encounter high maintenance cost and lack of resilience to the highly dynamic environment of P2P networks. In this paper, we propose a Flabellate overlAy Network (FAN), a scalable and structured underlying P2P overlay supporting resource queries over multi-dimensional attributes. In FAN, the resources are mapped into a multi-dimensional Cartesian space based on the consistent hash values of the resource attributes. The mapping space is divided into non-overlapping and continuous subspaces based on the peer’s distance. This paper presents strategies for managing the extended adjacent subspaces, which is crucial to network maintenance and resource search in FAN. The algorithms of a basic resource search and range query over FAN are also presented in this paper. To alleviate the load of the hot nodes, a virtual replica network (VRN) consisting of the nodes with the same replicates is proposed for replicating popular resources adaptively. The queries can be forwarded from the heavily loaded nodes to the lightly loaded ones through VRN. Theoretical analysis and experimental results show that FAN has a higher routing efficiency and lower network maintenance cost over the existing multi-attribute search methods. Also, VRN efficiently balances the network load and reduces the querying delay in FAN while invoking a relatively low overhead.     

DHTJoin: processing continuous join queries using DHT networks

Continuous query processing in data stream management systems (DSMS) has received considerable attention recently. Many applications share the same need for processing data streams in a continuous fashion. For most distributed streaming applications, the centralized processing of continuous queries over distributed data is simply not viable. This paper addresses the problem of computing approximate answers to continuous join queries over distributed data streams. We present a new method, called DHTJoin, which combines hash-based placement of tuples in a Distributed Hash Table (DHT) and dissemination of queries by exploiting the embedded trees in the underlying DHT, thereby incurring little overhead. DHTJoin also deals with join attribute value skew which may hurt load balancing and result completeness. We provide a performance evaluation of DHTJoin which shows that it can achieve significant performance gains in terms of network traffic.     

High-throughput query scheduling with spatial clustering based on distributed exponential moving average

In distributed scientific query processing systems, leveraging distributed cached data is becoming more important. In such systems, a front-end query scheduler distributes queries among many application servers rather than processing queries in a few high-performance workstations. Although many query scheduling policies exist such as round-robin and load-monitoring, they are not sophisticated enough to exploit cached results as well as balance the workload. Efforts were made to improve the query processing performance using statistical methods such as exponential moving average. However, existing methods have limitations for certain query patterns: queries with hotspots, or dynamic query distributions. In this paper, we propose novel query scheduling policies that take into account both the contents of distributed caching infrastructure and the load balance among the servers. Our experiments show that the proposed query scheduling policies outperform existing policies by producing better query plans in terms of load balance and cache-hit ratio.     

Dynamic storage and access load balancing for answering range queries in peer-to-peer networks

Recently, many applications have used Peer-to-Peer (P2P) systems to overcome the current problems with client/server systems such as non-scalability, high bandwidth requirement and single point of failure. In this paper, we propose an efficient scheme to support efficient range query processing over structured P2P systems, while balancing both the storage load and access load. The paper proposes a rotating token scheme to balance the storage load by placing joining nodes in appropriate locations in the identifier space to share loads with already overloaded nodes. Then, to support range queries, we utilize an order-preserving mapping function to map keys to nodes in order preserving way and without hashing. This may result in an access load imbalance due to non-uniform distribution of keys in the identifier space. Thus, we propose an adaptive replication scheme to relieve overloaded nodes by shedding some load on other nodes to balance the access load. We derive a formula for estimating the overhead of the proposed adaptive replication scheme. In this study, we carry simulation experiments with synthetic data to measure the performance of the proposed schemes. Our simulation experiments show significant gains in both storage load balancing and access load balancing.     

Hadoop环境下基于数据本地化的Reduce任务调度策略

在MapReduce模型任务处理过程中,当Reduce任务开始执行,远程拉取Map阶段的输出数据时,会消耗大量的网络带宽,甚至会出现网络瓶颈问题。本文提出基于数据本地化和负载均衡的任务分配策略。该策略中用户首先设置采样数据量M,在Map阶段对前M个数据块进行采样;其次根据采样结果,同时考虑数据本地化因素,将Reduce任务进行分配;然后基于负载均衡将Reduce任务进行再分配,通过任务分配,系统生成一个任务分配表;最后启动Reduce任务,系统开始数据拉取,未被采样的数据根据任务分配表进行任务分配。通过大量实验验证,基于数据本地化和负载均衡的任务分配策略,既能减少Shuffle阶段数据的传输量,又能降低网络带宽的消耗,同时可以避免出现某些节点空闲而其它节点任务量大甚至处理不了的情况,从而提高了集群处理数据的整体能力。      

Popularity-aware prefetch in P2P range caching

Unstructured peer-to-peer infrastructure has been widely employed to support large-scale distributed applications. Many of these applications, such as location-based services and multimedia content distribution, require the support of range selection queries. Under the widely-adopted query shipping protocols, the cost of query processing is affected by the number of result copies or replicas in the system. Since range queries can return results that include poorly-replicated data items, the cost of these queries is usually dominated by the retrieval cost of these data items. In this work, we propose a popularity-aware prefetch-based approach that can effectively facilitate the caching of poorly-replicated data items that are potentially requested in subsequent range queries, resulting in substantial cost savings. We prove that the performance of retrieving poorly-replicated data items is guaranteed to improve under an increasing query load. Extensive experiments show that the overall range query processing cost decreases significantly under various query load settings.     

一种基于P2P的移动对象查询框架

近年来,时空数据查询方法的研究成为人们普遍关注的研究热点.但大部分研究主要集中在集中式环境,在分布式环境下对海量时空数据进行高效的轨迹查询和窗口查询是一件十分有意义且具有挑战性的工作.设计了一种基于P2P的解决方案,提出了对移动对象运动空间进行双层划分的方法来同时支持两种查询.应用网格过滤技术有效地解决了数据频繁更新的问题.对运动空间进行高效的划分,具有比空间填充曲线方法更好的负载平衡性,同时设计了高效的Overlay--SmartChord来支持窗口查询.实验结果表明,和现有方案相比所提方案可以有效减少更新通信量,负载平衡性和路由效率有显著提高.     

Skyframe: a framework for skyline query processing in peer-to-peer systems

This paper looks at the processing of skyline queries on peer-to-peer (P2P) networks. We propose Skyframe, a framework for efficient skyline query processing in P2P systems, which addresses the challenges of quick response time, low network communication cost and query load balancing among peers. Skyframe consists of two querying methods: one is optimized for network communication while the other focuses on query response time. These methods are different in the way in which the query search space is defined. In particular, the first method uses a high dominating point that has a large dominating region to prune the search space to achieve a low cost in network communication. On the other hand, the second method relaxes the search space in order to allow parallel query processing to speed up query response. Skyframe achieves query load balancing by both query load conscious data space splitting/merging during the join/departure of nodes and dynamic load migration. We further show how to apply Skyframe to both the P2P systems supporting multi-dimensional indexing and the P2P systems supporting single-dimensional indexing. Finally, we have conducted extensive experiments on both real and synthetic data sets over two existing P2P systems: CAN (Ratnasamy in A scalable content-addressable network. In: Proceedings of SIGCOMM Conference, pp. 161–172, 2001) and BATON (Jagadish et al. in A balanced tree structure for peer-to-peer networks. In: Proceedings of VLDB Conference, pp. 661–672, 2005) to evaluate the effectiveness and scalability of Skyframe.