A proactive low-overhead file replication scheme for structured P2P content delivery networks

File replication is a widely used technique for high performance in peer-to-peer content delivery networks. A file replication technique should be efficient and at the same time facilitates efficient file consistency maintenance. However, most traditional methods do not consider nodes’ available capacity and physical location in file replication, leading to high overhead for both file replication and consistency maintenance. This paper presents a proactive low-overhead file replication scheme, namely Plover. By making file replicas among physically close nodes based on nodes’ available capacities, Plover not only achieves high efficiency in file replication but also supports low-cost and timely consistency maintenance. It also includes an efficient file query redirection algorithm for load balancing between replica nodes. Theoretical analysis and simulation results demonstrate the effectiveness of Plover in comparison with other file replication schemes. It dramatically reduces the overhead of both file replication and consistency maintenance compared to other schemes. In addition, it yields significant improvements in reduction of overloaded nodes.     

Perspectives on optimistically replicated,peer-to-peer filing

This research proposes and tests an approach to engineering distributed file systems that are aimed at wide-scale, Internet-based use. The premise is that replication is essential to deliver performance and availability, yet the traditional conservative replica consistency algorithms do not scale to this environment. Our Ficus replicated file system uses a single-copy availability, optimistic update policy with reconciliation algorithms that reliably detect concurrent updates and automatically restore the consistency of directory replicas. The system uses the peer-to-peer model in which all machines are architectural equals but still permits configuration in a client-server arrangement where appropriate. Ficus has been used for six years at several geographically scattered installations. This paper details and evaluates the use of optimistic replica consistency, automatic update conflict detection and repair, the peer-to-peer (as opposed to client-server) interaction model, and the stackable file system architecture in the design and construction of Ficus. The paper concludes with a number of lessons learned from the experience of designing, building, measuring, and living with an optimistically replicated file system. © 1998 John Wiley & Sons, Ltd.     

PATCOM:基于分割树的无结构P2P系统一致性维护方法

无结构P2P技术逐渐被应用在新型的协同计算系统中.这些新型业务支持数据的动态更新,不仅要求副本数据的强一致性,而且要求更新数据的快速传播.高效的一致性维护方法是保证新业务顺利开展的基础.在比较分析现有的P2P系统一致性维护方法的基础上,针对无结构P2P系统,提出了一种基于分割树的一致性维护方法--PATCOM.PATCOM使用Chord协议作为组管理协议,通过不断分割由副本节点组成的Chord环,动态地建立更新消息传播树(Update Message Propagation Tree,UMPT).论文进一步从理论上分析了UMPT的平均高度、PATCOM的性能、容错能力以及算法开销,并和基于Gossip的一致性维护方法进行了比较.理论分析和仿真实验结果表明:PATCOM不仅能够快速地维护P2P系统的强一致性,而且产生的冗余更新消息少.     

P2P系统中基于副本链的一致性维护算法

提出一种无结构纯P2P的副本一致性维护算法。利用副本节点发起的第一次更新消息在P2P网络中的广播,由其他收到消息的副本节点给出响应,构建副本链。副本链建立后,更新消息在副本节点间进行传播,不再在网络中洪泛。副本链的维护通过记录首次发起更新的副本节点IP地址完成。仿真试验证明该算法是简单有效的。     

结构化P2P系统中副本一致性管理的一种新算法

结构化P2P系统通常使用数据复制来提高数据可用性,但P2P环境中的节点搅动、多节点并发更新以及恶意节点的存在也为副本的一致性管理带来了新的挑战.基于协商的算法要求节点间以全交换的方式通讯,在P2P环境中其可伸缩性不够理想.本文针对结构化P2P系统提出一种基于Quorum的副本管理算法:使用混合失效模型降低容错开销,利用DHT服务处理节点搅动,将数据存储与其元信息管理分离,使数据可靠性和数据可用性得以独立调整.模拟实验表明该算法可以明显改善系统的可伸缩性,减少系统的容错开销.     

面向海量数据的数据一致性研究

复制是实现海量数据管理的关键技术之一,多副本之间的数据一致性维护是提高分布式系统的容错能力与性能的重要保证。强一致性确保并发的修改操作不会发生冲突,但是限制了系统的可用性、连通性以及副本数量;弱一致性确保副本的最终一致,提高了系统的容错能力。本文从已有的一致性维护方法出发,结合海量数据的特点,对一致性维护过程中所涉及的更新发布、更新传播方式、更新传播内容以及更新冲突解决等几个方面进行了分析,提出了相应的解决方法。     

File replication,maintenance, and consistency management services in data grids

Data replication and consistency refer to the same data being stored in distributed sites, and kept consistent when one or more copies are modified. A good file maintenance and consistency strategy can reduce file access times and access latencies, and increase download speeds, thus reducing overall computing times. In this paper, we propose dynamic services for replicating and maintaining data in grid environments, and directing replicas to appropriate locations for use. To address a problem with the Bandwidth Hierarchy-based Replication (BHR) algorithm, a strategy for maintaining replicas dynamically, we propose the Dynamic Maintenance Service (DMS). We also propose a One-way Replica Consistency Service (ORCS) for data grid environments, a positive approach to resolving consistency maintenance issues we hope will strike a balance between improving data access performance and replica consistency. Experimental results show that our services are more efficient than other strategies.     

Fault-tolerant data management in the gaston peer-to-peer file system

Gaston is a peer-to-peer large-scale file system designed to provide a fault-tolerant and highly available file service for a virtually unlimited number of users. Data management in Gaston disseminates and stores replicas of files on multiple machines to achieve the requested level of data availability and uses a dynamic tree-topology structure to connect replication schema members. We present generic algorithms for replication schema creation and maintenance according to file user requirements and autonomous constraints that are set on individual nodes. We also show specific data object structure as well as mechanisms for secure and efficient update propagation among replicas with data consistency control. Finally, we introduce a scalable and efficient technique improving fault-tolerance of the tree-topology structure connecting replicas.     

一种基于有限编码的多副本分簇管理方法

针对大量数据副本所带来的资源管理问题,提出一种基于有限编码的多副本分簇管理方法.在该方法中,根据单副本复制产生新副本的过程对副本分级和分簇,通过定义"副本级别+副本顺序"的编码规则对划分后的副本进行编码和组织,并依据编码规则对由于副本的动态调整(增加或撤消)而引起的簇的动态变化进行有效管理.通过该方法,在大量副本之间建立局域集中、广域对等的管理模式,再结合定义的"最小更新传播时间"可以降低大量副本的一致性维护开销.讨论了方法中编码规则与副本规模之间的关系,以及副本失效和恢复时的解决方法.性能测试结果表明,该方法能够有效组织大规模的数据副本,具有较好的可扩展性,对适度的结点失效不敏感,适合更新频繁的应用.     

一种基于关键属性的优化数据一致性维护方法

针对关键属性更新的易描述、更新项较小和弱相关性三个特点。提出一种基于关键属性的优化数据一致性维护方法.在该方法中,首先分离出用户提交的更新请求中关于关键属性的更新;然后采用基于延迟.覆盖的更新传播模型进行更新传播,即基于副本间的网络延迟选择具有最大和最小网络延迟的结点转发更新,并在传播过程中记录和综合考虑更新的确定传播和不确定传播两条路径;在对关键更新冲突分类的基础上,采用更新缓冲区和更新日志两层更新协商机制并结合最新写胜出和分而治之规则,优化关键更新冲突的发现和解决:更新日志信息是方法中各种策略开展的基础,讨论了更新日志中信息管理方法以及结点失效和网络划分发生时信息的维护方法.关键属性更新的优化处理使得不会产生因为关键属性更新的延迟而降低系统基于关键属性的资源定位效率,满足面向Internet的P2P系统的要求.模拟测试结果表明,该方法在一致性维护开销、资源定位开销与资源访问开销以及鲁棒性方面均具有较好的性能.     

A novel dynamic network data replication scheme based on historical access record and proactive deletion

Data replication is becoming a popular technology in many fields such as cloud storage, Data grids and P2P systems. By replicating files to other servers/nodes, we can reduce network traffic and file access time and increase data availability to react natural and man-made disasters. However, it does not mean that more replicas can always have a better system performance. Replicas indeed decrease read access time and provide better fault-tolerance, but if we consider write access, maintaining a large number of replications will result in a huge update overhead. Hence, a trade-off between read access time and write updating cost is needed. File popularity is an important factor in making decisions about data replication. To avoid data access fluctuations, historical file popularity can be used for selecting really popular files. In this research, a dynamic data replication strategy is proposed based on two ideas. The first one employs historical access records which are useful for picking up a file to replicate. The second one is a proactive deletion method, which is applied to control the replica number to reach an optimal balance between the read access time and the write update overhead. A unified cost model is used as a means to measure and compare the performance of our data replication algorithm and other existing algorithms. The results indicate that our new algorithm performs much better than those algorithms.     

基于Paillier加密的数据多副本持有性验证方案

云存储服务中,用户将数据存储在不可信的云储存服务器上,为检查云存储中服务提供商(CSP)是否按协议完整地存储了用户的所有数据副本,提出一种 支持对数据副本进行动态操作 的基于Paillier加密的数据多副本持有性验证方案, 即DMR-PDP方案。该方案为实现多副本检查,将文件块以文件副本形式存储在云服务器上,将各副本编号与文件连接后利用Paillier密码系统生成副本文件以防止CSP各服务器的合谋攻击。利用BLS签名实现对所有副本的批量验证。将文件标志和块位置信息添加到数据块标签中,以保证本方案的安全性,支持对文件的动态更新操作。安全性分析和仿真实验结果表明,该方案在安全性、通信和计算开销方面的性能优于其他文献提出的方案,极大地提高了文件存储和验证的效率,减少了计算开销。     

Efficient and Scalable Consistency Maintenance for Heterogeneous Peer-to-Peer Systems

Consistency maintenance mechanism is necessary for the emerging Peer-to-Peer applications due to their frequent data updates. Centralized approaches suffer single point of failure, while previous decentralized approaches incur too many duplicate update messages because of locality-ignorant structures. To address this issue, we propose a scalable and efficient consistency maintenance scheme for heterogeneous P2P systems. Our scheme takes the heterogeneity nature into account and forms the replica nodes of a key into a locality-aware hierarchical structure, in which the upper layer is DHT-based and consists of powerful and stable replica nodes, while a replica node at the lower layer attaches to a physically close upper layer node. A d-ary update message propagation tree (UMPT) is dynamically built upon the upper layer for propagating the updated contents. As a result, the tree structure does not need to be maintained all the time, sav-ing a lot of cost. Through theoretical analyses and comprehensive simulations, we examine the efficiency and scalability of this design. The results show that, compared with previous designs, especially locality-ignorant ones, our approach is able to reduce the cost by about 25-67 percent.     

Replica Placement Strategies in Data Grid

Replication is a technique used in Data Grid environments that helps to reduce access latency and network bandwidth utilization. Replication also increases data availability thereby enhancing system reliability. The research addresses the problem of replication in Data Grid environment by investigating a set of highly decentralized dynamic replica placement algorithms. Replica placement algorithms are based on heuristics that consider both network latency and user requests to select the best candidate sites to place replicas. Due to dynamic nature of Grid, the candidate site holds replicas currently may not be the best sites to fetch replicas in subsequent periods. Therefore, a replica maintenance algorithm is proposed to relocate replicas to different sites if the performance metric degrades significantly. The study of our replica placement algorithms is carried out using a model of the EU Data Grid Testbed 1 [Bell et al. Comput. Appl., 17(4), 2003] sites and their associated network geometry. We validate our replica placement algorithms with total file transfer times, the number of local file accesses, and the number of remote file accesses.     

基于P2P计算的动态多副本开销模型

在多个结点上保持副本是提高P2P或网格计算环境可用性的一个有效途径。不可靠的对等点和网络环境以及用户访问模式的多样性使得难以确定多少个副本能够满足用户高可用性的需求。文章提出了一种最小开销模型来预测和动态控制副本数量。为了隐藏系统中对等点的不可靠性,在考虑多副本的同时通过一种冗余机制预测系统中的副本数量。仿真结果表明,该系统具有更好的可用性和低的副本开销。     

一种解决更新相关的更新冲突协商算法

针对数据一致性维护过程中存在的更新相关问题,本文提出一种更新冲突协商算法,利用Bloom Filter压缩技术表示更新,通过结点间的分工协作发现和解决更新相关冲突,并实现副本最终一致。模拟测试结果表明,该方法在一致性维护开销方面具有较好的性能。     

An optimized strategy for update path selection in unstructured P2P networks

With the growth of P2P file sharing systems, people are no longer satisfied with the sharing of the read-only and static files, and thus the systems with mutually writable and dynamic files have emerged, resulting in the replica inconsistency problem. To maintain the replica consistency, too many update messages need to be redundantly propagated due to the lack of the sharing of globally updated path information in the existing strategies. To address this problem, we propose an optimized strategy for update path selection, which makes the nodes share the update path information by using clone, variation and crossover operations for the update paths. We also present a repeated update strategy to cope with the churn problem so as to maintain replica consistency as far as possible even if some nodes temporarily leave the network. The simulation results show that our strategy can significantly reduce the number of the redundant update messages without lowering the message coverage, thus improving the availability of the unstructured P2P networks.     

针对负载均衡的P2P动态副本策略

对等网络系统面临的最大的问题就是负载均衡.针对P2P网络中负载均衡的问题,提出了两种基于动态副本策略的负载均衡技术:周期性副本策略和基于需求的副本策略.在周期性副本策略里,每个超级节点周期性地把文件的副本发送给请求该文件频率最高的远程超级节点.在基于需求的副本策略里,当一个超级节点发现其对某个文件的访问频率过高时,就向拥有该文件的超级节点发送文件请求,以获得该文件副本.本文分析了这两种技术的平均访问代价和副本负载代价.一系列的模拟实验证实了这两种技术的可行性及其带来的好处.     

面向Android系统的目录自适应日志模式选择机制

在写磁盘的过程中如果意外掉电或系统崩溃很有可能导致文件系统中用户数据和元数据不一致,现有文件系统主要采取写前日志或写时拷贝等一致性技术来解决该问题,但均没有考虑目录对可靠性需求的差异性。针对现有的文件自适应日志模式存在的需要逐一修改应用程序的不足,提出了面向Android系统的目录自适应的日志模式选择机制,即针对不同的目录对可靠性需求的高低等级,选择相应的强弱程度不同的日志模式,同时分配可靠性不同的存储区域,这种机制对开发者完全透明,也符合Android系统的应用特征,在可靠性需求不变的前提下最大限度地减少了一致性保证引入的额外开销。实验结果表明,修改后的文件系统可以识别文件所在目录,并根据目录选择预先定义的日志模式。     

Hierarchical data replication strategy to improve performance in cloud computing

Cloud computing environment is getting more interesting as a new trend of data management. Data replication has been widely applied to improve data access in distributed systems such as Grid and Cloud. However, due to the finite storage capacity of each site, copies that are useful for future jobs can be wastefully deleted and replaced with less valuable ones. Therefore, it is considerable to have appropriate replication strategy that can dynamically store the replicas while satisfying quality of service (QoS) requirements and storage capacity constraints. In this paper, we present a dynamic replication algorithm, named hierarchical data replication strategy (HDRS). HDRS consists of the replica creation that can adaptively increase replicas based on exponential growth or decay rate, the replica placement according to the access load and labeling technique, and finally the replica replacement based on the value of file in the future. We evaluate different dynamic data replication methods using CloudSim simulation. Experiments demonstrate that HDRS can reduce response time and bandwidth usage compared with other algorithms. It means that the HDRS can determine a popular file and replicates it to the best site. This method avoids useless replications and decreases access latency by balancing the load of sites.