对象存储系统中自适应的元数据负载均衡机制

面向对象的存储系统在研究、工程以及服务领域均得到了广泛的应用.在面向对象的存储系统中,元数据的负载均衡对于提高整个系统的I/O性能具有重要的作用.现有的元数据负载均衡策略不能动态地平衡元数据的访问负载,而且自适应性以及容错特性有待提高.提出了一种自适应的分布式元数据负载均衡机制(adaptabledistributed load balancing of metadata,简称ADMLB),包含基本的负载均衡算法和分布式的增量负载均衡算法.采用基本的负载均衡算法按照服务器的性能公平地分布负载,使用分布式的负载均衡算法定时地调整负载的分布.ADMLB采取分布式的方法均衡地在元数据服务器之间分布负载,根据负载的变化自适应地进行调整,具有很好的容错特性,而且用户可以高效地定位元数据服务器.     

一种分布式元数据的动态管理系统

为得到有效的元数据分布,获得多元数据服务器的负载均衡,提出一种分布式元数据的动态管理系统。利用负载均衡算法选择合适热度的子树,通过子树迁移策略将选定的子树迁移到合适的元数据服务器上进行管理,采用子树复制策略降低元数据服务器负载。实验结果证明,该系统能实现元数据的均匀分布。     

面向大数据分布式存储的动态负载均衡算法

随着大数据时代的到来,分布式存储技术应运而生。目前主流大数据技术Hadoop的HDFS分布式存储系统的元数据存储架构上一直存在可扩展性差和写延迟高等问题,其在官方2.0版本中针对可扩展性的解决方案(Fe-deration)仍不完美,仅解决了原有HDFS扩展性的问题,在元数据分配的问题上没有考虑NameNode的异构性能差异,也未解决NameNode集群动态负载均衡的问题。针对该情况,提出了一种动态负载均衡的分布NameNode算法,通过元数据多副本异构节点的动态适应性备份,使元数据在考虑节点性能及负载的情况下实现了动态分布,保证了元数据服务器集群的性能;同时结合缓存策略及自动恢复机制,提高了元数据的读写性及可用性。该算法在试验验证中达到了较为理想的效果。     

分布式文件系统元数据服务的负载均衡框架

请求负载均衡,是分布式文件系统元数据管理需要面对的核心问题.以最大化元数据服务器集群吞吐量为目标,在已有元数据管理层之上设计实现了一种分布式缓存框架,专门管理热点元数据,均衡不断变化的负载.与已有的元数据负载均衡架构相比,这种两层的负载均衡架构灵活度更高,对负载的感知能力更强,并且避免了热点元数据重新分布、迁移引起的元数据命名空间结构被破坏的情况.经观察分析,元数据尺寸小、数量大,预取错误元数据带来的代价远远小于预取错误数据带来的代价.针对元数据的以上鲜明特点,提出一种元数据预取策略和基于预取机制的元数据缓存替换算法,加强了上述分布式缓存层的性能,这种两层的元数据负载均衡框架同时考虑了缓存一致性的问题.最后,在一个真实的分布式文件系统中验证了框架及方法的有效性.     

基于固态硬盘的云存储分布式缓存策略

为满足海量数据存储的需求,提出一种基于低功耗、高性能固态硬盘的云存储系统分布式缓存策略.该策略对不同存储介质的硬盘虚拟化,将热点访问数据的缓存与存储相结合,实现在不同存储介质之间的热点数据迁移,解决热点元数据的访问一致性与存储服务器的动态负载均衡问题.工作负载压力测试结果表明,该策略可使云存储系统的读峰值速率最高提升约86％,并且能提高存储服务器的吞吐量.     

异构分布式环境下DNS动态负载均衡模型研究

DNS负载均衡是服务器群集负载均衡策略的典型应用方案之一。分析了目前DNS负载均衡的现状和异构分布式系统的特征，提出一种自适应生存期的动态调度负载均衡模型，并且描述了该模型的实现策略。     

一种新型分布式元数据管理策略

高效、可扩展的元数据管理系统是提高分布式存储系统整体性能的关键. 传统的元数据分配策略会导致元数据负载不均衡,以及在多进程资源抢占的情况下,会存在响应处理用户请求效率不高,存储文件数目受限等问题. 上述问题在高并发、低延迟的数据存储需求中尤为突出. 提出了一个基于一致性Hash与目录树的元数据管理策略,并实现了相应的分布式元数据管理系统：利用负载均衡算法,对元数据进行迁移,保证了粗粒度负载信息收集,细粒度调整的均衡策略. 多项实验的结果表明,该策略能实现元数据负载均衡,降低用户请求处理延迟,提高分布式系统的可扩展性和可用性.     

云环境下的数据库扩展策略的设计

针对云环境下的应用系统规模越来越庞大的问题,提出了一种扩展性较好的数据库服务器扩展模型。该模型架构分为三个层次：逻辑SQL处理层、DA和CP层、物理数据库层。采用了读/写分离策略、数据库复制、负载均衡策略、服务器群集策略等技术,提出基于虚拟节点的加权一致性哈希负载均衡算法,根据物理节点的性能权值计算分配的虚拟节点数。通过仿真实验表明,该模型在负载均衡的性能上具有优势,在数据库层具有较好的扩展性。     

基于强挂起弱预测机制的负载均衡模型研究

大数据时代的快速发展和大数据战略的明确提出,使得Web服务器集群将面临更加复杂和严峻的负载挑战。传统的负载均衡算法存在着明显的局限性。提出了一种基于强挂起弱预测机制的负载均衡模型,该模型利用强挂起机制和基于层次分析的三次指数平滑预测算法进行负载均衡动态调度。实验结果表明该模型在系统瞬时性能异常、高并发和重负载交互情况下的负载均衡效果优于传统负载均衡算法。     

一种元数据服务器集群的负载均衡算法

在基于对象的存储系统中,元数据访问非常频繁,大规模存储系统中元数据的访问是潜在的系统性能瓶颈.元数据服务器集群中必须负载均衡,以防某个元数据服务器成为存储系统访问的瓶颈.现有文章中很少有研究元数据服务器集群的负载均衡的文章.本文中采用元数据请求的响应时间来衡量一个元数据服务器的负载情况,首先从映射算法上实现静态负载均衡,并针对元数据热度差别大而引起的负载不均衡引入动态负载均衡,通过仿真结果显示其有效性.     

Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems

Big data is an emerging term in the storage industry, and it is data analytics on big storage, i.e., Cloud-scale storage. In Cloud-scale (or EB-scale) file systems, load balancing in request workloads across a metadata server cluster is critical for avoiding performance bottlenecks and improving quality of services.Many good approaches have been proposed for load balancing in distributed file systems. Some of them pay attention to global namespace balancing, making metadata distribution across metadata servers as uniform as possible. However, they do not work well in skew request distributions, which impair load balancing but simultaneously increase the effectiveness of caching and replication. In this paper, we propose Cloud Cache (C²), an adaptive and scalable load balancing scheme for metadata server cluster in EB-scale file systems. It combines adaptive cache diffusion and replication scheme to cope with the request load balancing problem, and it can be integrated into existing distributed metadata management approaches to efficiently improve their load balancing performance. C² runs as follows: 1) to run adaptive cache diffusion first, if a node is overloaded, loadshedding will be used; otherwise, load-stealing will be used; and 2) to run adaptive replication scheme second, if there is a very popular metadata item (or at least two items) causing a node be overloaded, adaptive replication scheme will be used, in which the very popular item is not split into several nodes using adaptive cache diffusion because of its knapsack property. By conducting performance evaluation in trace-driven simulations, experimental results demonstrate the efficiency and scalability of C².     

Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal

In this Exa byte scale era, data increases at an exponential rate. This is in turn generating a massive amount of metadata in the file system. Hadoop is the most widely used framework to deal with big data. Due to this growth of huge amount of metadata, however, the efficiency of Hadoop is questioned numerous times by many researchers. Therefore, it is essential to create an efficient and scalable metadata management for Hadoop. Hash-based mapping and subtree partitioning are suitable in distributed metadata management schemes. Subtree partitioning does not uniformly distribute workload among the metadata servers, and metadata needs to be migrated to keep the load roughly balanced. Hash-based mapping suffers from a constraint on the locality of metadata, though it uniformly distributes the load among NameNodes, which are the metadata servers of Hadoop. In this paper, we present a circular metadata management mechanism named dynamic circular metadata splitting (DCMS). DCMS preserves metadata locality using consistent hashing and locality-preserving hashing, keeps replicated metadata for excellent reliability, and dynamically distributes metadata among the NameNodes to keep load balancing. NameNode is a centralized heart of the Hadoop. Keeping the directory tree of all files, failure of which causes the single point of failure (SPOF). DCMS removes Hadoop’s SPOF and provides an efficient and scalable metadata management. The new framework is named ‘Dr. Hadoop’ after the name of the authors.     

蓝鲸分布式文件系统的分布式分层资源管理模型

为了高效地管理海量分布式存储资源,蓝鲸分布式文件系统抛弃了传统的集中式资源管理方式。实现了分布式分层资源管理模型．该模型可以管理多个存储服务器,还能支持多个元数据服务器组成的集群进行分布式元数据处理,支持各种元数据和数据的负载平衡策略．同时,该模型中的带外数据传输功能克服了系统的性能瓶颈。提高了系统支持并发访问的能力．理论分析和实际测试结果都表明此模型能够满足多种不同的需求,提供很好的性能和良好的扩展性．     

Adaptive Server Load Balancing in SDN Using PID Neural Network Controller

Web service applications are increasing tremendously in support of high-level businesses. There must be a need of better server load balancing mechanism for improving the performance of web services in business. Though many load balancing methods exist, there is still a need for sophisticated load balancing mechanism for not letting the clients to get frustrated. In this work, the server with minimum response time and the server having less traffic volume were selected for the aimed server to process the forthcoming requests. The Servers are probed with adaptive control of time with two thresholds L and U to indicate the status of server load in terms of response time difference as low, medium and high load by the load balancing application. Fetching the real time responses of entire servers in the server farm is a key component of this intelligent Load balancing system. Many Load Balancing schemes are based on the graded thresholds, because the exact information about the network flux is difficult to obtain. Using two thresholds L and U, it is possible to indicate the load on particular server as low, medium or high depending on the Maximum response time difference of the servers present in the server farm which is below L, between L and U or above U respectively. However, the existing works of load balancing in the server farm incorporate fixed time to measure real time response time, which in general are not optimal for all traffic conditions. Therefore, an algorithm based on Proportional Integration and Derivative neural network controller was designed with two thresholds for tuning the timing to probe the server for near optimal performance. The emulation results has shown a significant gain in the performance by tuning the threshold time. In addition to that, tuning algorithm is implemented in conjunction with Load Balancing scheme which does not tune the fixed time slots.     

Dynamic storage and access load balancing for answering range queries in peer-to-peer networks

Recently, many applications have used Peer-to-Peer (P2P) systems to overcome the current problems with client/server systems such as non-scalability, high bandwidth requirement and single point of failure. In this paper, we propose an efficient scheme to support efficient range query processing over structured P2P systems, while balancing both the storage load and access load. The paper proposes a rotating token scheme to balance the storage load by placing joining nodes in appropriate locations in the identifier space to share loads with already overloaded nodes. Then, to support range queries, we utilize an order-preserving mapping function to map keys to nodes in order preserving way and without hashing. This may result in an access load imbalance due to non-uniform distribution of keys in the identifier space. Thus, we propose an adaptive replication scheme to relieve overloaded nodes by shedding some load on other nodes to balance the access load. We derive a formula for estimating the overhead of the proposed adaptive replication scheme. In this study, we carry simulation experiments with synthetic data to measure the performance of the proposed schemes. Our simulation experiments show significant gains in both storage load balancing and access load balancing.     

基于移动Agent的双层负载均衡策略

提出了一种适用于Web集群的负载均衡策略,策略采用两次转发结合动态门限调整的双层均衡机制,将移动Agent应用于各子策略中,提高了系统调度决策的实时性,从分组和服务器节点,两个层次上实现了负载均衡,提高了服务器的cache命中率。同时,该策略还大幅度减少了均衡过程中的节点间交互,较好解决了额外通信的问题。     

Exploration on the Load Balancing Technique for Platform of Internet of Things

In recent years, the Internet of Things technology has developed rapidly, and smart Internet of Things devices have also been widely popularized. A large amount of data is generated every moment. Now we are in the era of big data in the Internet of Things. The rapid growth of massive data has brought great challenges to storage technology, which cannot be well coped with by traditional storage technology. The demand for massive data storage has given birth to cloud storage technology. Load balancing technology plays an important role in improving the performance and resource utilization of cloud storage systems. Therefore, it is of great practical significance to study how to improve the performance and resource utilization of cloud storage systems through load balancing technology. On the basis of studying the read strategy of Swift, this article proposes a reread strategy based on load balancing of storage resources to solve the problem of unbalanced read load between interruptions caused by random data copying in Swift. The storage asynchronously tracks the I/O conversion to select the storage with the smallest load for asynchronous reading. The experimental results indicate that the proposed strategy can achieve a better load balancing state in terms of storage I/O utilization and CPU utilization than the random read strategy index of Swift.