基于固态硬盘的云存储分布式缓存策略

为满足海量数据存储的需求,提出一种基于低功耗、高性能固态硬盘的云存储系统分布式缓存策略.该策略对不同存储介质的硬盘虚拟化,将热点访问数据的缓存与存储相结合,实现在不同存储介质之间的热点数据迁移,解决热点元数据的访问一致性与存储服务器的动态负载均衡问题.工作负载压力测试结果表明,该策略可使云存储系统的读峰值速率最高提升约86％,并且能提高存储服务器的吞吐量.     

面向大数据分布式存储的动态负载均衡算法

随着大数据时代的到来,分布式存储技术应运而生。目前主流大数据技术Hadoop的HDFS分布式存储系统的元数据存储架构上一直存在可扩展性差和写延迟高等问题,其在官方2.0版本中针对可扩展性的解决方案(Fe-deration)仍不完美,仅解决了原有HDFS扩展性的问题,在元数据分配的问题上没有考虑NameNode的异构性能差异,也未解决NameNode集群动态负载均衡的问题。针对该情况,提出了一种动态负载均衡的分布NameNode算法,通过元数据多副本异构节点的动态适应性备份,使元数据在考虑节点性能及负载的情况下实现了动态分布,保证了元数据服务器集群的性能;同时结合缓存策略及自动恢复机制,提高了元数据的读写性及可用性。该算法在试验验证中达到了较为理想的效果。     

面向海量空间数据存储的元数据管理机制

为了有效管理海量空间数据存储的元数据,引入了一种基于一致性哈希的分布式元数据服务器管理架构,并在此基础上提出了一种元数据轮式备份策略,将经过一致性哈希算法散列后存储元数据的节点按轮转方式进行数据备份,有效缓解了元数据管理的单点问题与访问瓶颈.最后对轮式备份策略进行测试,得出最佳元数据节点个数备份方案,与单点元数据服务器相比提高了元数据的安全性,降低了访问延迟,并结合虚拟节点改善了分布式元数据服务器的负载均衡.     

一种机群文件系统的缓存模型

提出了一种新的机群文件系统缓存模型，它充分利用机群系统累积的系统资源和高速的互联网络，将文件系统元数据和内容数据分离，分别使用分布式元数据缓存和统一缓存模型进行管理。元数据缓存使用改进的广播一致性协议和LRU替换算法。内容数据统一缓存则将磁盘缓存整个文件和内存缓存文件块相结合组成一个单一映像的多层次分布协作缓存，并使用单拷贝优先LRU和向前传递调度缓存块替换算法以及一种贪心的数据预取方法。实验结果表明，这两种缓存机制结合使用能极大地提高机群文件系统的性能。     

虚拟现实环境下的分布式服务器架构策略研究

虚拟现实环境下,数据实时性和系统稳定性的高要求对服务器架构的设计和优化提出了新的挑战。针对虚拟现实环境下海量数据存储效率的提升和系统性能的优化,提出了一种新的分布式服务器架构,该架构基于分布式协调框架ZooKeeper、分布式缓存架构Redis以及MongoDB分片机制,并改进一致性Hash算法来优化Redis缓存架构,同时优化MongoDB分片的负载均衡机制。经过相应的仿真验证,该架构在虚拟现实环境下具有有效性。     

对象存储系统中自适应的元数据负载均衡机制

面向对象的存储系统在研究、工程以及服务领域均得到了广泛的应用.在面向对象的存储系统中,元数据的负载均衡对于提高整个系统的I/O性能具有重要的作用.现有的元数据负载均衡策略不能动态地平衡元数据的访问负载,而且自适应性以及容错特性有待提高.提出了一种自适应的分布式元数据负载均衡机制(adaptabledistributed load balancing of metadata,简称ADMLB),包含基本的负载均衡算法和分布式的增量负载均衡算法.采用基本的负载均衡算法按照服务器的性能公平地分布负载,使用分布式的负载均衡算法定时地调整负载的分布.ADMLB采取分布式的方法均衡地在元数据服务器之间分布负载,根据负载的变化自适应地进行调整,具有很好的容错特性,而且用户可以高效地定位元数据服务器.     

柔性微服务监控框架

微服务细化了服务的粒度,针对微服务的监控是微服务管理需要面对的核心问题,以可扩展全链路应用服务监控为目标,在已有服务管理层之上设计实现了一种柔性微服务监控框架,用于监控微服务状态和不断变化的服务负载,与已有的分布式监控架构相比,这种监控框架灵活度更高,对服务变化的感知能力更强,使用Raft算法增强了数据一致性,避免了单点故障的情况,经试验分析,代价更小,在实际项目中验证了框架及方法的有效性.     

BWMMS元数据分布信息缓存管理

BWMMS是BWFS的分布式文件系统元数据服务子系统。它充分利用系统访问负载的动态性和局部性特征，通过简单的集中决策机制管理元数据请求负载在多个元数据服务器的分布。为降低集中决策点可能的瓶颈限制，集中决策点位于元数据请求处理路径的末端。本文介绍各个元数据服务器上用来降低对后端集中决策点的压力，提高元数据访问效率的元数据分布信息缓存，并通过测试数据评估缓存命中率对后端集中决策点和元数据访问效率的影响。     

基于镜像层关联的Docker注册表缓存预取策略

随着容器技术的广泛普及,大型Docker公共注册表使用对象存储服务来解决镜像数量剧增的问题,但这种松耦合的注册表设计导致较高的延迟开销。为了增强注册表性能,提出一种基于镜像层关联的Docker注册表缓存预取策略LCPA,当注册表服务器缓存未命中时,通过分析镜像元数据文件构建镜像的存储结构,由关联度模型对存储结构计算得到相关镜像层集合,并从后端存储中主动式预取回注册表中以提高缓存命中率。经真实工作负载下收集的Docker数据集测试,实验结果表明LCPA策略比LRU、LIRS和GDFS等缓存算法提高12%~29%的平均缓存命中率,拉取镜像的平均延迟节省率提高了21.1%~49.4%。与现有的LPA预取算法相比,拉取镜像的平均缓存命中率提升25.6%。仿真实验表明该策略可以有效地利用缓存空间,大幅提升注册表的缓存命中率,并降低镜像拉取的延迟开销。     

MooseFS中chunkserver负载均衡算法研究

作为云存储的核心基础平台,分布式文件系统的重要性日益凸显.分布式文件系统中数据存储在多台计算机节点上,必然会出现负载均衡问题.首先,对MooseFS的系统架构进行了研究,然后分析了MooseFS分布式文件系统中chunkserver选择算法,研究了chunkserver算法的负载均衡性能,最后对其进行了改进.经过实验测试对比,实验结果显示改进算法能显著提高chunkserver的负载均衡性能.     

Adaptive and scalable load balancing for metadata server cluster in cloud-scale file systems

Big data is an emerging term in the storage industry, and it is data analytics on big storage, i.e., Cloud-scale storage. In Cloud-scale (or EB-scale) file systems, load balancing in request workloads across a metadata server cluster is critical for avoiding performance bottlenecks and improving quality of services.Many good approaches have been proposed for load balancing in distributed file systems. Some of them pay attention to global namespace balancing, making metadata distribution across metadata servers as uniform as possible. However, they do not work well in skew request distributions, which impair load balancing but simultaneously increase the effectiveness of caching and replication. In this paper, we propose Cloud Cache (C²), an adaptive and scalable load balancing scheme for metadata server cluster in EB-scale file systems. It combines adaptive cache diffusion and replication scheme to cope with the request load balancing problem, and it can be integrated into existing distributed metadata management approaches to efficiently improve their load balancing performance. C² runs as follows: 1) to run adaptive cache diffusion first, if a node is overloaded, loadshedding will be used; otherwise, load-stealing will be used; and 2) to run adaptive replication scheme second, if there is a very popular metadata item (or at least two items) causing a node be overloaded, adaptive replication scheme will be used, in which the very popular item is not split into several nodes using adaptive cache diffusion because of its knapsack property. By conducting performance evaluation in trace-driven simulations, experimental results demonstrate the efficiency and scalability of C².     

一种自适应文件系统元数据服务负载均衡策略

随着大数据时代的到来,全球信息存储量呈现爆发式的增长,传统的存储系统在存储性能、存储容量、数据可靠性和成本等方面存在诸多不足。近年来,以云计算平台为依托的存储技术得到了飞速的发展,成为了处理海量数据的重要工具。本文针对分布式文件系统元数据管理的问题,提出了一种自适应元数据服务负载均衡策略。该策略主要包括以下三点内容：第一,介绍了一种实时的元数据服务器的性能评价模型;第二,提出了一种基于服务器负载变化的检测周期自适应调整机制;第三,提出了一种基于元数据服务器性能指标的自适应负载均衡算法。实验证明了该方法的可行性,有效性和稳定性。     

DEAM: Decoupled,Expressive, Area-Efficient Metadata Cache

Chip multiprocessor presents brand new opportunities for holistic on-chip data and coherence management solutions. An intelligent protocol should be adaptive to the fine-grain accessing behavior. And in terms of storage of metadata, the size of conventional directory grows as the square of the number of processors, making it very expensive in large-scale systems. In this paper, we propose a metadata cache framework to achieve three goals: 1) reducing the latency of data access and coherence activities, 2) saving the storage of metadata, and 3) providing support for other optimization techniques. The metadata is implemented with compact structures and tracks the dynamically changing access pattern. The pattern information is used to guide the delegation and replication of decoupled data and metadata to allow fast access. We also use our metadata cache as a building block to enhance stream prefetching. Using detailed execution-driven simulation, we demonstrate that our protocol achieves an average speedup of 1.12X compared with a shared cache protocol with 1/5 of the storage of metadata.     

蓝鲸分布式文件系统的分布式分层资源管理模型

为了高效地管理海量分布式存储资源,蓝鲸分布式文件系统抛弃了传统的集中式资源管理方式。实现了分布式分层资源管理模型．该模型可以管理多个存储服务器,还能支持多个元数据服务器组成的集群进行分布式元数据处理,支持各种元数据和数据的负载平衡策略．同时,该模型中的带外数据传输功能克服了系统的性能瓶颈。提高了系统支持并发访问的能力．理论分析和实际测试结果都表明此模型能够满足多种不同的需求,提供很好的性能和良好的扩展性．     

一种结构化P2P 协议中的自适应负载均衡方法

提出一种自适应负载均衡方法,方法采用一种被动式结点负载统计方法生成局部负载视图;一种文件访问统计方法生成局部文件访问视图;当系统内结点负载存在差异时,重载结点把指向自身的逻辑链路迁移至指向局部负载视图中的轻载结点,通过减小重载结点入度和增加轻载结点入度来减小结点间负载差异;当结点的请求负载较高时,通过局部文件访问视图计算需要缓存的热点文件及目标结点,降低承载热点文件的结点请求负载.实验结果表明,在用户查询服从Zipf 分布的环境下,自适应负载均衡方法可使系统负载达到较好的均衡;缓存方法虽然在一定程度上增加了缓存和更新开销,但在一定条件下比查询消息命中缓存节省的网络开销要小,降低了系统的整体负载.     

Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal

In this Exa byte scale era, data increases at an exponential rate. This is in turn generating a massive amount of metadata in the file system. Hadoop is the most widely used framework to deal with big data. Due to this growth of huge amount of metadata, however, the efficiency of Hadoop is questioned numerous times by many researchers. Therefore, it is essential to create an efficient and scalable metadata management for Hadoop. Hash-based mapping and subtree partitioning are suitable in distributed metadata management schemes. Subtree partitioning does not uniformly distribute workload among the metadata servers, and metadata needs to be migrated to keep the load roughly balanced. Hash-based mapping suffers from a constraint on the locality of metadata, though it uniformly distributes the load among NameNodes, which are the metadata servers of Hadoop. In this paper, we present a circular metadata management mechanism named dynamic circular metadata splitting (DCMS). DCMS preserves metadata locality using consistent hashing and locality-preserving hashing, keeps replicated metadata for excellent reliability, and dynamically distributes metadata among the NameNodes to keep load balancing. NameNode is a centralized heart of the Hadoop. Keeping the directory tree of all files, failure of which causes the single point of failure (SPOF). DCMS removes Hadoop’s SPOF and provides an efficient and scalable metadata management. The new framework is named ‘Dr. Hadoop’ after the name of the authors.     

一种基于授权机制的分布式文件系统小文件访问优化策略

针对分布式文件系统应用于海量小文件访问模式时, 元数据请求过多导致系统性能下降的问题, 提出了客户端元数据缓存授权机制的解决方案. 客户端从服务器读取元数据时, 申请相应类型的授权, 服务器分析请求并决定是否授予此客户端所访问的元数据的授权. 若客户端成功获取授权, 则将其与本地缓存的元数据相关联, 作为缓冲有效性的凭证. 当再次访问本地缓存的元数据时, 若有相关授权, 则可以直接从本地获取元数据, 无需向服务器发送缓存数据的有效性验证RPC. 仿真实验表明, 文中的方法有效的降低了客户端发送元数据请求RPC的数量, 节省了宝贵的网络带宽资源, 降低了元数据服务器的负载.     

An SSD-based accelerator for directory parsing in storage systems containing massive files

Data explosion introduces new challenges to storage systems. In a file system for big data, a large number of directories and files exist, which are usually organized in a large tree. Parsing directories in a large tree is difficult. In this paper, we propose an accelerator, which helps file systems to fetch the metadata of files rapidly. Contributions of this work include two aspects. First, we propose an accelerator for directory parsing. The accelerator is actually an SSD-based (Solid State Drive-based) cache, which keeps the metadata of frequently or recently accessed files and directories. When a file is demanded, the accelerator attempts to obtain its metadata directly from SSD. If the metadata is kept in SSD, the file system can rapidly obtain the metadata. However, if the metadata is not in SSD, the accelerator consumes a long time to access SSD, but to no avail. In order to avoid non-beneficial SSD accesses, the accelerator predicts whether the metadata is kept by SSD before issuing a read request. Only if the metadata has a high probability of being kept in SSD, the accelerator issues a request to the SSD. The second contribution of this paper is a new bloom filter used to predict whether a piece of data is kept in SSD. Bloom filter is a space-efficient data structure supporting membership query. But, the standard bloom filter cannot support element deletion. Whereas, our accelerator is a cache, which evicts items periodically. The standard bloom filter is not suitable for our accelerator. In this work, we designed a new bloom filter with low overhead, which supports element deletion. The new bloom filter perfectly suits the proposed accelerator. With the prediction of our bloom filter, the accelerator can accelerate the process of directory parsing with nearly no negative impact. We evaluated the accelerator by using a prototype. Experimental results demonstrate that, the accelerator can speed up the directory parsing process by nearly four times compared with a file system without an accelerator.     

蓝鲸分布式文件系统的客户端元数据缓存模型

在蓝鲸分布式文件系统中,客户端的所有元数据操作都是通过远程过程调用由元数据服务器完成,所有数据读写都是直接与存储服务器交换完成的.由于通信延迟,在客户端进行频繁数据读写时,元数据信息交换影响了整个系统的性能.我们设计了一种在客户端尽量缓存文件元数据信息的模型,有效地减少了元数据通信,缩短了整个读写过程的延迟,极大地提高了蓝鲸分布式文件系统的性能.