基于三方传送的存储管理系统设计与实现

存储系统已由NAS向SAN发展,如何融合NAS和SAN技术,提高存储系统的可管理性和性能,是目前存储领域研究的热点。提出了一种扩展文件共享的方法,采用三方传送机制来融合NAS和SAN,通过存储管理器和存储服务器的功能划分,由存储管理器负责存储设备的空间和文件的管理,存储服务器能共享SAN中的存储设备以及直接访问存储设备上的数据。该系统采用Lease协议来保证存储服务器端数据一致性,在此基础上分析了Lease协议模型。     

国产化环境下的海量小文件数据分布式存储技术

为缓解单一存储设备存储海量小文件的压力,提出了一种国产化环境下的海量小文件数据分布式存储技术。利用聚类算法实现海量小文件合并。以达到最大均衡度为目标,在多项约束条件下利用人工鱼群算法求解分布式存储方案。按照分布式存储方案将海量小文件数据迁移到存储节点及其存储设备上,完成海量小文件数据分布式存储。结果表明：14个存储节点和28个存储设备的内存占用较为均衡,内存资源利用率较高。将小文件样本迁移并存储到节点的过程中,分布式存储均衡度整体波动均超过设定的阈值1.0,说明分布式存储均衡度较好,证明了所提存储技术的有效性。     

面向海量高清视频数据的高性能分布式存储系统

通过对视频监控数据的特点和传统存储方案进行分析,提出一种高性能分布式存储系统解决方案.不同于传统的基于文件存储的方式,设计了一种逻辑卷结构,将非结构化的视频流数据以此结构进行组织并直接写入RAW磁盘设备,解决了传统存储方案中随机磁盘读写和磁盘碎片导致存储性能下降的问题.该方案将元数据组织为两级索引结构,分别由状态管理器和存储服务器管理,极大地减少了状态管理器需要管理元数据的数量,消除了性能瓶颈,并提供精确到秒级的检索精度.此外,该方案灵活的存储服务器分组策略和组内互备关系使得存储系统具备容错能力和线性扩展能力.系统测试结果表明,该方案在成本低廉的PC服务器上实现了单台服务器能同时记录400路1080P视频流,写入速度是本地文件系统的2.5倍.     

基于freeNAS存储服务器实现XenServer资源池的存储共享

基于VMware workstation上虚拟出的两台Citrix公司的产品XenServer机器,一台虚拟freeNAS（一种免费的NAS存储服务器）,通过使用freeNAS存储服务器做成的NFS存储和iSCSI存储,并且将这些存储设备添加到XenServer的资源池中,以成为XenServer的公用存储设备。阐述了XenServer和NAS提供的NFS存储的关系,剖析了NAS存储服务器提供iSCSI存储和NFS存储的详细过程,包括服务器的架构以及存储设备和XenServer的相连,总结了NAS存储在虚拟世界中的作用。     

共产存储Tivoli有方

SAN(存储区域网络)的概念被提出后,立即迅速发展成为企业管理电子商务系统所产生的大量数据的首选方案。作为一个专有的、集中化管理的、安全高速的存储网络,SAN 可跨平台、跨系统来管理数据,实现实时备份和恢复、数据移植和资源共享。但在目前企业的文件系统中,由于存在着多种异构的文件服务器/工作站,而文件服务器/工作站却无法对存储设备进行跨平台访问,因此每个文件服务器/工作站都必须拥有各自的存储设备,这就造成了存储设备的大量浪费。同时,由于客户端/服务器的最高传输速度目前仅为4MB/s,因此使服务器成为了数据访问和传输的瓶颈,造成带宽不够用、CPU 占用率大等问题。针对上述这些问题,Tivoli 系统公司于5月11日发布了 Tivoli SANergy 文件共享软件。     

NoSQL的分布式存储与扩展解决方法

传统的关系型数据库已无法满足海量数据的存储与访问需求。针对该问题,提出一种非关系型数据库(NoSQL)的分布式存储与扩展解决方法。分析并改进NoSQL,讨论基于一致性哈希算法键值对的分布式存储,以及基于双hash环的数据库服务器节点的扩展方法,提出将NoSQL作为镜像引入数据库架构系统。实际应用结果表明,该方法可以避免资源浪费及服务器过载。     

基于Web的海量存储柔性分布式文件服务器设计

分析了以往分布式文件服务器的不足 ,提出了一种以J2EE技术为核心的基于Web的支持海量文件存储的分布式文件服务器的设计方案。该分布式文件服务器由一个集中控制单元和多个本地文件服务器组成。集中控制单元对文件逻辑信息进行集中式管理 ;文件的物理信息则由位于Web不同节点上的本地文件服务器管理。     

基于云预存储技术的Docker在线迁移方法

针对基于Docker容器的分布式云计算下出现负载不均衡问题,有必要将较高负载服务器中的Docker容器进程迁移到其他相对空闲的服务器上.而传统的容器迁移算法忽视了容器本身的特征,从而导致在迁移过程中传输效率低下.基于此,利用第三方管理平台和数据预存储阈值机制,提出一种Docker容器动态迁移预存储算法PF-Docker.首先将Docker容器内部进程运行相关文件和流动数据预存至云端存储器,然后通过预存储阈值机制减少流动数据的无效传输,最后在停机传输阶段将流动数据和冗余数据传输给目的服务器.实验表明,该方法在Docker容器迁移中能有效地降低迁移时间,减少数据传输量,提高容器的容错率.     

SUN直盯EMC

SUN 正在专攻高容量 Internet 存储业务,存储市场的领先者 EMC正是 SUN 公司明确的目标。最近,SUN 宣布将成立一个新的存储销售队伍,主要任务是执行 SUN 的网络存储策略.SUN 销售附加于服务器的存储设备以及与服务器相独立的 SAN系统。SUN 公司称在未来的几个月内将对 Jiro 进行扩展。Jiro 是一种基于 Java 的技术,它超越所有的存储产品设备,在存储系统之中提     

分布式文件系统HDFS处理小文件的优化方案

Hadoop分布式文件系统(HDFS)是为可靠地存储和管理海量文件而设计。在HDFS中,所有的文件由单一的服务器NameNode来管理。因此,随着小文件数量的增加,会使HDFS系统性能下降。为了提高存储和访问HDFS上小文件的效率,本文提出了一个解决方案,即:扩展的Hadoop分布式文件系统(EHDFS)。这种方法把一组相关文件组合成一个大文件来减少文件的数量,然后建立一种索引机制,从这个组合文件中识别并访问客户所要的单个文件。实验结果表明EHDFS提高了存储和访问大量小文件的效率。     

基于Web服务的分布式文件系统模型

文章研究了面向服务架构的高性能网络海量文件存储系统。文中采用具有高可扩展性的Web服务体系,将Internet上大量分散的文件服务器组织成一个逻辑整体,形成一个遍布Internet的大规模分布式系统,各文件服务器相互协作,共同对外提供文件服务。文中探讨了分布式体系结构中的高扩展性能和提高广域网范围文件访问性能的方法。     

A dual-direction technique for fast file downloads with dynamic load balancing in the Cloud

File downloads make up a large percentage of the Internet traffic to satisfy various clients using distributed environments for their Cloud, Grid and Internet applications. In particular, the Cloud has become a popular data storage provider and users (individuals and corporates) are relying heavily on it to keep their data. Furthermore, most cloud data servers replicate their data storage infrastructures and servers at various sites to meet the overall high demands of their clients and increase availability. However, most of them do not use that replication to enhance the download performance per client. To make use of this redundancy and to enhance the download speed, we introduce a fast and efficient concurrent technique for downloading large files from replicated Cloud data servers and traditional FTP servers as well. The technique, DDFTP utilizes the availability of replicated files on distributed servers to enhance file download times through concurrent downloads of file blocks from opposite directions in the files. DDFTP does not require coordination between the servers and relies on the in-order and reliability features of TCP to provide fast file downloads. In addition, DDFTP offers efficient load balancing among multiple heterogeneous data servers with minimal overhead. As a result, we can maximize network utilization while maintaining efficient load balancing on dynamic environments where resources, current loads and operational properties vary dynamically. We implemented and evaluated DDFTP and experimentally demonstrated considerable performance gains for file downloads compared to other concurrent/parallel file/data download models.     

Block I/O Scheduling on Storage Servers of Distributed File Systems

This paper presents a new scheme of I/O scheduling on storage servers of distributed/parallel file systems, for yielding better I/O performance. To this end, we first analyze read/write requests in the I/O queue of storage server (we name them block I/Os), by using our proposed technique of horizontal partition. Then, all block requests are supposed to be divided into multiple groups, on the basis of their offsets. This is to say, all requests related to the same chunk file will be grouped together, and then be satisfied within the same time slot between opening and closing the target chunk file on the storage server. As a result, the time resulted by completing block I/O requests can be significantly decreased, because of less file operations on the corresponding chunk files at the low-level file systems of server machines. Furthermore, we introduce an algorithm to rate a priority for each group of block I/O requests, and then the storage server dispatches groups of I/Os by following the priority order. Consequently, the applications having higher I/O priorities, e.g. they have less I/O operations and small size of involved data, can finish at a earlier time. We implement a prototype of this server-side scheduling in the PARTE file system, to demonstrate the feasibility and applicability of the proposed scheme. Experimental results show that the newly proposed scheme can achieve better I/O bandwidth and less I/O time, compared with the strategy of First Come First Served, as well as other server-side I/O scheduling approaches.     

一个网络文件存储系统TNS关键技术

基于多服务器架构、为多用户服务的网络文件存储系统普遍存在资源分配不均,重复文件多,存储空间浪费严重的问题。设计并实现了TNS网络文件存储系统,该系统基于多服务器存储架构,分别由用户服务器、索引服务器、数据服务器、共享服务器、管理服务器和登录服务器组成,为多用户服务,采用一致性Hash实现负载均衡,支持在客户端进行文件粒度的重复数据删除。经过实际生产环境运行测试,具有良好的负载均衡能力和重复数据删除功能,可以有效节省存储空间,提高存储设备利用率。     

存储系统细粒度加密安全设计

存储系统会有成百上千的存储设备和大量的数据。其中有相当部分敏感数据需要加密存储,然而现有的存储系统只能对整个文件进行加密,而不能对文件某一部分进行加密。尽管其实现方法比较简单,但它会导致系统额外的开销并且极大的影响系统性能。本文提出了存储系统的一种文件细粒度加密方案,它可以充分利用分布式存储内在的特性,有效地利用元数据信息对文件内多个任意大小的数据块进行加密。用户还可以根据数据块的保密级别对不同的数据块定义不同的加密算法以满足需要。对于广泛存在的视频、地图、档案等大文件实现不同粒度加密策略,可减少文件中不必要的数据加密操作,极大提高存储系统的性能。     

A paracasting model for concurrent access to replicated Internet content

In this paper, we develop a model to study how to effectively download a document from a set of replicated servers. We propose a generalized application-layer anycasting protocol, known as paracasting, to advocate concurrent access of a subset of replicated servers to cooperatively satisfy a client's request. Each participating server satisfies the request in part by transmitting a subset of the requested file to the client. The client can recover the complete file when different parts of the file sent from the participating servers are received. This model allows us to estimate the average time to download a file from the set of homogeneous replicated servers, and the request blocking probability when each server can accept and serve a finite number of concurrent requests. Our results show that the file download time drops when a request is served concurrently by a larger number of homogeneous replicated servers, although the performance improvement quickly saturates when the number of servers increases. If the total number of requests that a server can handle simultaneously is finite, the request blocking probability increases with the number of replicated servers used to serve a request concurrently. Therefore, paracasting is effective when a small number of servers, say, up to four, are used to serve a request concurrently.     

A novel distributed architecture of large-scale multimedia storage system using autonomous object-based storage devices

In a large-scale multimedia storage system (LMSS) where the user requests for different multimedia objects may have different demands, placement and replication of the objects is an important factor, as it may result in an imbalance in loading across the system. Since replica management and load balancing is a crucial issue in multimedia systems, normally this problem is handled by centralized servers, e.g., metadata servers (MDS) in distributed file systems. Each object-based storage device (OSD) responds to the requests coming from the centralized servers independently and has no communication with other OSDs among the system. In this paper, we design a novel distributed architecture of LMSS, in which the OSDs have some kind of intelligences and can cooperate to achieve a high performance. Such an OSD, named as autonomous object-based storage device (AOSD), can replicate the objects to and balance the requests among other AOSDs, and handle fail-over and recovery autonomously. In the proposed architecture, we move the request balancing from centralized MDS to AOSDs and make the system more scalable, flexible, and robust. Based on the proposed architecture, we propose two different object replication and load balancing algorithms, named as “Minimum Average Waiting Time” (MAWT) and “One of the Best Two Choices” (OBTC), respectively. We validate the performance of the algorithms via rigorous simulations with respect to several influencing factors. Our findings conclusively demonstrate that the proposed architecture minimizes the average waiting time and at the same time carries out load balancing across servers.     

蓝鲸分布式文件系统的客户端元数据缓存模型

在蓝鲸分布式文件系统中,客户端的所有元数据操作都是通过远程过程调用由元数据服务器完成,所有数据读写都是直接与存储服务器交换完成的.由于通信延迟,在客户端进行频繁数据读写时,元数据信息交换影响了整个系统的性能.我们设计了一种在客户端尽量缓存文件元数据信息的模型,有效地减少了元数据通信,缩短了整个读写过程的延迟,极大地提高了蓝鲸分布式文件系统的性能.     

Adaptive metadata rebalance in exascale file system

This paper presents an effective method of metadata rebalance in exascale distributed file systems. Exponential data growth has led to the need for an adaptive and robust distributed file system whose typical architecture is composed of a large cluster of metadata servers and data servers. Though each metadata server can have an equally divided subset from the entire metadata set at first, there will eventually be a global imbalance in the placement of metadata among metadata servers, and this imbalance worsens over time. To ensure that disproportionate metadata placement will not have a negative effect on the intrinsic performance of a metadata server cluster, it is necessary to recover the balanced performance of the cluster periodically. However, this cannot be easily done because rebalancing seriously hampers the normal operation of a file system. This situation continues to get worse with both an ever-present heavy workload on the file system and frequent failures of server components at exascale. As one of the primary reasons for such a degraded performance, file system clients frequently fail to look up metadata from the metadata server cluster during the period of metadata rebalance; thus, metadata operations cannot proceed at their normal speed. We propose a metadata rebalance model that minimizes failures of metadata operations during the metadata rebalance period and validate the proposed model through a cost analysis. The analysis results demonstrate that our model supports the feasibility of online metadata rebalance without the normal operation obstruction and increases the chances of maintaining balance in a huge cluster of metadata servers.