ZFS文件系统中双容错编码性能的研究

ZFS是Sun推出的一款革新性的文件系统,它支持用户构建镜像以及单容错/双容错的软Raid。其双容错编码方案采用的是Reed-Solomon编码(RS码)。由于RS码是基于有限域运算,编码/解码时间复杂性差是其根本性的缺陷。ZFS对写操作的处理采用的是聚合后追加的方式而非传统的覆盖方式,每次写操作都会进行一次编码计算。因此,编码计算性能是影响文件系统整体性能的重要因素之一。本文的工作是将RDP这一基于奇偶校验的双容错编码与ZFS相结合,替代Reed-Solomon编码,以优化文件系统写操作的性能。我们设计了Cache优化的RDP编码算法,在ZFS中进行了实现,并通过实验验证了这一方法的有效性。     

云文件系统中纠删码技术的研究与实现

云文件系统凭借高性能、高扩展、高可用、易管理等特点,成为云存储和大数据的基础和核心。云文件系统一般采用完全副本技术来提升容错能力,提高数据资源的使用效率和系统性能。但完全副本的存储开销随着副本数目的增加呈线性增长,存储副本时造成额外的写带宽和数据管理开销。纠删码在没有增加过量的存储空间的基础上,通过合理的冗余编码来保证数据的高可靠性和可用性。研究了纠删码技术在云文件系统中的应用,从纠删码类型、编码对象、编码时机、数据更改、数据访问方式和数据访问性能等六个方面,对云文件系统中纠删码的设计进行了探究,以增强云文件系统的存储模型。在此基础上,设计并实现了纠删码原型系统,并通过实验证明了纠删码能有效地保障云文件系统的数据可用性,并且节省存储空间。     

On the effect of Reed–Solomon coding with maximum block interleaving on MPEG-4 FGS video quality in 3G broadcasting

In Broadcast and Multicast Services (BCMCS) for CDMA2000 networks, Reed–Solomon (RS) coding is applied to the layers above the turbo code. An additional block interleaving mechanism improves the performance of RS coding when there are long bursts of errors. We propose a performance model of RS coding for varying conditions of Rayleigh fading on the broadcast channel with the maximum amount of block interleaving, and verify this analytic model by comparing its results with a simulation. Evaluating the quality of the resulting MPEG-4 FGS (fine granular scalability) video demonstrates the effectiveness of RS coding in achieving high perceived video quality.     

分布式存储系统（OceanStore）的复制策略

OceanStore项目是因特网上基于P2P结构的分布存储应用,目标是向用户提供强持久性、高可用性、可扩展性和安全性的服务。它的整个复制策略分为两个大的方面：ErasureCoding和Replication。ErasureCoding主要是保证数据的持久性,Replication主要是保证用户的访问速度。一份数据同时用ErasureCoding和Replication实现。     

RS编码与卷积编码级连在遥感卫星数据传输中的应用

针对具有多载荷、不同码速率数据流的遥感卫星,在星地数据传输方案中,一般采用空间数据系统咨询委员会(CCS-DS)建议的高级在轨系统(AOS),对数据进行动态管理、信道虚拟,同时进行RS编码与卷积编码级连。它有利于提高物理信道的利用率、增加数据编排的灵活性,有利于提高数据传输的质量。本文以我国的新一代极轨气象卫星的星地数据传输编码方式为例,分析了在数据传输中实际使用的RS编码和卷积编码级连的编码方案及流程,从理论和实践上分析了这种编码方案的纠错特性。     

一种高效纠错编码方案的计算机仿真设计

本文研究了高斯信道下RS码与PTCM码级联编码方式的性能,比较了应用PTCM码、RS码级联PTCM码、采用交织技术的级联码等使用8PSK调制方式的不同纠错方案的性能,设计了一种具体的差错控制编码方案,并对其进行计算机模拟仿真。仿真结果表明,该级联编码方案具有良好的误码纠错性能,可用作通用数据链控制器的纠错控制方案。     

云存储多异构文件联合延迟尾概率凸优化分析

随着擦除码技术的流行,分布式存储中高数据可靠性和高空间效率存储性能逐渐实现,但是降低尾部延迟仍然是一个有待解决的问题。为此,提出一种量化和优化擦除编码存储系统尾延迟的算法框架。对于任意服务时间分布和异构文件,推导给出尾部延迟上界。提出了一个优化模型,使得所有文件在服务器上放置的加权延迟尾概率和访问请求文件的服务器选择共同最小化,并证明了其非凸问题特性,以便采用一种高效的交替优化算法求解。此外,通过描述延迟分布尾部的渐近行为,以闭合形式对任意擦除编码存储的服务延迟的尾部指数进行数学量化,证明了基于概率调度的算法是（渐近）最优的。实验结果表明,在实际工作负载下擦除编码存储系统的尾部延迟显著降低。     

Sporadic decentralized resource maintenance for P2P distributed storage networks

In this paper, we propose a novel decentralized resource maintenance strategy for peer-to-peer (P2P) distributed storage networks. Our strategy relies on the Wuala overlay network architecture, (The WUALA Project). While the latter is based, for the resource distribution among peers, on the use of erasure codes, e.g., Reed–Solomon codes, here we investigate the system behavior when a simple randomized network coding strategy is applied. We propose to replace the Wuala regular and centralized strategy for resource maintenance with a decentralized strategy, where users regenerate new fragments sporadically, namely every time a resource is retrieved. Both strategies are analyzed, analytically and through simulations, in the presence of either erasure and network coding. It will be shown that the novel sporadic maintenance strategy, when used with randomized network coding, leads to a fully decentralized solution with management complexity much lower than common centralized solutions.     

Zippy: A Framework for Computation and Visualization on a GPU Cluster

Due to its high performance/cost ratio, a GPU cluster is an attractive platform for large scale general‐purpose computation and visualization applications. However, the programming model for high performance general‐purpose computation on GPU clusters remains a complex problem. In this paper, we introduce the Zippy frame‐work, a general and scalable solution to this problem. It abstracts the GPU cluster programming with a two‐level parallelism hierarchy and a non‐uniform memory access (NUMA) model. Zippy preserves the advantages of both message passing and shared‐memory models. It employs global arrays (GA) to simplify the communication, synchronization, and collaboration among multiple GPUs. Moreover, it exposes data locality to the programmer for optimal performance and scalability. We present three example applications developed with Zippy: sort‐last volume rendering, Marching Cubes isosurface extraction and rendering, and lattice Boltzmann flow simulation with online visualization. They demonstrate that Zippy can ease the development and integration of parallel visualization, graphics, and computation modules on a GPU cluster.     

一种基于多层次校验的低恢复成本纠删码

随着纠删码在分布式存储系统中的实际应用,纠删码为存储系统提供了更加优秀的存储效率,但当节点丢失时,相较于传统副本技术更多的网络传输带宽开销成为了造成系统性能瓶颈的关键因素。为了解决MDS编码高带宽开销对系统性能的影响,一类新型编码方案——分组码被应用在分布式存储系统中,相较于传统MDS编码能够有效地降低节点修复时的数据传输量,从而减少网络带宽需求。在Pyramid分组码的基础上进行层次扩展,提出一种HLRC（hierarchical local repair codes）纠删码。HLRC相较于LRC引入了层次编码模型,将原始数据块构建为编码矩阵,根据层次进行分别编码,生成包含数据块范围不同的局部校验块;每个层次包含的数据块数量不同,可以保证修复节点时的低修复成本,同时还拥有较高的存储效率。HLRC相较于Pyramid拥有额外的校验块冗余,能够降低校验块出错和多节点出错时的恢复开销。在基于Ceph的分布式存储系统中的实验结果表明,HLRC与Pyramid等分组码相比,单节点修复开销最高可降低48.56%,多节点修复开销最高可降低25%。     

Instruction set extensions for software defined radio

Software defined radios provide programmable solutions for implementing the physical layer processing of multiple communication standards. Mobile devices implementing these standards require high-performance processors to perform high-bandwidth physical layer processing in real time. In this paper, we present instruction set extensions for several important communication algorithms including cyclic redundancy checking, convolutional encoding, Viterbi decoding, turbo decoding, and Reed–Solomon encoding and decoding. We also present hardware designs for implementing these extensions, along with estimates of their area, critical path delay, and power consumption. The performance benefits of these extensions are evaluated using a supercomputer-class vectorizing compiler and the Sandblaster low-power multithreaded processor for software defined radio. The proposed instruction set extensions provide significant performance improvements at relatively low cost, while maintaining a high degree of programmability.     

Hierarchical codes: A flexible trade-off for erasure codes in peer-to-peer storage systems

Redundancy is the basic technique to provide reliability in storage systems consisting of multiple components. A redundancy scheme defines how the redundant data are produced and maintained. The simplest redundancy scheme is replication, which however suffers from storage inefficiency. Another approach is erasure coding, which provides the same level of reliability as replication using a significantly smaller amount of storage. When redundant data are lost, they need to be replaced. While replacing replicated data consists in a simple copy, it becomes a complex operation with erasure codes: new data are produced performing a coding over some other available data. The amount of data to be read and coded is d times larger than the amount of data produced, where d, called repair degree, is larger than 1 and depends on the structure of the code. This implies that coding has a larger computational and I/O cost, which, for distributed storage systems, translates into increased network traffic. Participants of Peer-to-Peer systems often have ample storage and CPU power, but their network bandwidth may be limited. For these reasons existing coding techniques are not suitable for P2P storage. This work explores the design space between replication and the existing erasure codes. We propose and evaluate a new class of erasure codes, called Hierarchical Codes, which allows to reduce the network traffic due to maintenance without losing the benefits given by traditional erasure codes.     

Exploring high performance distributed file storage using LDPC codes

We explore the feasibility of implementing a reliable, high performance, distributed storage system on a commodity computing cluster. Files are distributed across storage nodes using erasure coding with small low-density parity-check (LDPC) codes, which provide high-reliability with small storage and performance overhead. We present performance measurements done on a prototype system comprising 50 nodes, which are self organised using a peer-to-peer overlay.     

高顽存性存储网格模型

结合网格技术和存储技术,提出了基于域的存储网格模型。在域模型架构下,深入分析了域划分原则、元数据目录服务和数据容错等问题,重点探讨了数据高顽存性存储策略的设计。在比较复制、分片冗余等不同数据冗余策略特点的基础上,提出了一种混合式的数据冗余策略。模拟实验表明它兼具复制策略和分片冗余策略的优点,既保持了存储数据的整体可靠性水平,又弥补了低冗余度下编码复制可靠性偏低的不足,使冗余存储数据具有更好的可靠性效果。     

一种基于短LT码的级联编译码算法

针对实时无线通信对短纠删码的需求,提出一种短码长LT码与传统纠错码的级联方案。在综合考虑算法复杂度与纠错性能的情况下,选取RS码和卷积码的级联(RS-CC码)以构造等效删除信道,并采用实时性高的短LT码实现纠删功能。文中设计了一种适合短LT码的译码算法,同时给出了编码度分布的选取方法。仿真结果表明,与已有短喷泉码相比,文中短LT码成功译码时所需编码冗余更少,应用到级联方案后的数据传输可靠性明显提高。     

Transform coding on programmable stream processors

Stream processors can achieve high performance in stream applications that share stream characteristics of large parallelism, intensive computation and little data reuse. Transform coding, as a core component in video compression, is widely used in video storage and video transmission. This paper summarizes stream execution mechanism and explores design approaches of programmable stream processors including the Imagine stream processor and graphics processing unit (GPU). Based on the stream processing model, stream algorithms for block-based and frame-based (nonblock-based) transform coding are presented and mapped onto stream processors. Especially, an Interleaved Streaming Transform (IST) algorithm on Imagine and a Row-wise Zonal Transform (RZT) algorithm on GPU for 4×4 integer transform in H.264 are proposed to exploit great potential of stream processing for block-based transform. Our experiments of transform coding suite on Imagine and GPU show that the coding efficiency of stream processors is far beyond the real-time requirements of current video applications, dealing with a variety of different video resolutions ranging from QCIF to high definition (HD). The performance evaluation of stream implementations discusses the architectural supports for transform coding, and presents the significant improvements over other programmable platforms. Transform coding may take advantage of the flexibility of programmable stream processors with high performance to play an important role in the future.

图形分布处理支撑环境DGPSE

本文详细介绍图形分布处理支撑环境ＤＧＰＳＥ的设计和实现技术．ＤＧＰＳＥ有效支持高性能分布式图形应用系统的开发和分布并行图形处理算法的研究．它具有以下特点：支持多种分布处理模型，通讯方式灵活，图形支撑功能强．使用ＤＧＰＳＥ已实现了一分布式图形应用系统和一组分布式图形算法．的设计和实现．南京大学计算机系硕士论文，１９９１．１３潘志庆，石教英．Ｆｒａｃｔａｌ生成的并行算法研究．全国第７届ＣＡＤ和图形学会议论文集，无锡，１９９２．１４石教英，劳志强，潘志庚．多面体扫描转换的分布并行算法．全国第７届ＣＡＤ和图形学会议论文集，无锡，１９９２．     

一种面向数据可用性和存储可靠性动态要求的自适应纠删码存储策略设计

为了满足指数级增长的大数据存储需求,现代的分布式存储系统需要提供大容量的存储空间以及快速的存储服务.因此在主流的分布式存储系统中,均应用了纠删码技术以节约数据中心的磁盘成本,保证数据的可靠性,并且满足应用程序和客户端的快速存储需求.在实际应用中数据往往重要程度并不相同,对数据可用性要求不一,且不同磁盘的故障率和可靠性动态不一的特点,对于传统RAID存储方式包括基于纠删码的存储系统提出了新的挑战.本文提出了一种面向数据可用性和磁盘可靠性动态要求的灵活自适应纠删码存储设计On-demand ARECS(On-demand Availability and Reliability Oriented Adaptive Erasure Coded Storage System),根据存储后端数据可用性和磁盘可靠性的多个维度进行设计,综合确定纠删码编码策略和存储节点选择,从而减少存储冗余度和存储延迟,同时提高数据可用性和存储可靠性.我们在Tahoe-LAFS开源分布式文件系统中进行了实验,实验结果验证了我们的理论分析,在保证具有多样性要求的数据可用性和磁盘可靠性的前提下,明显减少了数据冗余度和存储延迟.     

Scalable rendering on PC clusters

Sandia National Laboratories use PC clusters and commodity graphics cards to achieve higher rendering performance on extreme data sets. The main obstacle in using cluster-based graphics systems is the difficulty in realizing the full aggregate performance of all the individual graphics accelerators, particularly for very large data sets that exceed the capacity and performance characteristics of any one single node. Based on our efforts to achieve higher performance, we present results from a parallel sort-last implementation that the scalable rendering project at Sandia National Laboratories generated. Our sort-last library (libpglc) can be linked to an existing parallel application to achieve high rendering rates. We ran performance tests on a 64-node PC cluster populated with commodity graphics cards. Applications using libpglc have demonstrated rendering performance of 300 million polygons per second $approximately two orders of magnitude greater than the performance on an SGI Infinite Reality system for similar applications     

Systolic algorithm for tensor products of matrices: Implementation and applications

Tensor products of matrices play a very important role in approximation and interpolation. This paper describes a systolic algorithm for tensor products in mesh connected arrays and the closely related hypercube architectures (including the Connection Machine). It is based on a new operator called bullet operator which is a higher dimensional matrix operation. The applications of tensor products to multivariable spline blending approximation as well as graphics/image processing are indicated.