Issues and challenges in the performance analysis of real disk arrays

The performance modeling and analysis of disk arrays is challenging due to the presence of multiple disks, large array caches, and sophisticated array controllers. Moreover, storage manufacturers may not reveal the internal algorithms implemented in their devices, so real disk arrays are effectively black-boxes. We use standard performance techniques to develop an integrated performance model that incorporates some of the complexities of real disk arrays. We show how measurement data and baseline performance models can be used to extract information about the various features implemented in a disk array. In this process, we identify areas for future research in the performance analysis of real disk arrays.     

Performance of RAID5 disk arrays with read and write caching

In this paper, we develop analytical models and evaluate the performance of RAID5 disk arrays in normal mode (all disks operational), in degraded mode (one disk broken, rebuild not started) and in rebuild mode (one disk broken, rebuild started but not finished). Models for estimating rebuild time under the assumption that user requests get priority over rebuild activity have also been developed. Separate models were developed for cached and uncached disk controllers. Particular emphasis is on the performance of cached arrays, where the caches are built of Non-Volatile memory and support write caching in addition to read caching. Using these models, we evaluate the performance of arrayed and unarrayed disk subsystems when driven by a database workload such as those seen on systems running any of several popular database managers. In particular, we assume single-block accesses, flat device skew and little seek affinity.With the above assumptions, we find six significant results. First, in normal mode, we find there is no difference in performance between subsystems built out of either small arrays or large arrays as long as the total number of disks used is the same. Second, we find that if our goal is to minimize the average response time of a subsystem in degraded and rebuild modes, it is better to use small arrays rather than large arrays in the subsystem. Third, we find the counter-intuitive result that if our goal is to minimize the average response time of requests to any one array in the subsystem, it is better to use large arrays than small arrays in the subsystem. We call this the best worst-case phenomenon.Fourth, we find that when no caching is used in the disk controller, subsystems built out of arrays have a normal mode performance that is significantly worse than an equivalent unarrayed subsystem built of the same drives. For the specific drive, controller, workload and system parameters we used for our calculations, we find that, without a cache in the controller and operating at typical I/O rates, the normal mode response time of a subsystem built out of arrays is 50% higher than that of an unarrayed subsystem. In rebuild mode, we find that a subsystem built out of arrays can have anywhere from 100% to 200% higher average response time than an equivalent unarrayed subsystem.Out fifth result is that, with cached controllers, the performance differences between arrayed and equivalent unarrayed subsystems shrink considerably. We find that the normal mode response time in a subsystem built out of arrays is only 4.1% higher than that of an equivalent unarrayed system. In degraded (rebuild) mode, a subsystem built out of small arrays has a response time 11% (13%) higher and a subsystem built out of large arrays has a response time 15% (19%) higher than an unarrayed subsystem.Our sixth and last result is that cached arrays have significantly better response times and throughputs than equivalent uncached arrays. For one workload, a cached array with good hit ratios had 5 times the throughout and 10 to 40 times lower response times than the equivalent uncached array. With poor hit ratios, the cached array is still a factor of 2 better in throughput and a factor of 4 to 10 better in response time for this same workload.We conclude that 3 design decisions are important when designing disk subsystems built out of RAID level 5 arrays. First, it is important that disk subsystems built out of arrays have disk controllers with caches, in particular Non-Volatile caches that cache writes in addition to reads. Second, if one were trying to minimize the worst response time seen by any user, one would choose disk array subsystems built out of large RAID level 5 arrays because of the best worst-case phenomenon. Third, if average subsystem response time is the most important design metric, the subsystem should be built out of small RAID level 5 arrays.     

Efficient video file allocation schemes for video-on-demand services

A video-on-demand (VOD) server needs to store hundreds of movie titles and to support thousands of concurrent accesses. This, technically and economically, imposes a great challenge on the design of the disk storage subsystem of a VOD server. Due to different demands for different movie titles, the numbers of concurrent accesses to each movie can differ a lot. We define access profile as the number of concurrent accesses to each movie title that should be supported by a VOD server. The access profile is derived based on the popularity of each movie title and thus serves as a major design goal for the disk storage subsystem. Since some popular (hot) movie titles may be concurrently accessed by hundreds of users and a current high-end magnetic disk array (disk) can only support tens of concurrent accesses, it is necessary to replicate and/or stripe the hot movie files over multiple disk arrays. The consequence of replication and striping of hot movie titles is the potential increase on the required number of disk arrays. Therefore, how to replicate, stripe, and place the movie files over a minimum number of magnetic disk arrays such that a given access profile can be supported is an important problem. In this paper, we formulate the problem of the video file allocation over disk arrays, demonstrate that it is a NP-hard problem, and present some heuristic algorithms to find the near-optimal solutions. The result of this study can be applied to the design of the storage subsystem of a VOD server to economically minimize the cost or to maximize the utilization of disk arrays.     

Performability of disk-array-based video servers

Video services are likely to dominate the traffic in future broadband networks. Most of these services will be provided by large- scale public-access video servers. Research to date has shown that disk arrays are a promising technology for providing the storage and throughput required to serve many independent video streams to a large customer population. Large disk arrays, however, are susceptible to disk failures which can greatly affect their reliability. In this paper, we discuss suitable redundancy mechanisms to increase the reliability of disk arrays and compare the performance of the RAID-3 and RAID-5 redundancy schemes. We use cost and performability analyses to rigorously compare the two schemes over a variety of conditions. Accurate cost models are developed and Markov reward models (with time-dependent reward structures) are developed and used to give insight into the tradeoffs between system cost and revenue earning potential. The paper concludes that for large-scale video servers, coarse-grained striping in a RAID-5 style of disk array is most cost effective.     

磁盘存储测试技术研究

研究了测试海量存储系统中磁盘阵列IOPS、数据传输率两项性能指标的技术与方法,提出了针对高性能磁盘阵列的并行测试技术,并使用此方法实现了对最新研制的高性能磁盘阵列的评测。通过研究影响单个磁盘性能指标的各项因素,并借助实验对其进行量化分析,最终实现了对整个磁盘阵列的性能评测。研究了影响磁盘阵列整体性能的关键瓶颈点,并给出了各项测试结果的合理性证明。     

Stripped mirroring RAID architecture

Redundant arrays of independent disks (RAID) provide an efficient stable storage system for parallel access and fault tolerance. The most common fault tolerant RAID architecture is RAID-1 or RAID-5. The disadvantage of RAID-1 lies in excessive redundancy, while the write performance of RAID-5 is only 1/4 of that of RAID-0. In this paper, we propose a high performance and highly reliable disk array architecture, called stripped mirroring disk array (SMDA). It is a new solution to the small-write problem for disk array. SMDA stores the original data in two ways, one on a single disk and the other on a plurality of disks in RAID-0 by stripping. The reliability of the system is as good as RAID-1, but with a high throughput approaching that of RAID-0. Because SMDA omits the parity generation procedure when writing new data, it avoids the write performance loss often experienced in RAID-5.     

Performance Modeling and Evaluation of a Two-Dimensional Disk Array System

This paper presents a performance model of a two-dimensional disk array (TIDA) system, which is composed of several major subsystems including disk cache, intelligent disk array controller, SCSI-like I/O bus, and two-dimensional array of disk devices. Accessing conflict in these subsystems and fork/join synchronization of physical disk requests are considered in the model. The representation for the complex behavior, including the interactions among subsystems, of a whole disk array system distinguishes the model from others that model only individual subsystems. To assist evaluating the architectural alternatives of TIDA, we employ a subsystem access time modeling methodology, in which we model for each subsystem the mean subsystem access time per request (SATPR). Fed with a given set of representative workload parameters, the performance model is used to conduct performance evaluation and the SATPRs of the subsystems are utilized to identify the bottleneck subsystem for performance improvement. The results show that (1) the values of some key design parameters, such as data block size and I/O bus bandwidth that yield the best system throughput are dependent not only on the subsystem performance but also on the interaction among subsystems; (2) an I/O bus bandwidth of 5 Mbytes/s per disk device is large enough for data transfers from/to disk devices equipped with a cache of 1 Mbytes; and (3) the activity of fork/join synchronization of physical disk requests may cause performance degradation, which can be improved by using large I/O bus bandwidth and/or placing a cache in each disk device.     

Coding techniques for handling failures in large disk arrays

A crucial issue in the design of very large disk arrays is the protection of data against catastrophic disk failures. Although today single disks are highly reliable, when a disk array consists of 100 or 1000 disks, the probability that at least one disk will fail within a day or a week is high. In this paper we address the problem of designing erasure-correcting binary linear codes that protect against the loss of data caused by disk failures in large disk arrays. We describe how such codes can be used to encode data in disk arrays, and give a simple method for data reconstruction. We discuss important reliability and performance constraints of these codes, and show how these constraints relate to properties of the parity check matrices of the codes. In so doing, we transform code design problems into combinatorial problems. Using this combinatorial framework, we present codes and prove they are optimal with respect to various reliability and performance constraints.This paper is a revised and expanded version of material that appeared at the Third International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS III), Boston, MA, March 1989. The work here was supported in part by the National Science Foundation under Grant Numbers MIP-8715235 and CCR-8411954, as well as an AT&T Bell Labs GRPW grant, a Siemens Corporation grant, and an IBM graduate fellowship.     

The performance of disk arrays in shared-memory database machines

Disk arrays and shared-memory multiprocessors are new technologies that are rapidly becoming pervasive. They are complementary because disk arrays naturally balance the I/O workload by interleaving data across all disks while a shared-memory multiprocessor balances the processing workload across multiple processors. In this paper, we examine how disk arrays and shared memory multiprocessors lead to an effective method for constructing database machines for general-purpose complex query processing. We show that disk arrays can lead to cost-effective storage systems if they are configured from suitably small formfactor disk drives. We introduce the storage system metricdata temperature (IO/s/Gbyte) as a way to evaluate how well a disk configuration can sustain its workload, and we show that disk arrays can sustain the same data temperature as a more expensive mirrored-disk configuration. We use the metric to evaluate the performance of disk arrays in XPRS, an operational shared-memory multiprocessor database system being developed at the University of California, Berkeley.     

Issues in the design of a storage server for video-on-demand

We examine issues related to the design of a storage server for video-on-demand (VOD) applications. The storage medium considered is magnetic disks or arrays of disks. We investigate disk scheduling policies, buffer management policies and I/O bus protocol issues. We derive the number of sessions that can be supported from a single disk or an array of disks and determine the amount of buffering required to support a given number of users. Furthermore, we propose a scheduling mechanism for disk accesses that significantly lowers the buffer-size requirements in the case of disk arrays. The buffer size required under the proposed scheme is independent of the number of disks in the array. This property allows for striping video content over a large number of disks to achieve higher concurrency in access to a particular video object. This enables the server to satisfy hundreds of independent requests to the same video object or to hundreds of different objects while storing only one copy of each video object. The reliability implications of striping content over a large number of disks are addressed and two solutions are proposed. Finally, we examine various policies for dealing with disk thermal calibration and the placement of videos on disks and disk arrays.     

自修复磁盘阵列技术研究

以数据为中心的计算模式对数据的可靠性提出新的需求,高可靠存储设备成为高性能计算的基础.自从Xiotech公司提出ISE结构并取得成功后,关注磁盘失效之前的征兆而不是失效后的具体处理的"自修复"概念已成为磁盘阵列设计新的指导思想之一.提出存储系统可靠性系数的概念,分析单个磁盘可靠性与磁盘阵列可靠性关系,选择合适的磁盘S.M.A.R.T属性结合磁盘当前利用率和历史信息建立评定磁盘状态的T~2US算法,接着介绍RAID 5结合T2US监测的高可靠、自修复的RAID 5T结构,说明RAID 5T结构的运行方式和磁盘不符合T~2US监测标准时数据迁移策略,最后对结构的可靠性利用存储系统可靠性系数和MTTF分别进行了分析.     

RAID-RMS: A fault tolerant stripped mirroring RAID architecture for distributed systems

Disk arrays, or RAIDs, have become the solution to increase the capacity, bandwidth and reliability of most storage systems. In spite of its high redundancy level, disk mirroring is a popular RAID paradigm, because replicating data also doubles the bandwidth available for processing read requests, improves the reliability and achieves fault tolerance. In this paper, we present a new RAID architecture called RAID-RMS in which a special hybrid mechanism is used to map the data blocks to the cluster. The main idea behind the proposed algorithm is to combine the data block striping and disk mirroring technique with a data block rotation. The resulting architecture improves the parallelism reliability and efficiency of the RAID array. We show that the proposed architecture is able to serve many more disk requests compared to the other mirroring-based architectures. We also argue that a more balanced disk load is attained by the given architecture, especially when there are some disk failures.     

冗余磁盘阵列RAID5性能损失研究

冗余磁盘阵列虽然引入了容错机制使得磁盘阵列的数据可靠性得到了很大的提高，但同时也引起性能不降。而且随着磁盘数量的增加，磁盘失效的概率将明显增大，当单个磁盘失效后，虽然此时磁盘阵列数据并未矢失，且仍能服务于系统的请量此时磁盘阵列是带“病”工作，处于一种降级模式，本文对冗余磁盘阵列ＲＡＩＤ５进行了队列建模和仿真计算，提出了性能损失率的概念，并作为评价磁盘阵列性能损失的衡量指标。计算结果分析表明，ＲＡＩ     

Performance analysis of advanced I/O architectures for PC-based video servers

In the personal computing and workstation environments, more and more I/O adapters are becoming complete functional subsystems that are intelligent enough to handle I/O operations on their own without much intervention from the host processor. The IBM subsystem control block (SCB) architecture has been defined to enhance the potential of these intelligent adapters by defining services and conventions that deliver command information and data to and from the adapters. In recent years, a new storage architecture, the redundant array of independent disks (RAID), has been quickly gaining acceptance in the world of computing. In this paper, we discuss and present a performance analysis of the SCB architecture and disk array technology in typical video server environments. In particular, we would like to see whether a disk array can outperform a group of disks (of the same type, the same data capacity, and same cost) operating independently (not in parallel as in a disk array) in a video server environment where most disk I/O operations are large sequential reads.     

异构磁盘阵列RAID5结构数据布局研究

由于应用需求的快速发展以及网络存储系统的出现,因此异构磁盘阵列的变得越来越常见。RAID5由于较高的性能和可靠性以及较低的代价,是应用最为广泛的RAID结构。目前对异构磁盘阵列RAID5结构的研究,重点主要放在充分利磁盘存储空间以及对性能的定性研究。论文提出了一种异构磁盘阵列RAID5结构数据布局优化方法,该方法充分考虑异构磁盘的相对容量和性能,以及校验单元的散布对RAID5小数据写性能的影响,可以生成负载均匀分布或接近均匀分布的布局。仿真实验结果表明,对于多用户小数据访问模式,优化布局的性能明显优于简单RAID5布局,且具有更高的伸缩性。     

Architectures and algorithms for on-line failure recovery in redundant disk arrays

The performance of traditional RAID Level 5 arrays is, for many applications, unacceptably poor while one of its constituent disks is non-functional. This paper describes and evaluates mechanisms by which this disk array failure-recovery performance can be improved. The two key issues addressed are thedata layout, the mapping by which data and parity blocks are assigned to physical disk blocks in an array, and thereconstruction algorithm, which is the technique used to recover data that is lost when a component disk fails.The data layout techniques this paper investigates are instantiations of thedeclustered parity organization, a derivative of RAID Level 5 that allows a system to trade some of its data capacity for improved failure-recovery performance. We show that our instantiations of parity declustering improve the failure-mode performance of an array significantly, and that a parity-declustered architecture is preferable to an equivalent-size multiple-group RAID Level 5 organization in environments where failure-recovery performance is important. The presented analyses also include comparisons to a RAID Level 1 (mirrored disks) approach.With respect to reconstruction algorithms, this paper describes and briefly evaluates two alternatives,stripeoriented reconstruction anddisk-oriented reconstruction, and establishes that the latter is preferable as it provides faster reconstruction. The paper then revisits a set of previously-proposed reconstruction optimizations, evaluating their efficacy when used in conjunction with the disk-oriented algorithm. The paper concludes with a section on the reliability versus capacity trade-off that must be addressed when designing large arrays.Portions of this material are drawn from papers at the 5th Conference on Architectural Support for Programming Languages and Operating Systems, 1992, and at the 23rd Symposium on Fault-Tolerant Computing, 1993. The work was supported by the National Science Foundation under grant number ECD-8907068, by the Defense Advanced Research Project Agency monitored by ARPA/CMO under contract MDA972-90-C-0035, and by an IBM Graduate Fellowship.     

MT2RAID:一种高可靠大规模磁盘阵列结构

传统的磁盘阵列一般采用集中式控制结构,其连接的底层磁盘数受系统总线的制约,容易出现性能瓶颈,且不能容两个以上磁盘出错。从模块化系统的组织方法出发,提出一种采用标准模块化存储单元组成的通过胖树结构互连的大规模磁盘阵列结构MT2RAID,分别就其各种数据分布的性能和可靠性进行了分析和讨论。原型系统测试结果表明,相比集中式磁盘阵列结构,MT2RAID也具有较高的性能。     

Understanding the relationship between energy conservation and reliability in parallel disk arrays

Energy conservation schemes based on power management or workload skew for disk arrays adversely affect disk reliability due to either workload concentration or frequent disk speed transitions. A thorough understanding of the relationship between energy-saving techniques and disk reliability is still an open problem, which prevents effective design of new energy-saving techniques and application of existing approaches in reliability-critical environments. This paper presents an empirical reliability model, called PRESS (Predictor of Reliability for Energy-Saving Schemes). Fed by operating temperature, disk utilization, and disk speed transition frequency, PRESS estimates the reliability of an entire disk array. Further, a new energy-saving strategy with reliability awareness named READ (Reliability and Energy Aware Distribution) is developed in the light of the insights provided by PRESS. Experimental results demonstrate that READ consistently performs better than existing approaches in performance and reliability while achieving a comparable level of energy consumption.     

NAS设备的磁盘调度模型分析

附网存储 (NAS)设备的性能目标是优化网络存储数据访问和存储子系统的管理 .本文旨在显示随着磁盘转速的增加 ,NAS应该从磁盘硬件的最佳工作性能出发 ,整体配合以提高它的 I/O操作性能 .为了发掘 NAS最佳的工作性能 ,我们建立同时基于磁臂位置和旋转位置的精确的磁盘调度模型 ,并提出测量调度算法性能的方法 .以 HP975 6 0性能参数为基础 ,进行理论分析和模拟测试 .结果磁盘转速越快 ,磁盘访问的开销越大 .因此 ,NAS设备的设计必须从整体上考虑磁盘调度策略的选择 ,并行多磁盘结构的选择 ,文件 cache的分配和文件系统布局等 ,以便提高网络存储数据访问的性能     

基于磁盘阵列的VLBI计算机存储系统

完成对深空测控外部存储系统磁盘阵列的设计是深空战略工作中的一部分.本文在分析磁盘组阵各性能的基础上,为深空测控计算机外部存储系统给出磁盘阵列组阵建议.根据深空测控任务要求出发,对磁盘的各组阵方式的容量、速度和可靠性等方面进行理论分析和实际测量,结果显示RAID0组阵方式是符合存储系统需求的组阵方式.