Enabling dynamic file I/O path selection at runtime for parallel file system

Parallel file systems are experiencing more and more applications from various fields. Various applications have different I/O workload characteristics, which have diverse requirements on accessing storage resources. However, parallel file systems often adopt the “one-size-fits-all” solution, which fails to meet specific application needs and hinders the full exploitation of potential performance. This paper presents a framework to enable dynamic file I/O path selection with fine granularity at runtime. The framework adopts a file handle-rich scheme to allow file systems choose corresponding optimizations to serve I/O requests. Consistency control algorithms are proposed to ensure data consistency while changing optimizations at runtime. One case study on our prototype shows that choosing proper optimizations can improve the I/O performance for small files and large files by up to 40 and 64.4 %, respectively. Another case study shows that the data prefetch performance for real-world application traces can be improved by up to 193 % by selecting correct prefetch patterns. Simulations in large-scale environment also show that our method is scalable and both the memory consumption and the consistency control overhead can be negligible.     

并行文件系统I/O特征

并行文件系统是并行计算系统的存储子系统,I/O性能是并行计算系统研究的重要方面.本文分析了并行文件系统I/O研究的难点,研究了并行文件系统的I/O特征和关键技术,指出了并行文件系统I/O性能研究未来的研究方向,为并行计算系统的设计和优化提供重要参考.     

Accelerating big data analytics on HPC clusters using two-level storage

Data-intensive applications that are inherently I/O bound have become a major workload on traditional high-performance computing (HPC) clusters. Simply employing data-intensive computing storage such as HDFS or using parallel file systems available on HPC clusters to serve such applications incurs performance and scalability issues. In this paper, we present a novel two-level storage system that integrates an upper-level in-memory file system with a lower-level parallel file system. The former renders memory-speed high I/O performance and the latter renders consistent storage with large capacity. We build a two-level storage system prototype with Tachyon and OrangeFS, and analyze the resulting I/O throughput for typical MapReduce operations. Theoretical modeling and experiments show that the proposed two-level storage delivers higher aggregate I/O throughput than HDFS and OrangeFS and achieves scalable performance for both read and write. We expect this two-level storage approach to provide insights on system design for big data analytics on HPC clusters.     

基于机器学习的动态分区并行文件系统性能优化

近年来,随着大数据、云计算技术的发展,应用系统越来越集中,规模亦越来越大,使得存 储系统的性能问题越来越突出。为应对其性能要求,并行文件系统得到了大量的应用。然而现有的并 行文件系统优化方法,大多只考虑应用系统或并行文件系统本身,较少考虑两者之间的协同。该文基 于应用系统在并行文件系统上的访问模式对存储系统的性能有显著影响这一特点,提出基于动态分区 的并行文件系统优化方法。首先,利用机器学习技术来分析挖掘各个性能影响因素和性能指标之间的 关系和规律,生成优化模型。其次,以优化模型为基础,辅助并行文件系统的参数调优工作。最后, 基于 Ceph 存储系统进行原型实现,并设计了三层架构应用系统进行了性能测试,最终达到优化并行 文件系统访问性能的目的。实验结果表明,所提出方法可以达到 85% 的预测优化准确率;在所提出模 型的辅助优化下,并行文件系统的吞吐量性能得到约 3.6 倍的提升。     

Parallel Array Object I/O Support on Distributed Environments

This paper presents a parallel file object environment to support distributed array store on shared-nothing distributed computing environments. Our environment enables programmers to extend the concept of array distributions from memory levels to file levels. It allows parallel I/O that facilitates the distribution of objects in an application. When objects are read and/or written by multiple applications using different distributions, we present a novel scheme to help programmers to select the best data distribution pattern according to a minimum amount of remote data movements for the store of array objects on distributed file systems. Our selection scheme, to the best of our knowledge, is the first work to attempt to optimize the distribution patterns in the secondary storage for HPF-like programs with inter-application cases. This is especially important for a class of problems called multiple disciplinary optimization (MDO) problems. Our test bed is built on an 8-node DEC Farm connected with an ethernet, FDDI, or ATM switch. Our experimental results with scientific applications show that not only our parallel file system can provide aggregate bandwidths, but also our selection scheme effectively reduces the communication traffic for the system.     

并行文件系统的设计

在大规模并行处理巨型机（ＭＰＰ）的设计中，提高Ｉ／Ｏ性能同提高计算能力和通信能力同样重要。并行文件系统（ＰＦＳ）在多个Ｉ／Ｏ结点的多个磁盘上，分布文件系统和文件的磁盘块，将文件读写在计算结点转化成多个对物理块的直接Ｉ／Ｏ请求，利用预读，预分配，磁盘缓冲式区和异步Ｉ／Ｏ增加Ｉ／Ｏ的并发生，在特定的文件使用模式下，也是ＭＰＰ应用的主要Ｉ／Ｏ模式，获得很高的Ｉ／Ｏ效率。     

几个常见分布式文件系统特征分析和性能对比

近年来随着云计算市场规模不断增长,作为云计算平台基础设施的云存储系统也随之显得越来越重要。数以万计的互联网应用已经运行于云计算环境,同时大量不同的应用也即将从传统运行环境转移到云计算平台。不同的互联网应用的存储需求可能不一样。例如:应用中涉及的单个文件大小,文件数量,IO访问模式,读写比率等,都对底层存储系统提出了不同的要求。这说明在云计算环境中,单个文件系统可能无法满足全部应用的存储需求,本文尝试通过在单一云计算平台中部署多个不同分布式文件系统来优化存储系统的总体性能。为了优化混合式文件系统的性能,首先需要分析不同文件系统的性能特征。本文通过量化方法分析了云计算环境下几个常用的分布式文件系统,这些文件系统分别是ceph,moosefs,glusterfs和hdfs。实验结果显示:即使针对同一文件的相同读写操作,不同分布式文件系统之间的性能也差异显著,当单个文件的大小小于256MB时,moosefs的平均写性能比其它几个文件系统高22.3%;当单个文件大小大于256KB时,glusterfs的平均读性能比其它几个文件系统高21.0%。这些结果为设计和实现一个基于以上几个分布式文件系统的混合式文件系统提供了基础。     

Filter Cache:一种提高Lustre I/O性能的方法

高性能计算系统需要一个可靠高效的并行文件系统.Lustre集群文件系统是典型的基于对象存储的集群文件系统,它适合大数据量聚合I/O操作.大文件I/O操作能够达到很高的带宽,但是小文件I/O性能低下.针对导致Lustre的设计中不利于小文件I/O操作的两个方面,提出了Filter Cache方法.在Lustre的OST组件中设计一个存放小文件I/O数据的Cache,让OST端的小文件I/O操作异步进行,以此来减少用户感知的小文件I/O操作完成的时间,提高小文件I/O操作的性能.     

Block I/O Scheduling on Storage Servers of Distributed File Systems

This paper presents a new scheme of I/O scheduling on storage servers of distributed/parallel file systems, for yielding better I/O performance. To this end, we first analyze read/write requests in the I/O queue of storage server (we name them block I/Os), by using our proposed technique of horizontal partition. Then, all block requests are supposed to be divided into multiple groups, on the basis of their offsets. This is to say, all requests related to the same chunk file will be grouped together, and then be satisfied within the same time slot between opening and closing the target chunk file on the storage server. As a result, the time resulted by completing block I/O requests can be significantly decreased, because of less file operations on the corresponding chunk files at the low-level file systems of server machines. Furthermore, we introduce an algorithm to rate a priority for each group of block I/O requests, and then the storage server dispatches groups of I/Os by following the priority order. Consequently, the applications having higher I/O priorities, e.g. they have less I/O operations and small size of involved data, can finish at a earlier time. We implement a prototype of this server-side scheduling in the PARTE file system, to demonstrate the feasibility and applicability of the proposed scheme. Experimental results show that the newly proposed scheme can achieve better I/O bandwidth and less I/O time, compared with the strategy of First Come First Served, as well as other server-side I/O scheduling approaches.     

A GPU-Accelerated In-Memory Metadata Management Scheme for Large-Scale Parallel File Systems

Driven by the increasing requirements of high-performance computing applications,supercomputers are prone to containing more and more computing nodes.Applications running on such a large-scale computing system are likely to spawn millions of parallel processes,which usually generate a burst of I/O requests,introducing a great challenge into the metadata management of underlying parallel file systems.The traditional method used to overcome such a challenge is adopting multiple metadata servers in the scale-out manner,which will inevitably confront with serious network and consistence problems.This work instead pursues to enhance the metadata performance in the scale-up manner.Specifically,we propose to improve the performance of each individual metadata server by employing GPU to handle metadata requests in parallel.Our proposal designs a novel metadata server architecture,which employs CPU to interact with file system clients,while offloading the computing tasks about metadata into GPU.To take full advantages of the parallelism existing in GPU,we redesign the in-memory data structure for the name space of file systems.The new data structure can perfectly fit to the memory architecture of GPU,and thus helps to exploit the large number of parallel threads within GPU to serve the bursty metadata requests concurrently.We implement a prototype based on BeeGFS and conduct extensive experiments to evaluate our proposal,and the experimental results demonstrate that our GPU-based solution outperforms the CPU-based scheme by more than 50％under typical metadata operations.The superiority is strengthened further on high concurrent scenarios,e.g.,the high-performance computing systems supporting millions of parallel threads.     

曙光并行文件系统DPFS的研究与设计

并行文件系统是高性能计算系统中的快速I／O库。它的目的是为并行计算应用提供快速Input／Output的手段。文章总结了并行应用程序的读写特点,其中的关键问题和在并行文件系统的通常使用的技术,并以此为基础设计了面向曙光高性能服务器的曙光并行文件系统（Dawning Parallel File System,DPFS）。     

SNFS并行文件系统负载均衡技术的研究与实现

在大规模集群系统的并行运算环境中,I/O效率一直是影响系统整体性能的关键因素,并行文件系统技术是目前解决I/O性能瓶颈的有效途径之一。介绍当前并行文件系统的发展现状以及并行文件系统的类型,阐述SNFS并行文件系统的架构以及负载均衡DLC(分布式LAN客户端)技术的实现原理,并给出大规模集群系统环境中SNFS文件系统负载均衡技术的实现方法,最后,通过实际应用分析说明该技术在提升I/O性能上的优势。     

并行网络文件系统PNFS性能评测与分析

传统的网络文件系统难以满足高性能计算系统的I/O 需求,并行网络文件系统——PNFS可以有效地解决传统网络文件系统在可扩展性、可用性和性能上存在的问题。首先对PNFS的体系结构进行了设计,实现了元数据服务器与存储服务器的分离,消除了由于集中服务器结构引发的I/O瓶颈问题。然后,对PNFS的原型系统进行了性能测试,并与相同环境下NFS的测试结果进行比较与分析,结果表明PNFS能够为客户端提供并行访问文件数据的能力,有着较高的I/O读写带宽和较低的访问延迟,同时实现了客户端I/O带宽与存储服务器规模之间的线性可扩展关系,能较好地满足高性能计算中的I/O需求。     

Towards minimizing disk I/O contention: A partitioned file assignment approach

One problem with data-intensive computing facilitating is how to effectively manage massive amounts of data stored in a parallel I/O system. The file assignment method plays a significant role in data management. However, in the context of a parallel I/O system, most existing file assignment approaches share the following two limitations. First, most existing methods are designed for a non-partitioned file, while the file in a parallel I/O system is generally partitioned to provide aggregated bandwidth. Second, the file allocation metric, e.g. service time, of most existing methods is difficult to determine in practice, and also these metrics only reflect the static property of the file. In this paper, a new metric, namely file access density is proposed to capture the dynamic property of file access, i.e. disk contention property. Based on file access density definition, this paper introduces a new static file assignment algorithm named MinCPP and its dynamic version DMinCPP, both of which aim at minimizing the disk contention property. Furthermore MinCPP and DMinCPP take the file partition property into consideration by trying to allocate the partitions belonging to the same file onto different disks. By assuming file request arrival follows the Poisson process, we prove the effectiveness of the proposed schemes both analytically and experimentally. The MinCPP presented in this study can be applied to reorganize the files stored in a large-scale parallel I/O system and the DMinCPP can be integrated into file systems which dynamically allocate files in a batch.     

高强度I/O的应用对并行存储系统的挑战和解决方法研究

具有高I/O密集特性的高性能计算应用对高性能计算机存储系统综合性能的要求越来越高.以石油地震勘探数据处理为代表的一类重要应用表现出I/O数据量巨大、I/O访问密度大,对单个磁盘阵列存储部件的读写带宽要求高的特征.在Lustre文件系统中,充当对象存储服务功能的磁盘阵列设备输出带宽的不足将成为阻碍存储系统整体性能发挥的重要因素.针对此类问题,提出了一种缓存管理方法,分别在客户端添加VDISK模块,在OST端添加Cache模块,二者协同提高并行文件系统I/O的输出带宽的使用效率;另外,充分利用客户端空闲内存以及客户端之间的通信带宽,降低应用程序对磁盘阵列设备输出带宽的要求.通过大规模并行模型的验证表明,VDISK提高了实际可用的输出带宽,提高了外部存储系统的I/O效率.     

Reduced Function Set Abstraction (RFSA) for MPI-IO

Programmers productivity has always been overlooked as compared to the performance optimizations in high performance computing (HPC) community. In many parallel programming languages like MPI/MPI-IO, the performance optimizations are provided as various function options to the programmers. In order to write an efficient code, they are required to know the exact usage of the optimization functions, hence programmer productivity is limited. In this paper, we present RFSA, a Reduced Function Set Abstraction based on an existing parallel programming interface (MPI-IO) for I/O. The purpose of RFSA is to hide the performance optimization functions from the application developer, and relieve the application developer from deciding on a specific function. The proposed set of functions rely on a selection algorithm to decide among the most common optimizations provided by MPI-IO. We implement a selection algorithm for I/O functions like read, write, etc., and also merge a set of functions for data types and file views. By running different parallel I/O benchmarks on both medium-scale clusters and NERSC supercomputers, we show an improved programmer productivity (35.7% on average). This approach incurs an overhead of 2–5% for one particular optimization, and shows performance improvement of 17% when a combination of different optimizations is required by an application.     

高性能分布计算的数据管理与优化

在高性能计算程序对海量分布存储数据的操控中,有效的数据管理很重要。该文提出一个新的高性能分布计算的数据管理与优化系统,它包含一个元数据管理系统和存储系统,提供一个容易使用且能自动进行存储访问优化的平台。该平台采用的多存储资源体系结构能够满足性能和存储容量需求,并能自适应地利用当前的I/O优化方法。     

基于单汉字索引的全文检索系统的优化研究

对于按照单汉字建立倒排索引的全文检索系统,最需要解决的问题是如何提高其存储效率和运算速度。本文针对此问题提出了以下优化方法:一是利用参数化的Golomb编码对倒排文件进行压缩;二是对求集合交集的逻辑乘算法进行改进;三是运用并行计算和双缓冲技术。实验结果表明,经过优化后的单汉字全文检索系统已达到实用化的程度。     

Hybrid hierarchy storage system in MilkyWay-2 supercomputer

With the rapid improvement of computation capability in high performance supercomputer system, the imbalance of performance between computation subsystem and storage subsystem has become more and more serious, especially when various big data are produced ranging from tens of gigabytes up to terabytes. To reduce this gap, large-scale storage systems need to be designed and implemented with high performance and scalability.MilkyWay-2 (TH-2) supercomputer system with peak performance 54.9 Pflops, definitely has this kind of requirement for storage system. This paper mainly introduces the storage system in MilkyWay-2 supercomputer, including the hardware architecture and the parallel file system. The storage system in MilkyWay-2 supercomputer exploits a novel hybrid hierarchy storage architecture to enable high scalability of I/O clients, I/O bandwidth and storage capacity. To fit this architecture, a user level virtualized file system, named H²FS, is designed and implemented which can cooperate local storage and shared storage together into a dynamic single namespace to optimize I/O performance in IO-intensive applications. The evaluation results show that the storage system in MilkyWay-2 supercomputer can satisfy the critical requirements in large scale supercomputer, such as performance and scalability.     

基于Lustre文件系统的MPI检查点系统实现技术与性能测试

基于协同式检查点的回卷恢复是在大规模并行计算机系统中得到采用的一项重要容错技术,其性能开销主要为协同协议和检查点映像存储所决定.描述了一个在MPICH2中实现的应用透明的并行检查点系统,相比已有的技术,该系统有以下特点：1）协同协议操作利用了并行应用的近邻通信特性,通过虚连接方法减少协议的处理开销;2）采用Lustre文件系统简化检查点映像文件管理的复杂性;3）通过并行I/O操作提高性能,优化检查点映像的存储过程.实际应用的测试表明,该检查点系统具有较小的运行时间开销和良好的可扩展性.