并行I/O技术在气候数值模式中的应用研究

在气候变化数值模拟工作中,气候数值模式运行效率主要受到计算效率和I/O效率的共同影响。目前,模式计算部分已经基本实现并行,计算效率显著提升。随着气候数值模式时空分辨率的提高,对I/O效率的需求也不断增加,数据并行I/O技术已经成为提高模式整体运行效率的有效方法之一。文中深入分析了BCC_AGCM模式串行I/O算法及NetCDF数据结构特点,采用基于MPI-IO的高层I/O库对模式I/O算法进行并行优化,优化后可支持多类气象要素并行输出,输出效率明显提升。为我国应对气候变化数值模式的运行效率优化工作,进行了有益的技术探索和积累。     

MPI-IO中Collective I／O的优化

本文对MPI-IO库中Collective I／O的实现算法Two-phase I／O提出了改进。通过选取主联络进程来减少第一阶段进程间的通信量，从而减少Two-phase I／O方法在通信过程中的时间消耗，提高了整体的I／O性能。     

集群环境下I/O文件传输调度器的设计与实现

近年来研究人员对高性能计算中的并行I/O问题进行了深入研究,然而这些研究主要针对MPP问题,而对集群计算机系统中并行I/O问题的研究不多。因此,对于集群计算中的并行I/O系统进行研究是一个重要的研究课题。对集群计算中的并行I/O传输调度效率进行研究,设计了一个文件传输调度器,可以实现文件传输最快捷,节点资源最大利用,显著提高I/O节点吞吐率和反应时间。经过大量数据的测试和实验证明该调度器的有效性和适用性。     

SNFS并行文件系统负载均衡技术的研究与实现

在大规模集群系统的并行运算环境中,I/O效率一直是影响系统整体性能的关键因素,并行文件系统技术是目前解决I/O性能瓶颈的有效途径之一。介绍当前并行文件系统的发展现状以及并行文件系统的类型,阐述SNFS并行文件系统的架构以及负载均衡DLC(分布式LAN客户端)技术的实现原理,并给出大规模集群系统环境中SNFS文件系统负载均衡技术的实现方法,最后,通过实际应用分析说明该技术在提升I/O性能上的优势。     

分组技术及其在集群中的应用

在集群系统中,I/O节点担负着数据存储和分发的角色。针对I/O密集型应用,I/O节点网络带宽常常无法满足需要而成为整个系统的瓶颈。分组技术可以方便、高效地增加I/O节点带宽,并提供冗余保护。本文简要介绍了分组技术原理、模式,并通过在集群I/O节点上安装infel公司提供的iANS具体实现。最后给出了性能评价。此技术在实际应用中已取得良好效果。     

协作型虚拟机中设备虚拟化研究

针对当前系统虚拟化中设备模型难以通用、I/O执行效率低等问题,提出一种基于多核平台的设备虚拟化方案。将传统的消息式I/O处理转换成实时I/O处理,通过I/O传递协议消除I/O处理部件与虚拟机监控器(VMM)的紧耦合,在改善I/O处理效率的同时,提高设备虚拟化模型的通用性,结合协作型VMM完成对设备虚拟化的设计与实现。测试结果表明,该方案能提升I/O执行效率,适用于多种虚拟化系统。     

支持异构集群并行的高能物理数据处理系统

传统集群计算系统无法充分利用本地磁盘的存储资源和I/O,大量网络I/O成为系统瓶颈,导致资源利用率降低,并造成高昂的存储和网络成本.使用Hadoop处理分析作业可有效利用本地磁盘存储和I/O资源,而集群资源统一管理工具Mesos则使用轻量化的设计和高效的通信机制,能在不同计算集群之间动态共享集群资源.为此,分析高能物理数据处理的特点,利用Mesos构建异构集群间资源共享的高能物理实验数据处理系统,实现Torque/Maui和Hadoop集群的集成.测试结果表明,该系统能够在集群间动态分配集群资源,并利用本地存储和磁盘I/O显著降低网络I/O,提高集群资源利用率.     

IO负载均衡方式的创新及集群系统管理工具的开发与应用

在石油领域的高性能计算应用中会产生大量、频繁的I/O操作。因此,集群系统中的I/O负载问题一直是长期以来PC架构集群实现高性能以及提升效率的瓶颈。＂ZQ集群系统管理工具＂这套管理工具有效地解决了传统IO负载管理方式存在的问题,克服了常用集群系统管理软件存在的不足。此外,这套管理工具还具有诸多大大简化系统管理的创新功能。     

大规模集群文件系统中的I/O拥塞控制机制

首先研究了分布式集群存储系统中如何为各个客户端提供公平有效的I/O服务的问题,然后针对大规模集群文件系统提出了一种分布式I/O拥塞控制的策略.在拥塞控制下,当服务器轻载时,可以让单个客户端并行地发出更多的I/O请求给服务器,以达到最大化网络和服务器资源的利用率以及I/O吞吐率的目的；当服务器重载时,通过一种节流控制,限...     

PVFS代码结构及并行Meta服务研究

PVFS(ParallelVirtualFileSystem)广泛应用于PC集群并行计算环境中,通过ROMIO形式的MPI-IO接口与MPICH结合,用于提高数据文件的访问性能,通过对PVFS的源代码分析,得出PVFS的系统架构、运行机制与采用的策略,并在这个基础上找出元数据服务器并行化的可行性,以设计出并行元数据服务方案来提高元数据访问的吞吐率。     

Reduced Function Set Abstraction (RFSA) for MPI-IO

Programmers productivity has always been overlooked as compared to the performance optimizations in high performance computing (HPC) community. In many parallel programming languages like MPI/MPI-IO, the performance optimizations are provided as various function options to the programmers. In order to write an efficient code, they are required to know the exact usage of the optimization functions, hence programmer productivity is limited. In this paper, we present RFSA, a Reduced Function Set Abstraction based on an existing parallel programming interface (MPI-IO) for I/O. The purpose of RFSA is to hide the performance optimization functions from the application developer, and relieve the application developer from deciding on a specific function. The proposed set of functions rely on a selection algorithm to decide among the most common optimizations provided by MPI-IO. We implement a selection algorithm for I/O functions like read, write, etc., and also merge a set of functions for data types and file views. By running different parallel I/O benchmarks on both medium-scale clusters and NERSC supercomputers, we show an improved programmer productivity (35.7% on average). This approach incurs an overhead of 2–5% for one particular optimization, and shows performance improvement of 17% when a combination of different optimizations is required by an application.     

An adaptive cache coherence protocol specification for parallel input/output systems

Caching has been intensively used in memory and traditional file systems to improve system performance. However, the use of caching in parallel file systems and I/O libraries has been limited to I/O nodes to avoid cache coherence problems. We specify an adaptive cache coherence protocol that is very suitable for parallel file systems and parallel I/O libraries. This model exploits the use of caching, both at processing and I/O nodes, providing performance improvement mechanisms such as aggressive prefetching and delayed-write techniques. The cache coherence problem is solved by using a dynamic scheme of cache coherence protocols with different sizes and shapes of granularity. The proposed model is very appropriate for parallel I/O interfaces, such as MPI-IO. Performance results, obtained on an IBM SP2, are presented to demonstrate the advantages offered by the cache management methods proposed.     

Analysis of I/O Performance on an Amazon EC2 Cluster Compute and High I/O Platform

Cloud computing is currently being explored by the scientific community to assess its suitability for High Performance Computing (HPC) environments. In this novel paradigm, compute and storage resources, as well as applications, can be dynamically provisioned on a pay-per-use basis. This paper presents a thorough evaluation of the I/O storage subsystem using the Amazon EC2 Cluster Compute platform and the recent High I/O instance type, to determine its suitability for I/O-intensive applications. The evaluation has been carried out at different layers using representative benchmarks in order to evaluate the low-level cloud storage devices available in Amazon EC2, ephemeral disks and Elastic Block Store (EBS) volumes, both on local and distributed file systems. In addition, several I/O interfaces (POSIX, MPI-IO and HDF5) commonly used by scientific workloads have also been assessed. Furthermore, the scalability of a representative parallel I/O code has also been analyzed at the application level, taking into account both performance and cost metrics. The analysis of the experimental results has shown that available cloud storage devices can have different performance characteristics and usage constraints. Our comprehensive evaluation can help scientists to increase significantly (up to several times) the performance of I/O-intensive applications in Amazon EC2 cloud. An example of optimal configuration that can maximize I/O performance in this cloud is the use of a RAID 0 of 2 ephemeral disks, TCP with 9,000 bytes MTU, NFS async and MPI-IO on the High I/O instance type, which provides ephemeral disks backed by Solid State Drive (SSD) technology.     

自主协助动态并行I/O调度策略研究

随着处理器速度和网络传输速度的快速提升，I／O成为限制计算机性能的一大瓶颈，并行I／O提供了解决这一问题的有效途径。在并行I／O的实现中，数据访问调度策略对系统的性能有着重要的影响。文章通过对集群计算中现有I／O数据访问调度策略的研究，针对调度器瓶颈问题，提出了自主协助动态I／O调度策略，实验结果表明了所提出策略的有效性。     

集群计算系统中并行I/O模拟器的研究与设计

论文对集群计算系统中的并行I／O技术作了分析，在理论研究基础上提出并行I／O模拟器的框架模型及研究、设计该模拟器的思路，并开发了一个并行I／O模拟器CLU—PIO—SIM的部分功能。     

基于Linux的SMP机群环境中并行I/O模型研究

提出了一个基于数据通路的波浪推进式并行I/O模型框架,并在基于Linux的SMP机群系统中,根据波浪推进式并行I/O模型框架对各个数据通路进行建模,具体分析了这个波浪推进式并行I/O模型,从思路上解决了刻画并行I/O性能的并行I/O模型问题。