Reducing I/O variability using dynamic I/O path characterization in petascale storage systems |
| |
Authors: | Seung Woo Son Saba Sehrish Wei-keng Liao Ron Oldfield Alok Choudhary |
| |
Affiliation: | 1.University of Massachusetts Lowell,Lowell,USA;2.Fermi National Accelerator Laboratory,Batavia,USA;3.Northwestern University,Evanston,USA;4.Sandia National Laboratory,Albuquerque,USA |
| |
Abstract: | In petascale systems with a million CPU cores, scalable and consistent I/O performance is becoming increasingly difficult to sustain mainly because of I/O variability. The I/O variability is caused by concurrently running processes/jobs competing for I/O or a RAID rebuild when a disk drive fails. We present a mechanism that stripes across a selected subset of I/O nodes with the lightest workload at runtime to achieve the highest I/O bandwidth available in the system. In this paper, we propose a probing mechanism to enable application-level dynamic file striping to mitigate I/O variability. We implement the proposed mechanism in the high-level I/O library that enables memory-to-file data layout transformation and allows transparent file partitioning using subfiling. Subfiling is a technique that partitions data into a set of files of smaller size and manages file access to them, making data to be treated as a single, normal file to users. We demonstrate that our bandwidth probing mechanism can successfully identify temporally slower I/O nodes without noticeable runtime overhead. Experimental results on NERSC’s systems also show that our approach isolates I/O variability effectively on shared systems and improves overall collective I/O performance with less variation. |
| |
Keywords: | |
本文献已被 SpringerLink 等数据库收录! |
|