Performance models and dynamic characteristics analysis for HDFS write and read operations: A systematic view |
| |
Affiliation: | 1. MOE Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an, China;2. Department of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China;3. Faculty of Engineering and Computing, Coventry University, Coventry, UK;1. State Key Laboratory of Advanced Design and Manufacturing for Vehicle Body, Hunan University, Changsha 410082, China;2. School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, China;1. Institute of Oceanography, Hellenic Centre for Marine Research, Anavyssos, Greece;2. Department of Naval Architecture and Marine Engineering, National Technical University of Athens, Zografos, Athens, Greece |
| |
Abstract: | Hadoop has emerged as a successful framework for large-scale data-intensive computing applications. However, there is no research on performance models for the Hadoop Distributed File System (HDFS). Due to the complexity of HDFS and the difficulty of modeling the multiple impact factors for HDFS performance, to establish HDFS performance models based directly on these impact factors is very complicated. In this paper, the relationship between file size and HDFS Write/Read (denoted as W/R for short) throughput, i.e., the average flow rate of a HDFS W/R operation, is studied to build HDFS performance models from a systematic view. Based on the measured data of specially designed experiments (in which HDFS W/R operations can be viewed as single-input single-output systems), a system identification-based approach is applied to construct performance models for HDFS W/R operations under different conditions. Furthermore, dynamic characteristics metrics for HDFS performance are defined, and based on the identified performance models and these metrics, the dynamic characteristics of HDFS W/R operations, such as steady state and overshoot, are studied, and the relationships between impact factors and dynamic characteristics are analyzed. These analysis results can provide effective guidance and implications for the design and configuration of HDFS and Hadoop-based applications. |
| |
Keywords: | HDFS System modeling Performance model Dynamic characteristics System identification |
本文献已被 ScienceDirect 等数据库收录! |
|