| Abstract: | In recent years, big data has been one of the hottest development directions in the information field. With the development of artificial intelligence technology, mobile
smart terminals and high-bandwidth wireless Internet, various types of data are increasing exponentially. Huge amounts of data contain a lot of potential value, therefore
how to effectively store and process data efficiently becomes very important. Hadoop Distributed File System (HDFS) has emerged as a typical representative of dataintensive distributed big data file systems, and it has features such as high fault tolerance, high throughput, and can be deployed on low-cost hardwares. HDFS nodes
communicate with each other to make the big data systems work properly, using the Remote Procedure Call (RPC) mechanism. However, the RPC in HDFS is still not
good enough to work better in terms of network throughput and abnormal response. This paper presents an optimization method to improve the performance of HDFS.
The proposed method dynamically adjusts the RPC configurations between NameNode and DataNodes by sensing the data characters that stored in DataNodes. This
method can effectively reduce the NameNode processing pressure, and improve the network throughput generated by the information transmission between NameNode
and DataNodes. It can also reduce the abnormal response time of the whole system. Finally, the extensive experiments show the effectiveness and efficiency of our
proposed method. |
