基于新型存储器件的分布式文件系统性能优化

新型存储器件的I/O性能通常比传统固态驱动器（SSD）高一个数量级，然而使用新型存储器件的分布式文件系统相对于使用SSD的分布式文件系统性能并没有显著的提高，这说明目前的分布式文件系统并不能充分发挥新型存储器件的性能。针对这个问题，对Hadoop分布式文件系统（HDFS）的数据写入流程及传输过程进行了量化分析。通过量化分析HDFS数据写入过程各阶段的时间开销，发现在写入数据的各个阶段中，节点间数据传输的时间占比较大。因此提出了对应的优化方案，通过异步写入的方式并行化数据传输与处理过程，使得不同数据包的处理阶段叠加起来，减少了数据包整体的处理时间，从而提升了HDFS的写入性能。实验结果表明，所提方案将HDFS的写入吞吐量提升了15%~24%，总体的写入执行时间降低了28%~36%。

Performance optimization of distributed file system based on new type storage devices

The I/O performance of new type storage devices is usually an order of magnitude higher than that of traditional Solid State Disk (SSD). However, simply replacing SSD with new type storage device will not significantly improve the performance of distributed file system. This means that the current distributed file system cannot give full play to the performance of new type storage devices. To solve the problem, the data writing process and transmission process of Hadoop Distributed File System (HDFS) were analyzed quantitatively. Through quantitative analysis of the time consumptions of different stages of HDFS writing process, the most time-consuming data transmission between nodes was found in each stage of writing data. Therefore, the corresponding optimization strategy was proposed, that is, the processes of data transmission and processing were parallelized by using asynchronous write. So that the processing stages of different data packets were parallel to each other, shortening the total processing time of data writing, thereby the write performance of HDFS was improved. Experimental results show the proposed scheme improves the HDFS write throughput by 15%-24%, and reduces the overall write execution time by 28%-36%.