基于纠删码和动态副本策略的HDFS改进系统

为了让Hadoop分布式文件系统(HDFS)达到更高的存储效率以及更加优化的负载均衡能力，针对HDFS的多副本存储技术提出了改进方案——Noah。Noah引入了编码和译码模块，对HDFS中的block进行编码分解，生成更多数量的数据分片(section)，并随机地分散保存到集群当中，替代原有系统的多副本容灾策略；在集群出现节点失效的情况下，通过收集与失效block相关的任意70%左右的section进行原始数据的恢复；同时根据分布式集群运行情况以及对副本数目需求的不同采用动态副本策略。通过相关的集群实验，表明Noah在容灾效率、负载均衡、存储成本以及安全性上对HDFS作了相应的优化。

Improved HDFS scheme based on erasure code and dynamical-replication system

In order to improve the storage efficiency of Hadoop Distributed File System(HDFS) and its load balance ability,this paper presented an improved solution named Noah to replace the original multiple-replication strategy.Noah introduced a coding module to HDFS.Instead of adopting the multiple-replication strategy by the original system,the module encoded every data block of HDFS into a greater number of data sections(pieces),and saved them dispersedly into the clusters of the storage system in distributed fashion.In the case of cluster failure,the original data would be recovered via decoding by collecting any 70% of the sections,while the dynamic replication strategy also worked synchronously,in which the amount of copies would dynamically change with the demand.The experimental results in analogous clusters of storage system show the feasibility and advantages of new measures in proposed solution.