减少重建数据量的冗余编码技术研究

为防止硬件故障或机器宕机导致的数据丢失,冗余编码技术被广泛应用于分布式存储系统中来保证数据的可靠性。然而,传统的冗余编码技术,如里德-所罗门码,存在着重建数据量大的问题。副本技术在重建丢失数据时只需要读取和传输丢失的数据,而冗余编码需要读取和传输更大的数据量,从而消耗更多的磁盘I/O带宽和网络带宽。因此,基于冗余编码的分布式存储系统在重建数据时将消耗更长的时间,从而将整个系统长时间暴露在一种降级的模式下,进而增加了发生永久性数据丢失的风险。为解决这个问题,减少重建数据量的冗余编码技术不断被提出,然而只有这些冗余编码与传统的里德-所罗门码的比较,缺少它们在存储系统的综合比较。系统地从减少重建数据量等几个重要方面研究了这些减少重建数据量的冗余编码技术,从而为实际系统中采用合适的编码提供重要参考和依据。

Researches of Redundancy Coding Technologies on Reducing Reconstruction Data Amount

In order to avoid data loss due to hardware failure or server breakdown,redundancy coding technology is widely employed in distributed storage systems for data reliability.However,traditional erasure codes,such as Reed-Solomon codes,bear the burden of huge rebuilding data amount.Compared with the replication technique,which only needs to read and transfer the lost data,the erasure coding requires to read and transfer a much large amount of data,thereby consuming much more disk I/Os and network bandwidth.Thus,a erasure code based distributed storage system would cost longer time for data reconstruction than a replication based system,and exposes the whole system in a long-term degraded stage,increasing the risk of the permanent data loss.To solve this problem,many repair-bandwidth-efficient codes were constantly proposed,but these codes are only compared with the traditional Reed-Solomon codes and lack the comprehensive comparisons on practical storage systems.We systematically analyzed these repair-bandwidth-efficient codes from the some significant aspects,such as amount reduction on reconstruction data and so on,thus providing valuable basis and references for choosing suitable erasure codes for practical systems.