基于EHDFS的海量小文件存储与检索方法

为有效解决HDFS面对多类型的海量小文件存在存储效率与检索速率低下的问题,构建一种基于EHDFS架构的存取方案。存储阶段,引入最优化策略,建立新的合并存储模型,使小文件最大化填满且均匀分布于Block,提高DataNode空间利用,降低NameNode内存开销。检索阶段,改进MapFile映射关系结构、索引存储位置与组成元素以建立新的文件索引模型,避免跨跃式文件搜索,实现小文件的集中检索。实验结果表明,对比多种大数据存储模型,在不同数据量的分组压力测试下,该方案有效提高了HDFS的存取效率。

Storage and retrieval method of massive small files based on EHDFS

To effectively solve the problems of low storage efficiency and retrieval speed of HDFS in the face of multi-type massive small files,an improved access scheme based on EHDFS architecture was constructed.In the storage stage,a merge storage model based on the optimization strategy was established to maximize the file filling that even distributed on the block,the DataNode space utilization was improved,and the NameNode memory consumption was reduced.In the retrieval stage,by improving the MapFile mapping structure,index storage location and components,a file index model was established to avoid skipping file searches and realize centralized retrieval of small files.Experimental results show that the scheme effectively improves the access efficiency of HDFS under the pressure of grouping test with different data volumes,compared with various big data storage models.