基于NoSQL的海量航空物流小文件分布式多级存储方法

为了解决航空物流领域海量小文件存储效率和访问效率不高的问题，提出一种基于Nosql的海量小文件分布式多级存储方法，充分考虑到数据的时效性、本地性、操作的并发性以及文件之间的相关性，先根据相关性将文件合并，然后采用分布式多级存储，使用内存式Redis数据库做缓存，HDFS做数据的持久化存储，其过程采用预取机制。实验结果表明，该方法有效提高了小文件的存取效率和磁盘的利用率，显著地降低了网络的带宽占用和集群NameNode的内存消耗，适合解决航空领域海量小文件存储问题。

Method of distributed multi-level storage of massive small files of air logistics based on Nosql

In order to solve the problem of inefficient storage and inefficient access of massive small files in the field of aviation logistics, in this paper, a method of distributed multi-level storage of massive small files based on Nosql, is proposed, which fully considers the data timeliness, the data locality, the concurrency of operation and the correlation between files, according to the correlation, merging file, and then a distributed multi-level storage is applied, and which applies memory-type Redis database to the data-cache and Hdfs to permanently store the data with using prefetching mechanism.Experimental results show that,the method effectively improves the access efficiency of small files and the utilization of disk, and also significantly reduces the occupation of network bandwidth and the memory consumption of NameNode cluster, and is also suitable for solving problems of storage of massive small files in the field of aviation.