多格式海量数据统一存取的索引结构

为提高多格式海量数据统一存取效率,提出了一种基于Hadoop的分布式数据读取模式。并通过对海量数据非主键索引结构的研究,结合统一存取的描述理念,提出了基于HDFS的一种可适用于B-树和R-树及其变种的层次索引结构,改变了原键—值存储在非主键索引结构中的劣势。通过提出Hadoop缓冲策略、基于随机读取的新数据传输模型以及相应的查询处理策略,进一步降低了数据传输开销。实验表明,该系列方法优化了统一存取中随机存取效率,减少了相应的查询响应时间和数据传输开销,提高了多格式海量数据统一存取的性能。

Index structure of unified access in big data of multi-format

This paper proposed a distributed data access mode based on Hadoop underlying the unified access design of the big data of multi-format. Through the research on the structure of the big data with non-primary key index, combining with the theory of the unified access, this paper gave a HDFS-based hierarchical index structure, which was applied to both B-tree and R-tree and their variants, thereby changing the disadvantage of the original key-value storage in the structure with non-primary key index. To further reduce the data transfer overhead, it put forward Hadoop buffering strategy, a new data transfer model based on random read and the corresponding query processing strategy. The experiments show that the series above well improve the efficiency of the unified access and reduce the corresponding query response time and data transfer overhead.