一种面向HDFS的多层索引技术

SOH（SQL over HDFS）系统通常将数据存储于分布式文件系统HDFS中，采用Map/Reduce或分布式查询引擎来处理查询任务。得益于HDFS以及Map/Reduce的容错能力和可扩展性，SOH系统可以很好地应对数据规模的飞速增长，完成分析型查询处理。然而，在处理选择型查询或交互式查询时，这类系统暴露出性能上的缺陷。本文提出一个通用的索引技术，可以应用于SOH系统中，以提高其查询处理的效率。分析了SOH系统访问HDFS文件的过程，指出了其中影响数据加载时间的关键因素；提出了split层和split内部双层索引机制；设计并实现了聚集索引和非聚集索引。最后，在标准数据集上进行了大量实验，并与现有基于HDFS的索引技术进行了比较。实验结果表明，所提出的索引技术可以有效地提高查询处理的效率。

Multi-Layered Index for HDFS-Based Systems

The SOH (SQL over HDFS) systems usually store the data into distributed file systems like HDFS,and process queries by the Map/Reduce computing framework or distributed database query engine.Benefitting from the fault tolerance and scalability provided by Map/Reduce and HDFS,SOH systems perform well in processing analytical queries over big data.However,the efficiency of such systems is too low to meet the requirement of selective queries or interactive queries,which have strict limit on the query response time.This paper proposes a HDFS-based index,called HIndex,for SOH systems.HIndex can easily integrated into the existing SOH systems to improve the efficiency of query evaluation.We analyze the process that SOH systems access data stored in HDFS,and highlight the important factors affecting the time cost.We then propose a two-layer index structure,and implement both aggregated and non-aggregated index technique.According the experiments conducted on standard datasets,HIndex performs much better than Hadoop++,a state-of-the-art HDFS-based index.