面向时序数据的两阶段日志结构合并树文件合并框架

时序数据库中日志结构合并树（LSM-tree）在高写入负载或资源受限情况下的不及时的文件合并会导致LSM的C₀层数据大量堆积,从而造成近期写入数据的即席查询延迟增加。针对上述问题,提出了一种在保持面向大块数据的高效查询的基础上实现对最新写入的时序数据的低延迟查询的两阶段LSM合并框架。首先将文件的合并过程分为少量乱序文件快速合并与大量小文件合并这两个阶段,然后在每个阶段内提供多种文件合并策略,最后根据系统的查询负载进行两阶段合并的资源分配。通过在时序数据库Apache IoTDB上分别实现传统的LSM合并策略以及两阶段LSM合并框架和测试,结果表明与传统的LSM相比,两阶段的文件合并模块在提升策略灵活性的情况下使即席查询读盘次数大大降低,并且使历史数据分析查询性能提升了约20%。实验结果表明,两阶段的LSM合并框架能够提高近期写入数据的即席查询效率,提高历史数据分析查询性能,而且提升合并策略的灵活性。     

一种性能优化的小文件存储访问策略的研究

在分布式文件系统中,小文件的管理一般存在访问性能较差和存储空间浪费较大等缺点.为了解决这些问题,提出了一种性能优化的小文件存储访问(SFSA)策略.SFSA将逻辑上连续的数据尽可能存储在物理磁盘的连续空间,使用Cache充当元数据服务器的角色并通过简化的文件信息节点提高Cache利用率,提高了小文件访问性能;写数据时聚合更新数据及其文件夹域中的相关数据为一次I/O请求写入,减少了文件碎片数量,提高了存储空间利用率;文件传输时利用局部性原理,提前发送批量的高访问率的小文件,降低了建立网络连接开销,提升了文件传输性能.理论分析和实验证明,SFSA的设计思想和方法能有效地优化小文件的存储访问性能.     

Flexisample:个性化近似聚合查询系统

大数据交互式查询分析对于查询时延具有较高需求,基于采样技术的近似计算服务通过牺牲一定的准确性可以获得较少的查询时延,其在大数据近似查询分析方面具有良好的普适性和广阔的应用前景.论文所述系统Flexisample是一个基于采样技术的个性化近似聚合查询系统,实现了针对查询请求的解析重写和逻辑样本组合策略,使其可以满足个性化的多维聚合查询需求.为了在满足多样个性化聚合查询请求的同时保证一定的准确率,Flexisample维护了一组优化设计后的分层样本,并且为了扩大样本在时间维度上的覆盖范围,系统利用在线数据流对分层样本进行维护与更新.将系统应用于电能质量数据聚合查询,结果表明:针对多个个性化聚合查询请求和查询时延约束,系统可以在满足业务人员个性化查询需求的同时有效降低查询时延,在时间消耗仅为全量查询不足7％的条件下,全部分层的查询准确率均达到了88％以上,样本存储空间相比直接存储减少了87.5％.     

面向海量高清视频数据的高性能分布式存储系统

通过对视频监控数据的特点和传统存储方案进行分析,提出一种高性能分布式存储系统解决方案.不同于传统的基于文件存储的方式,设计了一种逻辑卷结构,将非结构化的视频流数据以此结构进行组织并直接写入RAW磁盘设备,解决了传统存储方案中随机磁盘读写和磁盘碎片导致存储性能下降的问题.该方案将元数据组织为两级索引结构,分别由状态管理器和存储服务器管理,极大地减少了状态管理器需要管理元数据的数量,消除了性能瓶颈,并提供精确到秒级的检索精度.此外,该方案灵活的存储服务器分组策略和组内互备关系使得存储系统具备容错能力和线性扩展能力.系统测试结果表明,该方案在成本低廉的PC服务器上实现了单台服务器能同时记录400路1080P视频流,写入速度是本地文件系统的2.5倍.     

面向轨迹流数据的索引构建与存储方法研究

移动社交网络等基于定位服务应用的快速发展导致时空数据流规模呈爆炸式增长,要求底层数据存储系统支持高吞吐量轨迹数据的插入以及空间和时间约束下的低延迟查询,而现有HBase等数据存储方案因索引更新开销过高无法满足该需求。针对时空数据流的应用特性,提出一种数据流内存索引及存储方法。根据键值和时间范围对历史与增量数据元组进行物理分区,将其以模板B+树的形式写入内存并构建索引以增强快速写入和查询能力,同时对数据进行压缩存储提升索引效率。在此基础上,采用多级索引根据数据分区将复杂查询分解为可独立处理的子查询。实验结果表明,与传统HBase、WaterWheel等方法相比,该方法在不同数据插入和查询条件下的数据存储性能与查询效率更优。     

一种基于多时间粒度的实时客流查询优化算法

随时间实时变化的客流数据属于时间序列数据,本文根据客流数据的接收频率,应用关系模型实现客流数据的存储建模;为了减弱数据采集频率对实时客流查询效率的影响,建立多时间粒度的客流视图,可提高实时客流查询的计算效率.     

基于HDFS的小文件存储与读取优化策略

本文对HDFS分布式文件系统进行了深入的研究,在HDFS中以流式的方式访问大文件时效率很高但是对海量小文件的存取效率比较低. 本文针对这个问题提出了一个基于关系数据库的小文件合并策略,首先为每个用户建立一个用户文件,其次当用户上传小文件时把文件的元数据信息存入到关系数据库中并将文件追加写入到用户文件中,最后用户读取小文件时通过元数据信息直接以流式方式进行读取. 此外当用户读取小于一个文件块大小的文件时还采取了数据节点负载均衡策略,直接由存储数据的DataNode传送给客户端从而减轻主服务器压力提高文件传送效率. 实验结果表明通过此方案很好地解决了HDFS对大量小文件存取支持不足的缺点,提高了HDFS文件系统对海量小文件的读写性能,此方案适用于具有海量小文件的云存储系统,可以降低NameNode内存消耗提高文件读写效率.     

BESⅢ分布式计算的元数据管理

北京谱仪III(BESIII)高能物理实验产生PB量级的实验数据,海量数据的处理和分析对计算资源提出较大挑战。分布式计算是整合异构计算资源和解决计算资源短缺的可行方案。根据BESIII实验需求对分布式计算所需的元数据管理进行研究,提出数据文件的元数据模型,利用中间件软件DIRAC的目录服务设计并实现元数据管理系统。该系统利用树型目录结构、物理文件名动态构建和虚拟数据集等技术,组织和存储各种类型的元数据,实现查询请求、逻辑文件以及物理文件之间的映射,使用数字证书和开放安全套接层协议保证系统安全。将该系统应用于实验数据分析和处理中,测试结果表明,当并发用户访问量为300时,查询时间仅为0.3 s,证明该系统性能较好,可以满足BESIII实验的应用需要。     

非均匀数据分布下的MapReduce连接查询算法优化

MapReduce分布式计算框架有助于提升大规模数据连接查询的效率,但当连接属性分布不均匀时,其简单的散列策略容易导致计算节点间负载不均衡,影响作业的整体性能。针对连接查询操作中的数据倾斜问题,研究了MapReduce框架下大规模数据连接查询操作的优化算法。首先对经典的改进重分区连接查询算法进行实验分析,研究了传统MapReduce计算框架下连接查询操作的执行流程,找出了基于MapReduce计算框架的连接查询算法在数据分布不均匀时的性能瓶颈;进而提出了组合分割平衡分区优化策略,设计并实现了基于组合分割平衡分区优化策略的改进型连接查询算法。实验结果表明,提出的优化策略在大规模数据的连接查询处理上很好地解决了数据倾斜带来的性能影响,具有好的时间性能和可扩展性。     

分布式聚集函数支持的内存OLAP并行查询处理技术

根据OLAP查询的特点和内存数据库的性能特征提出了由多个内存数据库组成的并行OLAP查询处理系统,将OLAP应用中的多维聚集查询分布到各个计算节点并行进行聚集计算,并将聚集计算的结果进行合并输出.与其他并行处理方法相比,该算法充分利用OLAP DB结构中维表远小于事实表的特性,根据数据库中事实表的数据量和节点的数据处理能力进行水平数据库分片,并根据聚集函数的可分布计算特性提高查询处理的并行度,延迟并行查询处理中的合并过程,充分利用节点的并行处理能力,减少并行查询处理过程中的数据通信量,提高系统并行查询处理性能.该算法易于实现,具有较好的可扩展性和性能,适用于企业级海量数据处理领域的需求.     

‘MaaS’: fast retrieval of E-file in cloud using metadata as a service

In cloud era as the data stored is enormous, efficient retrieval of data with reduced latency plays a major role. In cloud, owing to the size of the stored data and lack of locality information among the stored files, metadata is a suitable method of keeping track of the storage. This paper describes a novel framework for efficient retrieval of data from the cloud data servers using metadata with less amount of time. Performance of queries due to availability of files for query processing can be greatly improved by the efficient use of metadata and its analysis thereof. Hence this paper proposes a generic approach of using metadata in cloud, named ‘MaaS—Metadata as a Service’. The proposed approach has exploited various methodologies in reducing the latency during data retrieval. This paper investigates the issues on creation of metadata, metadata management and analysis of metadata in a cloud environment for fast retrieval of data. Cloud bloom filter, a probabilistic data structure used for efficient retrieval of metadata is stored across various metadata servers dispersed geographically. We have implemented the model in a cloud environment and the experimental results show that methodology used is efficient on increasing the throughput and also by handling large number of queries efficiently with reduced latency. The efficacy of the approach is tested through experimental studies using KDD Cup 2003 dataset. In the experimental results, proposed ‘MaaS’ has outperformed other existing methods.     

基于对象存储的集群存储系统设计

集群存储是解决大规模数据存储的重要方法。本文提出一种基于对象存储的集群存储系统结构,将文件分为目录路径元数据、文件元数据与数据对象三部分并独立管理。性能比较与分析表明,该方法能够支持超大规模的文件及超大容量的目录,明显地减少网络访问消息数量,提高访问性能,并且解决了因为修改目录而导致的大量元数捂迁移问题。     

A GPU-Accelerated In-Memory Metadata Management Scheme for Large-Scale Parallel File Systems

Driven by the increasing requirements of high-performance computing applications,supercomputers are prone to containing more and more computing nodes.Applications running on such a large-scale computing system are likely to spawn millions of parallel processes,which usually generate a burst of I/O requests,introducing a great challenge into the metadata management of underlying parallel file systems.The traditional method used to overcome such a challenge is adopting multiple metadata servers in the scale-out manner,which will inevitably confront with serious network and consistence problems.This work instead pursues to enhance the metadata performance in the scale-up manner.Specifically,we propose to improve the performance of each individual metadata server by employing GPU to handle metadata requests in parallel.Our proposal designs a novel metadata server architecture,which employs CPU to interact with file system clients,while offloading the computing tasks about metadata into GPU.To take full advantages of the parallelism existing in GPU,we redesign the in-memory data structure for the name space of file systems.The new data structure can perfectly fit to the memory architecture of GPU,and thus helps to exploit the large number of parallel threads within GPU to serve the bursty metadata requests concurrently.We implement a prototype based on BeeGFS and conduct extensive experiments to evaluate our proposal,and the experimental results demonstrate that our GPU-based solution outperforms the CPU-based scheme by more than 50％under typical metadata operations.The superiority is strengthened further on high concurrent scenarios,e.g.,the high-performance computing systems supporting millions of parallel threads.     

ROARS: a robust object archival system for data intensive scientific computing

As scientific research becomes more data intensive, there is an increasing need for scalable, reliable, and high performance storage systems. Such data repositories must provide both data archival services and rich metadata, and cleanly integrate with large scale computing resources. ROARS is a hybrid approach to distributed storage that provides both large, robust, scalable storage and efficient rich metadata queries for scientific applications. In this paper, we present the design and implementation of ROARS, focusing primarily on the challenge of maintaining data integrity across long time scales. We evaluate the performance of ROARS on a storage cluster, comparing to the Hadoop distributed file system and a centralized file server. We observe that ROARS has read and write performance that scales with the number of storage nodes, and integrity checking that scales with the size of the largest node. We demonstrate the ability of ROARS to function correctly through multiple system failures and reconfigurations. ROARS has been in production use for over three years as the primary data repository for a biometrics research lab at the University of Notre Dame.     

Spatiotemporal aggregate computation: a survey

Spatiotemporal databases are becoming increasingly more common. Typically, applications modeling spatiotemporal objects need to process vast amounts of data. In such cases, generating aggregate information from the data set is more useful than individually analyzing every entry. In this paper, we study the most relevant techniques for the evaluation of aggregate queries on spatial, temporal, and spatiotemporal data. We also present a model that reduces the evaluation of aggregate queries to the problem of selecting qualifying tuples and the grouping of these tuples into collections on which an aggregate function is to be applied. This model gives us a framework that allows us to analyze and compare the different existing techniques for the evaluation of aggregate queries. At the same time, it allows us to identify opportunities for research on types of aggregate queries that have not been studied.     

Efficient Relational Techniques for Processing Graph Queries

Graphs are widely used for modeling complicated data such as social networks, chemical compounds, protein interactions and semantic web. To effectively understand and utilize any collection of graphs, a graph database that efficiently supports elementary querying mechanisms is crucially required. For example, Subgraph and Supergraph queries are important types of graph queries which have many applications in practice. A primary challenge in computing the answers of graph queries is that pair-wise comparisons of graphs are usually hard problems. Relational database management systems (RDBMSs) have repeatedly been shown to be able to efficiently host different types of data such as complex objects and XML data. RDBMSs derive much of their performance from sophisticated optimizer components which make use of physical properties that are specific to the relational model such as sortedness, proper join ordering and powerful indexing mechanisms. In this article, we study the problem of indexing and querying graph databases using the relational infrastructure. We present a purely relational framework for processing graph queries. This framework relies on building a layer of graph features knowledge which capture metadata and summary features of the underlying graph database. We describe different querying mechanisms which make use of the layer of graph features knowledge to achieve scalable performance for processing graph queries. Finally, we conduct an extensive set of experiments on real and synthetic datasets to demonstrate the efficiency and the scalability of our techniques.