基于大规模廉价计算平台的海量数据处理系统的研究*

提出一种基于大规模廉价计算平台的海量数据处理模型,吸取了Map/Reduce计算模式和大规模分布式数据存储机制Bigtable的基本思想,实现了以数据为中心的计算密集型的经济性超级计算系统平台。系统选择电信部门的大规模业务数据为分析对象,对电信通话和数据业务的大规模数据集进行处理,从而向运营商和普通用户提供有价值的数据分析服务。该平台适用于其他多种海量数据的分布式处理,为其他的各种应用提供了一个具有良好参考价值的示范。     

基于DM2的可扩展DBMS体系结构研究

目前,日益变化的数据管理要求和新型数据库应用的出现,要求DBMS具有良好的可扩展性与可伸缩性,能方便快速地加入新的数据库功能,以适应新的应用需求。DM2是拥有国产自主版权的分布式DBMS系统,它采用当前最流行的客户/服务器模型结构。文章介绍了一种基于DM2的可扩展DBMS体系结构及数据库组件化技术的应用。     

Retrospective on Aurora

This experience paper summarizes the key lessons we learned throughout the design and implementation of the Aurora stream-processing engine. For the past 2 years, we have built five stream-based applications using Aurora. We first describe in detail these applications and their implementation in Aurora. We then reflect on the design of Aurora based on this experience. Finally, we discuss our initial ideas on a follow-on project, called Borealis, whose goal is to eliminate the limitations of Aurora as well as to address new key challenges and applications in the stream-processing domain.Received: 21 October 2003, Accepted: 16 April 2004, Published online: 14 September 2004Edited by: J. Gehrke and J. Hellerstein.     

An integrated data base management system for engineering applications based on an extended relational model

Engineering analysis and design of complex systems require the use of large software components. Development, maintenance, and extension of such a software system needs modern design and implementation techniques. Usually a large amount of data is generated. Flow of data is also quite complex, adding further complications in maintenance and extension of the software. A sophisticated data base management system (DBMS) is needed to support data handling during the run-time environment as well as for the integration of various software components. Design and development of such a DBMS needs new concepts and ideas such that efficiency of calculations is not sarcrificed. Degradation in efficiency due to the use of a DBMS can hinder large-scale applications. The paper describes a generalized relational model to handle large matrices and tables that are encountered in numerous engineering applications. A DBMS based on the model is designed and implemented. The system supports run-time data management as well as data sharing among software components. A preliminary evaluation of the system against some existing ones reveals the new concept and design to be quite appropriate for engineering applications. The system is very efficient and compact. Some details of design and performance of the system are given and discussed.     

一种基于对象关系模型的时空数据库管理系统体系结构

时空数据库的关键与难点在于其实现技术．本文提出了一种基于对象关系模型的优化型时空数据库管理系统体系结构，该体系结构采用时空数据类型扩展和时空操作扩展技术对数据库管理系统的内核进行扩充，使其具有内建的时空数据管理能力，同时以时空查询优化层实现时空查询的逻辑优化，解决了底层数据库管理系统的查询优化问题。     

Comparing columnar,row and array DBMSs to process recursive queries on graphs

Analyzing graphs is a fundamental problem in big data analytics, for which DBMS technology does not seem competitive. On the other hand, SQL recursive queries are a fundamental mechanism to analyze graphs in a DBMS, whose processing and optimization are significantly harder than traditional SPJ queries. Columnar DBMSs are a new faster class of database system, with significantly different storage and query processing mechanisms compared to row DBMSs, still the dominating technology. With that motivation in mind, we study the optimization of recursive queries on a columnar DBMS focusing on two fundamental and complementary graph problems: transitive closure and adjacency matrix multiplication. From a query processing perspective we consider the three fundamental relational operators: selection, projection and join (SPJ), where projection subsumes SQL group-by aggregation. We present comprehensive experiments comparing recursive query processing on columnar, row and array DBMSs to analyze large graphs with different shape and density. We study the relative impact of query optimizations and we compare raw speed of DBMSs to evaluate recursive queries on graphs. Results confirm classical query optimizations that keep working well in a columnar DBMS, but their relative impact is different. Most importantly, a columnar DBMS with tuned query optimization is uniformly faster than row and array systems to analyze large graphs, regardless of their shape, density and connectivity. On the other hand, there is no clear winner between the row and array DBMSs.     

Toward cost-effective storage provisioning for DBMSs

Data center operators face a bewildering set of choices when considering how to provision resources on machines with complex I/O subsystems. Modern I/O subsystems often have a rich mix of fast, high performing, but expensive SSDs sitting alongside with cheaper but relatively slower (for random accesses) traditional hard disk drives. The data center operators need to determine how to provision the I/O resources for specific workloads so as to abide by existing service level agreements, while minimizing the total operating cost (TOC) of running the workload, where the TOC includes the amortized hardware costs and the run-time energy costs. The focus of this paper is on introducing this new problem of TOC-based storage allocation, cast in a framework that is compatible with traditional DBMS query optimization and query processing architecture. We also present a heuristic-based solution to this problem, called DOT. We have implemented DOT in PostgreSQL, and experiments using TPC-H and TPC-C demonstrate significant TOC reduction by DOT in various settings.     

嵌入式数据库在通信系统中的应用研究

本文介绍了嵌入式数据库的典型应用,以及在某著名通讯企业各个交换设备普遍采用的嵌入式数据库系统的结构。本文综合了通讯行业普遍采用的数据库结构以及结合关系型数据库和网络模型数据库的特点,提出了基于事务处理和面向对象的轻量级实时嵌入式数据库的基本架构。