支持混合事务和分析处理的数据库管理系统综述

数据库管理系统根据应用场景分为事务型(OLTP)系统和分析型(OLAP)系统.随着实时数据分析需求增长, OLTP任务和OLAP任务混合的场景越来越普遍,业界开始重视支持混合事务和分析处理(HTAP)的数据库管理系统.这种HTAP数据库系统除了需要满足高性能的事务处理外,还需要满足实时分析对数据新鲜度的要求.因此,对数据库系统的设计与实现提出了新的挑战.近年来,在工业界和学术界涌现了一批架构多样、技术各异的原型和产品.综述HTAP数据库的背景和发展现状,并且从存储和计算的角度对现阶段的HTAP数据库进行分类.在此基础上,按照从下往上的顺序分别总结HTAP系统在存储和计算方面采用的关键技术.在此框架下介绍各类系统的设计思想、优劣势以及适用的场景.此外,结合HTAP数据库的评测基准和指标,分析各类HTAP数据库的设计与其呈现出的性能与数据新鲜度的关联.最后,结合云计算、人工智能和新硬件技术为HTAP数据库的未来研究和发展提供思路.     

HTAP数据库系统数据共享模型和优化策略

混合事务与分析处理数据库系统(HTAP)因其在一套系统上可以同时处理混合负载而逐渐获得大众认可. 为了不影响在线事务处理(OLTP)业务的写入性能, HTAP数据库系统往往会通过维护数据多版本或额外副本的方式来支持在线分析处理(OLAP)任务, 从而引入了TP/AP端版本的数据一致性问题. 同时, HTAP数据库系统面临资源隔离下实现高效数据共享的核心挑战, 且数据共享模型的设计综合权衡了业务对性能和数据新鲜度之间的要求. 因此, 为了系统地阐释现有HTAP数据库系统数据共享模型及优化策略, 首先根据TP生成版本与AP查询版本的差异, 通过一致性模型定义数据共享模型, 将HTAP数据共享的一致性模型分为3类, 分别为线性一致性, 顺序一致性与会话一致性. 然后, 梳理数据共享模型的全流程, 即从数据版本标识号分配, 数据版本同步, 数据版本追踪3个核心问题出发, 给出不同一致性模型的实现方法. 进一步, 以典型的HTAP数据库系统为例对具体实现进行深入的阐释. 最后, 针对数据共享过程中涉及的版本同步、追踪、回收等模块的优化策略进行归纳和分析, 并展望数据共享模型的优化方向, 指出数据同步范围自适应, 数据同步周期自调优和顺序一致性的新鲜度阈值约束控制是提高HTAP数据库系统性能和新鲜度的可能手段.     

云原生数据库综述

云基础设施的虚拟化、高可用、可弹性调度等特点,为云数据库提供了开箱即用、可靠可用、按需计费等优势.云数据库按照架构可以划分为云托管数据库(cloud-hosted database)以及云原生数据库(cloud-native database).云托管数据库将数据库系统直接部署到云上虚拟机环境中,具备低成本、易运维、高可靠的优势.在此基础上,云原生数据库充分利用云基础设施弹性伸缩的特点,采用计算存储分离的架构,实现了计算资源和存储资源的独立伸缩,进一步提升数据库性价比.然而计算存储分离的架构为数据库系统设计带来了新的挑战.深入分析云原生数据库系统的架构和技术.首先将云原生OLTP和云原生OLAP的数据库架构按照资源分离模式的差异分别进行归类分析,并对比各类架构的优势与局限.其次,基于计算存储分离的架构,按照各个功能模块深入探讨云原生数据库的关键技术:主要包括云原生OLTP关键技术(数据组织、副本一致性、主备同步、故障恢复以及混合负载处理)和云原生OLAP关键技术(存储管理、查询处理、无服务器感知计算、数据保护以及机器学习优化).最后,总结现有云原生数据库的技术挑战并展望未来研究方向.     

OLAP技术及其在SQL SERVER 2000中的实现

OLTP系统积累的大量数据如何为决策分析提供支持将成为下一阶段数据库应用的重点,OLAP技术通过数据立方体提供多维度的数据视图,并通过旋转切片等操作扩展查询语言的功能来满足这种需求.文中简要介绍了联机分析处理(On line Analysis Process)中的基本概念、OLAP与传统联机事务处理OLTP的差别、OLAP的相关实现技术,并给出了OLAP在SQL SERVER 2000中的具体实现.最后对OLAP的发展前景作出展望.     

基于时间戳实化视图动态维护技术的研究

由于源数据的不稳定性,其结构和数据的变化必须及时传播到实体化视图中,以保持实体化视图与源数据变化的一致性,否则会降低实体化视图中数据的新鲜度,并影响OLAP查询结果的真实性和有效性.为此,本文提出了基于时间戳的动态视图维护技术.该技术采用版本链控制技术,通过时间戳的控制进一步使视图更新和查询的同步进行,有效地解决了由于OLTP更新事务和OLAP事务同时访问数据所发生冲突的问题,在满足视图联机实时维护的同时,更好的提高了数据仓库的新鲜度和OLAP的查询效率.     

OLAP：将数据转化成信息的技术

作为对联机事务处理(OLTP,On-Line Transaction Processing)进行补充的一种信息技术,联机分析处理(OLAP,On-Line Analytical Processing)正开始在现代管理过程中被人们大量使用。OLAP技术的概念最早由E.F.Codd在1993年提出。OLTP系统处理的对象是大量的事务,每个事务中有相对小容量细节数据,而OLAP系统则注重于对相对大容量的、主要是聚合的数据进行分析。OLAP是行政信息系统(Executive Information System)进一步演变的方向,它使用户脱离了表单,脱离了传统关系数据库的制约。 多维数据服务器(MDD)是OLAP技术的基础,是表单模型的一个自然数据库服务器扩展。MDD用于     

气象大数据分布式存储设计与实现

随着气象数据种类、数量日益繁多,范围尺度越来越大,在海量气象数据场景下,无论传统数据库还是文件系统技术的查询和存储方式都不能很好地满足气象数据的高性能查询要求.基于大数据技术,以关系型数据存储、分布式NoSQL数据库存储、网格存储系统和分布式NAS存储相结合的混合云存储架构为基础,搭建了统一数据集、处理和服务的省局气象...     

OLAP:将数据转化成信息的技术

作为对联机事务处理（OLTP，On－LineTransactionPro－cessing）进行补充的一种信息技术，联机分析处理（OLAP，On－LineAnalyticalProcessing）开始在现代管理过程中被人们大量使用。OLAP技术的概念最早由E.F.Codd在1993年提出。OLTP系统处理的对象是大量的事务，每个事务中有相对小容量的细节数据；而OLAP系统则注重于对相对大容量的，主要是聚合的数据进行分析。OLAP是行政信息系统（ExecutiveInformationSystem）进一步演变的方向，它使用户脱离了表单，脱离了传统关系数据库的制约。多维数据服务器（MDD）是OLAP技术…     

基于Oracle的OLTP与OLAP数据库设计及实现

介绍了OLAP和OLTP处理系统的概念,根据这两种数据库应用在实时性、并发性及数据量大小等方面的不同,数据库在设计方面侧重的技术各有不同,阐述了数据库设计技术,即内存设计、变量绑定、SQL并行执行、表分区存储、磁盘IO能力等设计技术在这两种数据库中。     

OLAP技术及其在SQL SERVER 2000中的实现

OLAP系统积累的大量数据如何为决策分析提供支持将成为下一阶段数据库应用的重点，OLAP技术通过数据立方体提供多维度的数据视图，并通过旋转切片等操作扩展查询语言的功能来满足这种需求。文中简要介绍了联机分析处理(On line Analysis Process)中的基本概念、OLAP与传统联机事务处理OLTP的差别、OLAP的相关实现技术，并给出了OLAP在SQL SERVER 2000中的具体实现。最后对OLAP的发展前景作出展望。     

Spatial data warehouses and spatial OLAP come towards the cloud: design and performance

Cloud computing systems handle large volumes of data by using almost unlimited computational resources, while spatial data warehouses (SDWs) are multidimensional databases that store huge volumes of both spatial data and conventional data. Cloud computing environments have been considered adequate to host voluminous databases, process analytical workloads and deliver database as a service, while spatial online analytical processing (spatial OLAP) queries issued over SDWs are intrinsically analytical. However, hosting a SDW in the cloud and processing spatial OLAP queries over such database impose novel obstacles. In this article, we introduce novel concepts as cloud SDW and spatial OLAP as a service, and afterwards detail the design of novel schemas for cloud SDW and spatial OLAP query processing over cloud SDW. Furthermore, we evaluate the performance to process spatial OLAP queries in cloud SDWs using our own query processor aided by a cloud spatial index. Moreover, we describe the cloud spatial bitmap index to improve the performance to process spatial OLAP queries in cloud SDWs, and assess it through an experimental evaluation. Results derived from our experiments revealed that such index was capable to reduce the query response time from 58.20 up to 98.89 %.     

A mixed transaction processing and operational reporting benchmark

The importance of reporting is ever increasing in today’s fast-paced market environments and the availability of up-to-date information for reporting has become indispensable. Current reporting systems are separated from the online transaction processing systems (OLTP) with periodic updates pushed in. A pre-defined and aggregated subset of the OLTP data, however, does not provide the flexibility, detail, and timeliness needed for today’s operational reporting. As technology advances, this separation has to be re-evaluated and means to study and evaluate new trends in data storage management have to be provided. This article proposes a benchmark for combined OLTP and operational reporting, providing means to evaluate the performance of enterprise data management systems for mixed workloads of OLTP and operational reporting queries. Such systems offer up-to-date information and the flexibility of the entire data set for reporting. We describe how the benchmark provokes the conflicts that are the reason for separating the two workloads on different systems. In this article, we introduce the concepts, logical data schema, transactions and queries of the benchmark, which are entirely based on the original data sets and real workloads of existing, globally operating enterprises.     

基于OLAP技术的零售业经营分析系统模型的研究

数据仓库和联机分析处理(OLAP)技术是信息技术领域的新兴技术。使用OLAP技术可以将零售业中大量源数据有效地转化为有用的决策信息,并服务于决策过程。文中根据零售业经营分析业务提出的基于三层系统的实现方案,解决了传统的OLTP系统所无法解决的大量历史数据的使用问题以及用户所面临的基于大量历史数据统计问题。     

数据仓库及联机分析处理技术

数据仓库和联机分析处理（Ｏｎｉｎｅ Ａｎａｌｙｔｉｃａｌ Ｐｒｏｃｅｓｓｉｎｇ,简称ＯＬＡＰ）是决策支持系统的有机组成部分。本语文介绍了数据仓库、数据集市、联机分析处理的概念,分析并比较了建立数据仓库的三种策略及联机分析处理系统的三种结构,在文章结尾我们给出了联机分析处理系统的两个新的发展方向－ＷＥＢ
ＯＬＡＰＯＬＡＰ＋数据挖掘。     

High Performance OLAP and Data Mining on Parallel Computers

On-Line Analytical Processing (OLAP) techniques are increasingly being used in decision support systems to provide analysis of data. Queries posed on such systems are quite complex and require different views of data. Analytical models need to capture the multidimensionality of the underlying data, a task for which multidimensional databases are well suited. Multidimensional OLAP systems store data in multidimensional arrays on which analytical operations are performed. Knowledge discovery and data mining requires complex operations on the underlying data which can be very expensive in terms of computation time. High performance parallel systems can reduce this analysis time. Precomputed aggregate calculations in a Data Cube can provide efficient query processing for OLAP applications. In this article, we present algorithms for construction of data cubes on distributed-memory parallel computers. Data is loaded from a relational database into a multidimensional array. We present two methods, sort-based and hash-based for loading the base cube and compare their performances. Data cubes are used to perform consolidation queries used in roll-up operations using dimension hierarchies. Finally, we show how data cubes are used for data mining using Attribute Focusing techniques. We present results for these on the IBM-SP2 parallel machine. Results show that our algorithms and techniques for OLAP and data mining on parallel systems are scalable to a large number of processors, providing a high performance platform for such applications.     

MiNT-OLAP cluster: minimizing network transmission cost in OLAP cluster for main memory analytical database

Powerful storage, high performance and scalability are the most important issues for analytical databases. These three factors interact with each other, for example, powerful storage needs less scalability but higher performance, high performance means less consumption of indexes and other materializations for storage and fewer processing nodes, larger scale relieves stress on powerful storage and the high performance processing engine. Some analytical databases (ParAccel, Teradata) bind their performance with advanced hardware supports, some (Asterdata, Greenplum) rely on the high scalability framework of MapReduce, some (MonetDB, Sybase IQ, Vertica) highlight performance on processing engine and storage engine. All these approaches can be integrated into an storage-performance-scalability (S-P-S) model, and future large scale analytical processing can be built on moderate clusters to minimize expensive hardware dependency. The most important thing is a simple software framework is fundamental to maintain pace with the development of hardware technologies. In this paper, we propose a schema-aware on-line analytical processing (OLAP) model with deep optimization from native features of the star or snowflake schema. The OLAP model divides the whole process into several stages, each stage pipes its output to the next stage, we minimize the size of output data in each stage, whether in central processing or clustered processing. We extend this mechanism to cluster processing using two major techniques, one is using NetMemory as a broadcasting protocol based dimension mirror synchronizing buffer, the other is predicate-vector based DDTA-OLAP cluster model which can minimize the data dependency of star-join using bitmap vectors. Our OLAP model aims to minimize network transmission cost (MiNT in short) for OLAP clusters and support a scalable but simple distributed storagemodel for large scale clustering processing. Finally, the experimental results show the speedup and scalability performance.     

Integration of engineering change objects in product data management databases to support engineering change analysis

This study proposes a product data management (PDM) database that can support engineering change analysis (ECA). It can integrate ECA with the existing main product development process managed by PDM systems. Since engineering change (EC) history is a key EC data element that enables ECA, this study extends PDM databases to represent the EC history with existing entities for ECs and associated products and their product structures. To show the feasibility of the proposed PDM database, this study integrates a prototype PDM system with on-line analytical processing (OLAP) tools and a data mining module for ECA. It also applies the implemented tools to two typical ECA applications, EC evaluation and EC propagation problems. The illustrative application examples show that the proposed PDM database can support ECA through multidimensional data analysis with OLAP and data mining with association rules.