Data modeling in the NoSQL world

NoSQL systems have gained their popularity for many reasons, including the flexibility they provide in organizing data, as they relax the rigidity provided by the relational model and by the other structured models. This flexibility and the heterogeneity that has emerged in the area have led to a little use of traditional modeling techniques, as opposed to what has happened with databases for decades.In this paper, we argue how traditional notions related to data modeling can be useful in this context as well. Specifically, we propose NoAM (NoSQL Abstract Model), a novel abstract data model for NoSQL databases, which exploits the commonalities of various NoSQL systems. We also propose a database design methodology for NoSQL systems based on NoAM, with initial activities that are independent of the specific target system. NoAM is used to specify a system-independent representation of the application data and, then, this intermediate representation can be implemented in target NoSQL databases, taking into account their specific features. Overall, the methodology aims at supporting scalability, performance, and consistency, as needed by next-generation web applications.     

NoSQL数据库与关系数据库的比较分析

介绍了两个具有代表性的NoSQL数据库:Bigtable和Dynamo系统。首先,描述了Bigtable和Dynamo的适用范围及其产生原因。Bigtable和Dynamo可以高效的处理web数据提供相应服务;然后,介绍了Bigtable和Dynamo系统的架构、特性等,以及各自独特的设计方法。最后,将这两个数据库与传统的关系数据库进行比较分析,描述了它们之间的不同点,对比结果表明NoSQL数据库在处理web应用数据时是高效可用的,比传统关系数据库更占优势。     

An integration approach of hybrid databases based on SQL in cloud computing environment

As the applications with big data in cloud computing environment grow, many existing systems expect to expand their service to support the dramatic increase of data, and modern software development for services computing and cloud computing software systems is no longer based on a single database but on existing multidatabases and this convergence needs new software architecture design. This paper proposes an integration approach to support hybrid database architecture, including MySQL, MongoDB, and Redis, to make it possible of allowing users to query data simultaneously from both relational SQL systems and NoSQL systems in a single SQL query. Two mechanisms are provided for constructing Redis's indexes and semantic transforming between SQL and MongoDB API to add the SQL feature for these NoSQL databases. With the proposed approach, hybrid database systems can be performed in a flexible manner, ie, access can be either relational database or NoSQL, depending on the size of data. The approach can effectively reduce development complexity and improve development efficiency of the software systems with multidatabases. This is the result of further research on the related topic, which fills the gap ignored by relevant scholars in this field to make a little contribution to the further development of NoSQL technology.     

Uniform data access platform for SQL and NoSQL database systems

Integration of data stored in heterogeneous database systems is a very challenging task and it may hide several difficulties. As NoSQL databases are growing in popularity, integration of different NoSQL systems and interoperability of NoSQL systems with SQL databases become an increasingly important issue. In this paper, we propose a novel data integration methodology to query data individually from different relational and NoSQL database systems. The suggested solution does not support joins and aggregates across data sources; it only collects data from different separated database management systems according to the filtering options and migrates them. The proposed method is based on a metamodel approach and it covers the structural, semantic and syntactic heterogeneities of source systems. To introduce the applicability of the proposed methodology, we developed a web-based application, which convincingly confirms the usefulness of the novel method.     

NoSQL数据库间数据交换代价研究

随着互联网时代的到来,IT行业迅猛发展,NoSQL数据库以其在大数据环境下出色的业务处理处理能力,在IT行业内得到越来越广泛的应用。而各NoSQL数据库由于自身数据模型的不同,在数据组织方式上彼此存在差异。NoSQL数据库间进行数据交换时,数据模型的不同会导致数据库间数据传输的阻抗,以源数据库数据模型封装的业务数据可能无法直接被目标数据库解析,需进行额外的模型适配操作,参照目标数据库数据模型组织业务数据以供筛选存储。为此,拟定义一种数据描述模型,对NoSQL数据库数据模型特征建模,描述NoSQL数据库的数据组织方式,并定义NoSQL数据库数据模型间距离评估算法。根据数据描述模型与距离评估算法可设计实现一种通用数据模型,其在数据交换过程中可与相关NoSQL数据库进行数据模型上的转换,系统相关业务代码只需参照该数据模型设计,而独立于数据交换过程中NoSQL数据库具体的数据模型。     

一种基于中间件的异构数据库融合访问方法及系统

在大数据时代,信息化数据呈爆炸式增长,传统关系型数据库和新兴的NoSQL数据库都难以全面且高效地面对这些挑战。因此,提出一种基于中间件的异构数据库访问方法(MingleDB),以结合NoSQL和传统关系型数据库的优点。MingleDB透明融合了NoSQL数据库和传统数据库的主要运行逻辑,同时又能够根据当前用户请求的读写特征,自动选取合适的处理路径以避免二者的不足;它还支持轻量级的事务处理框架,该框架按需实施以保证异构数据库数据的最终一致性和完整性。将MingleDB分别与MongoDB,MySQL数据库进行读写性能对比,实验证明了MingleDB方法的正确性和合理性。同时将MingleDB部署在实际的社交网络系统中进行实际验证,结果亦证明了其实用性和可移植性。     

BASE analysis of NoSQL database

NoSQL databases are designed to address performance and scalability requirements of web based application which cannot be addressed by traditional relational databases. Due to their contrast in priorities and architecture to conventional relational databases using SQL, these databases are referred as “NoSQL” databases since they ​incorporate lots of additional features in addition to the features of conventional databases. The relational databases strongly follow the ACID (Atomicity, Consistency, Isolation, and Durability) properties while the NoSQL databases follow BASE (Basically Available, Soft State, Eventual consistency) principles. This survey paper is an analytical study on BASE features of some of NoSQL databases.     

A new fragment re-allocation strategy for NoSQL database systems

NoSQL databases are famed for the characteristics of high scalability, high availability, and high fault-tolerance. So NoSQL databases are used in a lot of applications. The data partitioning strategy and fragment allocation strategy directly affect NoSQL database systems’ performance. The data partition strategy of large, global databases is performed by horizontally, vertically partitioning or combination of both. In the general way the system scatters the related fragments as possible to improve operations’ parallel degree. But the operations are usually not very complicated in some applications, and an operation may access to more than one fragment. At the same time, those fragments which have to be accessed by an operation may interact with each other. The general allocation strategies will increase system’s communication cost during operations execution over sites. In order to improve those applications’ performance and enable NoSQL database systems to work efficiently, these applications’ fragments have to be allocated in a reasonable way that can reduce the communication cost i.e., to minimize the total volume of data transmitted during operations execution over sites. A strategy of clustering fragments based on hypergraph is proposed, which can cluster fragments which were accessed together in most operations to the same cluster. Themethod uses a weighted hypergraph to represent the fragments’ access pattern of operations. A hypergraph partitioning algorithmis used to cluster fragments in our strategy. This method can reduce the amount of sites that an operation has to span. So it can reduce the communication cost over sites. Experimental results confirm that the proposed technique will effectively contribute in solving fragments re-allocation problem in a specific application environment of NoSQL database system.     

新型数据管理系统研究进展与趋势

随着各类新型计算技术和新兴应用领域的浮现,传统数据库技术面临新的挑战,正在从适用常规应用的单一处理方法逐步转为面向各类特殊应用的多种数据处理方式.分析并展望了新型数据管理系统的研究进展和趋势,涵盖分布式数据库、图数据库、流数据库、时空数据库和众包数据库等多个领域.具体而言：分布式数据管理技术是支持可扩展的海量数据处理的关键技术;以社交网络为代表的大规模图结构数据的处理需求带来了图数据库技术的发展;流数据管理技术用来应对数据动态变化的管理需求;时空数据库主要用于支持移动对象管理;对多源、异构而且劣质数据源的集成需求催生出新型的众包数据库技术.最后讨论了新型数据库管理系统的未来发展趋势.     

NoSQL data management systems

In the last decade, a new class of data management systems collectively called NoSQL systems emerged and are now intensively developed. The main feature of these systems is that they abandon the relational data model and the SQL, do not fully support ACID transactions, and use distributed architecture (even though there are non-distributed NoSQL systems as well). As a result, such systems outperform the conventional SQL-oriented DBMSs in some applications; in addition, such systems are highly scalable under increasing workloads and huge amounts of data, which is important, in particular, for Web applications. Unfortunately, the absence of transactional semantics imposes certain constraints on the class of applications where NoSQL systems can be effectively used and the choice of a particular system significantly depends on the application. In this paper, a review of the main classes of NoSQL data management systems is given and examples of systems and applications where they can be used are discussed.     

传统RDBMS向非关系型MongoDB数据模型转换与数据迁移方法研究

为解决关系型数据库在大数据处理中遇到的瓶颈问题,满足企业对大数据处理的需求,提出将关系型数据库迁移到NoSQL文档型数据库中。针对RDBMS中的关系模型向MongoDB中的集合模型转化方法进行了研究,提出了表示关系间参照完整性的有向图表示模型,和基于关系型数据模型向MongoDB文档模型自动转化算法;实现了RDBMS中迁移数据到MongoDB的插入算法。针对上述方案和算法,结合典型开源RDBMS--MySQL实例,对上述关系有向图模型的生成、基于有向图模型的转化算法以及数据迁移算法应用验证。实验结果表明RDBMS可以按照一定的数据结构平滑地迁移到MongoDB中。     

Distributed snapshot maintenance in wide-column NoSQL databases using partitioned incremental ETL pipelines

Wide-column NoSQL databases are an important class of NoSQL (Not only SQL) databases which scale horizontally and feature high access performance on sparse tables. With current trends towards big Data Warehouses (DWs), it is attractive to run existing business intelligence/data warehousing applications on higher volumes of data in wide-column NoSQL databases for low latency by mapping multidimensional models to wide-column NoSQL models or using additional SQL add-ons. For examples, applications like retail management can run over integrated data sets stored in big DWs or in the cloud to capture current item-selling trends. Many of these systems also employ Snapshot Isolation (SI) as a concurrency control mechanism to achieve high throughput for read-heavy workloads. SI works well in a DW environment, as analytical queries can now work on (consistent) snapshots and are not impacted by concurrent update jobs performed by online incremental Extract-Transform-Load (ETL) flows that refresh fact/dimension tables. However, the snapshot made available in the DW is often stale, since at the moment when an analytical query is issued, the source updates (e.g. in a remote retail store) may not have been extracted and processed by the ETL process in time due to high input data volume or slow processing speed. This staleness may cause incorrect results for time-critical decision support queries. To address this problem, snapshots which are supposed to be accessed by analytical queries need to be first maintained by corresponding ETL flows to reflect source updates based on given freshness needs. Snapshot maintenance in this work means maintaining the distributed data partitions that are required by a query. Since most NoSQL databases are not ACID compliant and do not provide full-fledged distributed transaction support, snapshot may be inconsistently derived when its data partitions are updated by different ETL maintenance jobs.This paper describes an extended version of HBelt system [1] which tightly integrates the wide-column NoSQL database HBase with a clustered & pipelined ETL engine. Our objective is to efficiently refresh HBase tables with remote source updates while a consistent snapshot is guaranteed across distributed partitions for each scan request in analytical queries. A consistency model is defined and implemented to address so-called distributed snapshot maintenance. To achieve this, ETL jobs and analytical queries are scheduled in a distributed processing environment. In addition, a partitioned, incremental ETL pipeline is introduced to increase the performance of ETL (update) jobs. We validate the efficiency gain in terms of data pipelining and data partitioning using the TPC-DS benchmark, which simulates a modern decision support system for a retail product supplier. Experimental results show that high query throughput can be achieved in HBelt when distributed, refreshed snapshots are demanded.     

基于对象关系模型的HLA仿真数据库设计

为克服传统的HLA仿真数据库基于关系数据库开发、数据对象的结构及相互间关系需在设计阶段指定、一旦开发完成、存在数据模式难以修改和扩展等缺点,基于HLA仿真数据按面向对象设计的特点,研究了利用对象关系数据库技术开发结构可灵活扩展的HLA仿真数据库的问题,提出了基于HLA对象模型设计、实现、管理和使用可扩展的对象关系型仿真数据库的技术方法.应用于作战仿真领域的实际系统中,结构可扩展的对象关系型HLA仿真数据库在使用和维护上更加灵活、方便,适于HLA仿真领域的高级数据库应用.     

NoSQL数据管理技术

社交网络和微博等新型应用对数据管理技术提出了新的挑战,如海量数据高效存储、高并发访问、高可扩展性和高可用性等。而传统的关系数据库技术无法满足这些新型应用的需求,因此,NoSQL数据管理技术的研究、开发和应用越来越受到重视。本文从NoSQL数据模型、数据存储、查询处理以及SQL与NoSQL混合数据库解决方案等方面,综述了NoSQL数据管理技术发展现状和趋势,并介绍了几种典型的NoSQL产品。     

Performance evaluation of NoSQL big-data applications using multi-formalism models

Starting with the birth of Web 2.0, the quantity of data managed by large-scale web services has grown exponentially, posing new challenges and infrastructure requirements. This has led to new programming paradigms and architectural choices, such as map-reduce and NoSQL databases, which constitute two of the main peculiarities of the specialized massively distributed systems referred to as Big Data architectures. The underlying computer infrastructures usually face complexity requirements, resulting from the need for efficiency and speed in computing over huge evolving data sets. This is achieved by taking advantage from the features of new technologies, such as the automatic scaling and replica provisioning of Cloud environments. Although performances are a key issue for the considered applications, few performance evaluation results are currently available in this field. In this work we focus on investigating how a Big Data application designer can evaluate the performances of applications exploiting the Apache Hive query language for NoSQL databases, built over a Apache Hadoop map-reduce infrastructure.This paper presents a dedicated modeling language and an application, showing first how it is possible to ease the modeling process and second how the semantic gap between modeling logic and the domain can be reduced, by means of vertical multiformalism modeling.     

Pronto: High availability for standard off-the-shelf databases

Enterprise applications typically store their state in databases. If a database fails, the application is unavailable while the database recovers. Database recovery is time consuming because it involves replaying the persistent transaction log. To isolate end users from database failures we introduce Pronto, a protocol to orchestrate the transaction processing by multiple, standard databases so that they collectively implement the illusion of a single, highly available database. Pronto is a novel replication protocol that handles non-determinism without relying on perfect failure detection, does not require any modifications in existing applications and databases, and allows databases from different providers to be part of the replicated compound.     

PostgreSQL与MongoDB处理非结构化数据性能比较

互联网技术的发展产生的海量非结构化数据在传统关系型数据库中难以被高速有效地进行存储和处理,各类NoSQL数据库可以有效存储处理非结构化数据,但是对关系运算功能的弱化难以满足应用场景的需求。具备非结构化数据处理能力的新型关系型数据库提供了适用多种应用场景的高效存储方式。为了能够定量地比较关系型数据库和面向文档的NoSQL数据库的数据存储与处理能力,比较了PostgreSQL的hstore数据类型和MongoDB的内嵌文档对非结构化数据的储存方式,并通过非结构化数据的批量加载、磁盘占用、主键查询、非主键查询、地理空间坐标查询等方面的对比来以分析性能特征与适用场景。     

NoSQL real-time database performance comparison

The amount of data being produced is increasing constantly, as the number and variety of connected devices are growing and the advances in data storage and mining are supporting this evolution. However, storing and handling high quantities of data is challenging the current Relational Database Management Systems. Big Data and its related products came to help in this matter, and the NoSQL databases arise with the purpose to offer better solutions and features to handle massive amounts of data with higher performance, sometimes near real-time. The present study presents the NoSQL databases scenario and background, and elaborates a detailed study with the characteristics, a features comparison and a performance evaluation of three different NoSQL databases extensively used in the market nowadays: Couchbase, MongoDB and RethinkDB. Tests were performed in two different scenarios: single thread and multiple threads. The results reveal that Couchbase had a better performance at most of the operations, except for retrieving multiple documents and inserting documents with multiple threads, operations in which MongoDB scored better.     

面向NoSQL数据库的JSON文档异常检测与语义消歧模型

随着信息化技术的发展,面对材料等相关领域数据的多源异构、扩展性强、爆炸增长等特点,传统关系数据库无法对数据进行存储,因此可利用NoSQL的无模式存储、高扩展性等特性来解决这一难题。作为NoSQL数据库常用的数据存储格式,JSON因简单性和灵活性备受欢迎。然而,NoSQL数据库缺乏模式信息,在JSON文档存入数据库之前,需要对其进行数据验证与分析。目前,大多数方法是基于JSON schema对JSON文档格式的规范性进行校验,无法有效解决JSON文档的异常检测以及语义歧义问题。为此,文中提出了面向NoSQL数据库的JSON文档异常检测与语义消歧模型doctorJSON。该模型基于JSON schema对存入的JSON文档分别设计了异常检测算法deoutJSON和语义消歧算法disemaJSON,以检测JSON文档存在的异常和歧义。在真实数据集与合成数据集上的实验验证了所提模型的有效性和执行效率。