NoSQL数据管理技术

社交网络和微博等新型应用对数据管理技术提出了新的挑战,如海量数据高效存储、高并发访问、高可扩展性和高可用性等。而传统的关系数据库技术无法满足这些新型应用的需求,因此,NoSQL数据管理技术的研究、开发和应用越来越受到重视。本文从NoSQL数据模型、数据存储、查询处理以及SQL与NoSQL混合数据库解决方案等方面,综述了NoSQL数据管理技术发展现状和趋势,并介绍了几种典型的NoSQL产品。     

Data adapter for querying and transformation between SQL and NoSQL database

As the growing of applications with big data in cloud computing become popular, many existing systems expect to expand their service to support the explosive increase of data. We propose a data adapter system to support hybrid database architecture including a relational database (RDB) and NoSQL database. It can support query from application and deal with database transformation at the same time. We provide three modes of query approach in data adapter system: blocking transformation mode (BT mode), blocking dump mode (BD mode), and direct access mode (DA mode). We provide a data synchronization mechanism and describe the design and implementation in detail. This paper focuses on velocity with proposed three modes and partly variety with data stored in RDB, NoSQL database and temporary files. With the proposed data adapter system, we can provide a seamless mechanism to use RDB and NoSQL database at the same time.     

Persisting big-data: The NoSQL landscape

The growing popularity of massively accessed Web applications that store and analyze large amounts of data, being Facebook, Twitter and Google Search some prominent examples of such applications, have posed new requirements that greatly challenge traditional RDBMS. In response to this reality, a new way of creating and manipulating data stores, known as NoSQL databases, has arisen. This paper reviews implementations of NoSQL databases in order to provide an understanding of current tools and their uses. First, NoSQL databases are compared with traditional RDBMS and important concepts are explained. Only databases allowing to persist data and distribute them along different computing nodes are within the scope of this review. Moreover, NoSQL databases are divided into different types: Key-Value, Wide-Column, Document-oriented and Graph-oriented. In each case, a comparison of available databases is carried out based on their most important features.     

NoSQL数据库与关系数据库的比较分析

介绍了两个具有代表性的NoSQL数据库:Bigtable和Dynamo系统。首先,描述了Bigtable和Dynamo的适用范围及其产生原因。Bigtable和Dynamo可以高效的处理web数据提供相应服务;然后,介绍了Bigtable和Dynamo系统的架构、特性等,以及各自独特的设计方法。最后,将这两个数据库与传统的关系数据库进行比较分析,描述了它们之间的不同点,对比结果表明NoSQL数据库在处理web应用数据时是高效可用的,比传统关系数据库更占优势。     

支持大数据管理的NoSQL系统研究综述

针对大数据管理的新需求,呈现出了许多面向特定应用的NoSQL数据库系统。针对基于key-value数据模型的 NoSQL 数据库的相关研究进行综述。首先,介绍了大数据的特点以及支持大数据管理系统面临的关键技术问题;然后,介绍了相关前沿研究和研究挑战,其中典型的包括系统体系结构、数据模型、访问方式、索引技术、事务特性、系统弹性、动态负载均衡、副本策略、数据一致性策略、基于flash的多级缓存机制、基于MapReduce的数据处理策略和新一代数据管理系统等;最后给出了研究展望。     

NoSQL数据库间数据交换代价研究

随着互联网时代的到来,IT行业迅猛发展,NoSQL数据库以其在大数据环境下出色的业务处理处理能力,在IT行业内得到越来越广泛的应用。而各NoSQL数据库由于自身数据模型的不同,在数据组织方式上彼此存在差异。NoSQL数据库间进行数据交换时,数据模型的不同会导致数据库间数据传输的阻抗,以源数据库数据模型封装的业务数据可能无法直接被目标数据库解析,需进行额外的模型适配操作,参照目标数据库数据模型组织业务数据以供筛选存储。为此,拟定义一种数据描述模型,对NoSQL数据库数据模型特征建模,描述NoSQL数据库的数据组织方式,并定义NoSQL数据库数据模型间距离评估算法。根据数据描述模型与距离评估算法可设计实现一种通用数据模型,其在数据交换过程中可与相关NoSQL数据库进行数据模型上的转换,系统相关业务代码只需参照该数据模型设计,而独立于数据交换过程中NoSQL数据库具体的数据模型。     

传统RDBMS向非关系型MongoDB数据模型转换与数据迁移方法研究

为解决关系型数据库在大数据处理中遇到的瓶颈问题,满足企业对大数据处理的需求,提出将关系型数据库迁移到NoSQL文档型数据库中。针对RDBMS中的关系模型向MongoDB中的集合模型转化方法进行了研究,提出了表示关系间参照完整性的有向图表示模型,和基于关系型数据模型向MongoDB文档模型自动转化算法;实现了RDBMS中迁移数据到MongoDB的插入算法。针对上述方案和算法,结合典型开源RDBMS--MySQL实例,对上述关系有向图模型的生成、基于有向图模型的转化算法以及数据迁移算法应用验证。实验结果表明RDBMS可以按照一定的数据结构平滑地迁移到MongoDB中。     

Supporting set-valued joins in NoSQL using MapReduce

NoSQL systems are increasingly adopted for Web applications requiring scalability that relational database systems cannot meet. Although NoSQL systems have not been designed to support joins, as they are applied to a wide variety of applications, the need to support joins has emerged. Furthermore, joins performed in NoSQL systems are generally similarity joins, rather than exact-match joins, which find similar pairs of records. Since Web applications often use the MapReduce framework, we develop a solution to perform similarity joins in NoSQL systems using the MapReduce framework.     

一种基于NoSQL数据库的医疗监护大数据存储设计

移动医疗、家庭远程监护等患者健康管理模式的兴起,产生了海量的医疗监护数据,在对患者的海量健康数据处理时存在性能瓶颈问题。文章首先介绍了医疗实时监护大数据的特征和应用模型,分析了当前关系数据库存储海量医疗监护数据存在的问题,比较了多种NoSQL数据库的特性,提出了使用HBase分布式非关系数据库进行医疗监护大数据存储的方案,并给出了主要表结构设计,最后对HBase数据库的部署、表数据访问进行说明。     

Support for modeling relationships in object-oriented databases

Modeling real-world applications typically consists of two parts: the representations of entities of interest, and the representations of relationships among the entities. Object-oriented data models support the first task. This paper argues through examples that the second modeling task is not well supported in current object-oriented database systems. An extended object-oriented data model that facilitates both modeling tasks is developed. This extension provides support for both the representation of complex relationships among entities and their manipulations (queries, updates). A prototype implemented on the ONTOS database management system is briefly described.     

BASE analysis of NoSQL database

NoSQL databases are designed to address performance and scalability requirements of web based application which cannot be addressed by traditional relational databases. Due to their contrast in priorities and architecture to conventional relational databases using SQL, these databases are referred as “NoSQL” databases since they ​incorporate lots of additional features in addition to the features of conventional databases. The relational databases strongly follow the ACID (Atomicity, Consistency, Isolation, and Durability) properties while the NoSQL databases follow BASE (Basically Available, Soft State, Eventual consistency) principles. This survey paper is an analytical study on BASE features of some of NoSQL databases.     

关系型数据库与NoSQL数据库

通过对传统关系型数据库以及他目前存在的问题的说明,进而引出了NoSQL数据库,并指出NoSQL数据库的特点以及目前应用状况,最后总结出传统关系型数据库和NoSQL数据库在某种场景下结合使用,并给出具体实例加以说明。     

基于NoSQL系统的组合索引技术研究

分布式NoSQL系统旨在提供大规模数据的高可用性,但缺乏内在的支持复杂查询的应用程序。传统的基于单一词汇倒排表的解决方案未达到良好的效果。因此,文中就文档型数据库在处理动态文档集时不支持多键作为主索引的缺点展开研究,提出了一种改进的组合索引方法。通过存储组合条件的倒列表,查询驱动机制可以从最近的查询记录中自适应地存储比较受欢迎的条件组合。该方法可以降低整体的带宽消耗,只需占用较少的存储资源等额外开销,明显改善了NoSQL系统的容量和响应时间。     

利用NoSQL构建高性能全文检索系统

随着互联网的发展,数据的增加越来越快,从海量的信息中快速地提取出用户需求的信息成为新的挑战。传统的方式已经不能完全满足现在的需求,而NoSQL却能够有效地解决这一问题。本文首先分析Lucene的特点以及不足,并利用NoSQL来构建全文检索系统,最后通过实验来检验此架构的性能。     

基于数据源向图的数据库设计中数据关系的表示工具

数据库设计是信息系统需求分析到系统实现中的一个关键环节。传统的数据库设计方法需根据需求分析阶段的结果,依赖人的构造性思维抽象出对象和关系,由于对所需数据结构和关系的描述能力不足,缺少对数据项之间关系的描述,导致数据关系不明确,数据库设计结果容易出现偏差。在新的数据库设计工具——数据源向图的帮助下,对象、关系以及各数据项之间的关系能被直观、简洁、准确地表示,同时其也能极大地消除各种关系不明确而带来的二义性问题,可有效提高信息系统实现的效率。     

NoSQL的分布式存储与扩展解决方法

传统的关系型数据库已无法满足海量数据的存储与访问需求。针对该问题,提出一种非关系型数据库(NoSQL)的分布式存储与扩展解决方法。分析并改进NoSQL,讨论基于一致性哈希算法键值对的分布式存储,以及基于双hash环的数据库服务器节点的扩展方法,提出将NoSQL作为镜像引入数据库架构系统。实际应用结果表明,该方法可以避免资源浪费及服务器过载。     

NoSQL real-time database performance comparison

The amount of data being produced is increasing constantly, as the number and variety of connected devices are growing and the advances in data storage and mining are supporting this evolution. However, storing and handling high quantities of data is challenging the current Relational Database Management Systems. Big Data and its related products came to help in this matter, and the NoSQL databases arise with the purpose to offer better solutions and features to handle massive amounts of data with higher performance, sometimes near real-time. The present study presents the NoSQL databases scenario and background, and elaborates a detailed study with the characteristics, a features comparison and a performance evaluation of three different NoSQL databases extensively used in the market nowadays: Couchbase, MongoDB and RethinkDB. Tests were performed in two different scenarios: single thread and multiple threads. The results reveal that Couchbase had a better performance at most of the operations, except for retrieving multiple documents and inserting documents with multiple threads, operations in which MongoDB scored better.     

一种分布式大数据管理系统的设计与实现

随着云计算、物联网、移动互联网等技术的飞速发展,海量数据在这些崭新的领域迅猛地生长着,大数据作为一项颠覆性技术,为处理海量数据提供了无限可能。而传统的关系型数据库的不再适用,导致了分布式数据库NoSQL的应运而生。针对大数据领域面临的种种现实难题,设计并实现了一种基于Hadoop和NoSQL的新型分布式大数据管理系统(DBDMS),其提供大数据的实时采集、检索以及永久存储的功能。实验表明,DBDMS可以显著提高大数据处理能力,适用于海量日志备份和检索、海量网络报文抓取和分析等领域。     

基于NoSQL的FITS文件头元数据存储和查询研究

随着大型天文望远镜的投入使用,观测台站正面临PB量级的海量数据存储、快速检索难题;同时由于在数据检索中起着关键作用的FITS文件头的可变性,导致难以使用传统的关系型数据库来建立可适应这种变化需求的非结构化数据模型。针对这个难题,提出了使用NoSQL对天文上广泛使用的FITS文件头中所包含的可变元数据信息进行存储和查询;讨论了关系型数据模型存储可变FITS文件头的不足;分析了NoSQL存储可变FITS头元数据信息的可行性;使用形式化的关系型代数对这种存储查询方式进行了一般化的讨论。通过具体查询实例验证了该方案在存储天文可变FITS文件头的有效性和可行性。