基于NoSQL的FITS文件头元数据存储和查询研究

随着大型天文望远镜的投入使用,观测台站正面临PB量级的海量数据存储、快速检索难题;同时由于在数据检索中起着关键作用的FITS文件头的可变性,导致难以使用传统的关系型数据库来建立可适应这种变化需求的非结构化数据模型。针对这个难题,提出了使用NoSQL对天文上广泛使用的FITS文件头中所包含的可变元数据信息进行存储和查询;讨论了关系型数据模型存储可变FITS文件头的不足;分析了NoSQL存储可变FITS头元数据信息的可行性;使用形式化的关系型代数对这种存储查询方式进行了一般化的讨论。通过具体查询实例验证了该方案在存储天文可变FITS文件头的有效性和可行性。     

NoSQL数据管理技术

社交网络和微博等新型应用对数据管理技术提出了新的挑战,如海量数据高效存储、高并发访问、高可扩展性和高可用性等。而传统的关系数据库技术无法满足这些新型应用的需求,因此,NoSQL数据管理技术的研究、开发和应用越来越受到重视。本文从NoSQL数据模型、数据存储、查询处理以及SQL与NoSQL混合数据库解决方案等方面,综述了NoSQL数据管理技术发展现状和趋势,并介绍了几种典型的NoSQL产品。     

大数据背景下的NoSQL数据库课程建设

随着大数据时代的到来,传统关系型数据库面临着诸多挑战,如数据规模的爆炸性增长、数据类型的多样性以及高并发的应用需求等。为了解决这些问题,NoSQL数据库应运而生,并逐渐成为大数据领域的重要组成部分。基于此,探究如何建设一门有效的NoSQL数据库课程,以培养学生在大数据环境下运用NoSQL数据库解决实际问题的能力。     

NoSQL与关系数据库相结合的设计与实践

为了实现对数据库的字段属性动态扩展方面的需求,提出一种将NoSQL和关系数据库相结合的架构模式设计方案。该设计方案主要以NoSQL良好的扩展性弥补关系数据库在属性横向扩展方面的缺陷,其架构方法主要采用在原有的关系数据库架构模式上增加一层辅助的NoSQL存储,能够完成对字段属性的动态扩展。实际应用表明,该设计模式具有易理解、易实现的特点,达到了设计要求。     

NoSQL的分布式存储与扩展解决方法

传统的关系型数据库已无法满足海量数据的存储与访问需求。针对该问题,提出一种非关系型数据库(NoSQL)的分布式存储与扩展解决方法。分析并改进NoSQL,讨论基于一致性哈希算法键值对的分布式存储,以及基于双hash环的数据库服务器节点的扩展方法,提出将NoSQL作为镜像引入数据库架构系统。实际应用结果表明,该方法可以避免资源浪费及服务器过载。     

面向NoSQL数据库的JSON文档异常检测与语义消歧模型

随着信息化技术的发展,面对材料等相关领域数据的多源异构、扩展性强、爆炸增长等特点,传统关系数据库无法对数据进行存储,因此可利用NoSQL的无模式存储、高扩展性等特性来解决这一难题。作为NoSQL数据库常用的数据存储格式,JSON因简单性和灵活性备受欢迎。然而,NoSQL数据库缺乏模式信息,在JSON文档存入数据库之前,需要对其进行数据验证与分析。目前,大多数方法是基于JSON schema对JSON文档格式的规范性进行校验,无法有效解决JSON文档的异常检测以及语义歧义问题。为此,文中提出了面向NoSQL数据库的JSON文档异常检测与语义消歧模型doctorJSON。该模型基于JSON schema对存入的JSON文档分别设计了异常检测算法deoutJSON和语义消歧算法disemaJSON,以检测JSON文档存在的异常和歧义。在真实数据集与合成数据集上的实验验证了所提模型的有效性和执行效率。     

浅析NoSQL技术与数据库教学的关系

关系数据库在数据库领域长期占有主导地位,一直是高等院校数据库课程的主要内容.而随着Web2.0的兴起,在海量数据的环境下,NoSQL(非关系型的数据库)技术得到了广泛的应用,它对数据库技术的发展产生了强烈的影响,同时也对当前数据库课程教学产生了深远的影响.探讨了NoSQL技术及其主要特点,分析了NoSQL技术对数据库教学的挑战,将关系数据库和NoSQL进行对比,指明了使用NoSQL的原因,并提出有关NoSQL的启发式教学方法.     

基于大数据的实验室信息分享系统设计

基于传统方式设计的实验室信息分享系统不能满足海量实验数据分享的需求,面临用户数受限、存放数据受限等诸多扩展性问题,因此采用大数据技术、开源NoSQL数据库HBase存放用户信息以解决用户数量受限的问题。同时,采用开源MinIO进行数据的分布式存放,解决用户数据空间难以扩容的问题。在3台通用服务器上部署该系统,结果表明在10万级用户量下,能轻松支持5000名用户同时在线分享信息。     

基于文档数据库的NoSQL建模方法

电子商务、大型社交网站的出现对数据存储、查询和管理带来了新的挑战,也引起了非结构化的面向信息的数据爆炸,海量数据的存储对信息的查询和管理提出了更高的要求。而传统的关系数据技术在应对超大规模和高并发的数据处理方面已经显得力不从心,NoSQL作为一种针对大规模数据应用的新型数据库技术,越来越受到重视。本文以NoSQL数据模型的基础,介绍文档型数据库建模的方法。     

一切为了分布式——2009年Web后端技术回顾

NoSQL运动与分布式技术发展 NoSQL是指用非关系数据库的方式来存储数据,通常也指用key value方式存储。在2008年的时候,key value还是一个新鲜事物,当时比较早的key value存储产品之一Memcachedb发布时,JavaEye的Robin提到“第一眼看到Memcachedb,我就知道她是我寻觅良久的梦中情人”。     

Uniform data access platform for SQL and NoSQL database systems

Integration of data stored in heterogeneous database systems is a very challenging task and it may hide several difficulties. As NoSQL databases are growing in popularity, integration of different NoSQL systems and interoperability of NoSQL systems with SQL databases become an increasingly important issue. In this paper, we propose a novel data integration methodology to query data individually from different relational and NoSQL database systems. The suggested solution does not support joins and aggregates across data sources; it only collects data from different separated database management systems according to the filtering options and migrates them. The proposed method is based on a metamodel approach and it covers the structural, semantic and syntactic heterogeneities of source systems. To introduce the applicability of the proposed methodology, we developed a web-based application, which convincingly confirms the usefulness of the novel method.     

基于向量时钟模型的NoSQL最终一致性的研究

最终一致性作为衡量NoSQL数据库性能的重要指标,对NoSQL的应用与发展起着重要作用。针对最终一致性解决方案存在的问题,提出基于向量时钟的最终一致性模型。模型为数据加入了版本的概念,通过版本向量和时间戳的引入,为数据标识了版本信息,同时给出了冲突检测和处理方案。对比分析表明,方案很好地解决了问题,提高了最终一致性的性能,在NOSQL数据库的设计过程中具有较好的应用价值。     

一种基于中间件的异构数据库融合访问方法及系统

在大数据时代,信息化数据呈爆炸式增长,传统关系型数据库和新兴的NoSQL数据库都难以全面且高效地面对这些挑战。因此,提出一种基于中间件的异构数据库访问方法(MingleDB),以结合NoSQL和传统关系型数据库的优点。MingleDB透明融合了NoSQL数据库和传统数据库的主要运行逻辑,同时又能够根据当前用户请求的读写特征,自动选取合适的处理路径以避免二者的不足;它还支持轻量级的事务处理框架,该框架按需实施以保证异构数据库数据的最终一致性和完整性。将MingleDB分别与MongoDB,MySQL数据库进行读写性能对比,实验证明了MingleDB方法的正确性和合理性。同时将MingleDB部署在实际的社交网络系统中进行实际验证,结果亦证明了其实用性和可移植性。     

Data modeling in the NoSQL world

NoSQL systems have gained their popularity for many reasons, including the flexibility they provide in organizing data, as they relax the rigidity provided by the relational model and by the other structured models. This flexibility and the heterogeneity that has emerged in the area have led to a little use of traditional modeling techniques, as opposed to what has happened with databases for decades.In this paper, we argue how traditional notions related to data modeling can be useful in this context as well. Specifically, we propose NoAM (NoSQL Abstract Model), a novel abstract data model for NoSQL databases, which exploits the commonalities of various NoSQL systems. We also propose a database design methodology for NoSQL systems based on NoAM, with initial activities that are independent of the specific target system. NoAM is used to specify a system-independent representation of the application data and, then, this intermediate representation can be implemented in target NoSQL databases, taking into account their specific features. Overall, the methodology aims at supporting scalability, performance, and consistency, as needed by next-generation web applications.     

基于NoSQL数据库的大数据存储技术的研究与应用

实际工程中采集和处理的数据量特别大,这对传统数据库技术提出巨大挑战。针对传统关系型数据库存储速度慢、对硬件要求高的缺点,提出一种以NoSQL数据库为基础的大数据处理方法,打破了传统数据库的关系模型,数据以一种自由的方式存储,而不依赖固定的表结构。该方法主要是将经验模态分解并与NoSQL数据库技术相结合,应用于大型结构件的变形监测中,构建出一个基于NoSQL数据库系统的大型结构件变形监测系统。仿真结果表明,该方法可以实现大型结构件变形监测数据的实时处理,在计算收敛性、算法稳定性和处理速度上都优于传统数据库技术。     

NoSQL数据库间数据交换代价研究

随着互联网时代的到来,IT行业迅猛发展,NoSQL数据库以其在大数据环境下出色的业务处理处理能力,在IT行业内得到越来越广泛的应用。而各NoSQL数据库由于自身数据模型的不同,在数据组织方式上彼此存在差异。NoSQL数据库间进行数据交换时,数据模型的不同会导致数据库间数据传输的阻抗,以源数据库数据模型封装的业务数据可能无法直接被目标数据库解析,需进行额外的模型适配操作,参照目标数据库数据模型组织业务数据以供筛选存储。为此,拟定义一种数据描述模型,对NoSQL数据库数据模型特征建模,描述NoSQL数据库的数据组织方式,并定义NoSQL数据库数据模型间距离评估算法。根据数据描述模型与距离评估算法可设计实现一种通用数据模型,其在数据交换过程中可与相关NoSQL数据库进行数据模型上的转换,系统相关业务代码只需参照该数据模型设计,而独立于数据交换过程中NoSQL数据库具体的数据模型。     

An integration approach of hybrid databases based on SQL in cloud computing environment

As the applications with big data in cloud computing environment grow, many existing systems expect to expand their service to support the dramatic increase of data, and modern software development for services computing and cloud computing software systems is no longer based on a single database but on existing multidatabases and this convergence needs new software architecture design. This paper proposes an integration approach to support hybrid database architecture, including MySQL, MongoDB, and Redis, to make it possible of allowing users to query data simultaneously from both relational SQL systems and NoSQL systems in a single SQL query. Two mechanisms are provided for constructing Redis's indexes and semantic transforming between SQL and MongoDB API to add the SQL feature for these NoSQL databases. With the proposed approach, hybrid database systems can be performed in a flexible manner, ie, access can be either relational database or NoSQL, depending on the size of data. The approach can effectively reduce development complexity and improve development efficiency of the software systems with multidatabases. This is the result of further research on the related topic, which fills the gap ignored by relevant scholars in this field to make a little contribution to the further development of NoSQL technology.     

Persisting big-data: The NoSQL landscape

The growing popularity of massively accessed Web applications that store and analyze large amounts of data, being Facebook, Twitter and Google Search some prominent examples of such applications, have posed new requirements that greatly challenge traditional RDBMS. In response to this reality, a new way of creating and manipulating data stores, known as NoSQL databases, has arisen. This paper reviews implementations of NoSQL databases in order to provide an understanding of current tools and their uses. First, NoSQL databases are compared with traditional RDBMS and important concepts are explained. Only databases allowing to persist data and distribute them along different computing nodes are within the scope of this review. Moreover, NoSQL databases are divided into different types: Key-Value, Wide-Column, Document-oriented and Graph-oriented. In each case, a comparison of available databases is carried out based on their most important features.     

NoSQL data management systems

In the last decade, a new class of data management systems collectively called NoSQL systems emerged and are now intensively developed. The main feature of these systems is that they abandon the relational data model and the SQL, do not fully support ACID transactions, and use distributed architecture (even though there are non-distributed NoSQL systems as well). As a result, such systems outperform the conventional SQL-oriented DBMSs in some applications; in addition, such systems are highly scalable under increasing workloads and huge amounts of data, which is important, in particular, for Web applications. Unfortunately, the absence of transactional semantics imposes certain constraints on the class of applications where NoSQL systems can be effectively used and the choice of a particular system significantly depends on the application. In this paper, a review of the main classes of NoSQL data management systems is given and examples of systems and applications where they can be used are discussed.