大数据环境下数据读取关键技术研究

针对大数据环境下数据读取面临的主要挑战,文中重点研究了分布式文件系统中数据读取关键技术。根据数据存放结构的不同,从数据加载、查询处理和存储空间利用三个方面分析比较行存储、列存储和行列混合存储的优缺点和面临的挑战,重点介绍列存储中涉及到的压缩和物化技术,具体分析了存储压缩中经常运用的行程编码算法、词典编码算法、位向量编码算法和元组重构中运用的延迟物化技术。通过分析现有技术存在的问题,探讨相关的解决方案,并展望了未来研究的发展方向。     

高能物理事例级数据管理与传输系统的研究

高能物理实验不断的进步与发展产生了PB乃至EB级的数据,这些数据的采集、存储、传输与共享、分析与管理都面临着极大的问题与挑战。为了应对这些挑战,设计和实现了面向事例的数据管理系统,有效解决事例数据处理效率低以及分站点资源利用率低的问题。设计了一个基于Nosql数据库的事例索引系统,通过事例数据特征抽取,选取物理学家最感兴趣的属性作为索引,存储在数据库中,并采用倒排索引技术,提高事例数据检索的效率。针对事例数据进行缓存优化,减少数据转化和存储开销。提出数据跨域传输方案,充分利用网络带宽,降低分站点处理数据的延迟。系统进行了相关验证,实验结果表明,事例级的索引技术能够显著提高事例数据的检索效率,数据传输系统的网络带宽也可以利用到百分之九十以上。     

地理信息系统与大数据的耦合应用

针对快速增长的地理空间数据为地理信息系统(geographical information system,GIS)在大数据时代的应用带来众多机遇和挑战,却一直缺乏系统论述的问题,在对大数据及地理空间大数据特征进行分析的基础上,通过若干典型实例展示了大数据与GIS耦合所带来的巨大应用,并分析了大数据时代对GIS数据采集、数据存储、数据处理和分析等诸多方面所带来的挑战,进一步指出引入概率论的数据筛选、分布式云存储和数据快速分析等数据分析技术与方法,是增强大数据时代中GIS与决策支持系统之间耦合度的着力点。     

大数据环境下的平台架构技术研究

大数据时代的到来,对数据的管理和存储带来了挑战。为了更好地管理和利用大数据,对大数据的基本特征进行了剖析,从大数据和分析技术平台的视角,分析了海量并行处理架构、云计算、网格计算和MapReduce这几种重要的技术,比较了其技术特点,对运用不同的技术对大数据时代数据管理产生的作用进行了分析,讨论了大数据分析工作所需要的混合环境,整合环境资源以使能够协同工作,使得每一项技术变得更加强大有效。     

优化多PB级存储

昆腾公司近日宣布,通过将优化目标存储技术集成到新的分层存储和管理产品中,它将扩大自身在管理大数据内容方面的领导地位。这些新产品将使关注大数据内容和分析的客户应对长期以来维护及保护其磁盘上数据的当前挑战。PB级信息存储     

生物医疗健康大数据应用支撑平台与关键技术

随着生物信息学的不断发展,生物医学领域积累了大量的数据,大数据已经贯穿基础研究、临床诊断、医药开发、健康管理等生物医学领域的各个环节。如何有效存储、管理、分析这些海量数据面临严峻的而挑战。基于超级计算机的计算分析和存储能力,在生物医学大数据处理的异构融合架构,面向生物医学大数据的层次式存储系统,生物医学大数据处理的异构并行计算和多源数据的汇聚机制与分析方法,突破生物医学大数据的汇聚、存储、分析等方面的关键技术,构建一个计算、分析处理和存储融合平台,以满足多种类型生物医学大数据应用的不同需求。     

基于分布式文件电力异构数据存储综述

随着国网公司信息化建设的不断推进,在整个电网的运检和管理的过程中都会产生海量的数据,这些数据中包含各场景产生的视频、图片、传感器数据和一些企业档案信息等非结构（异构）化数据.在面对如此大规模非结构化的数据存储要求时,传统关系型数据库已经表现的力不从心了.如何对此类数据进行高效地、廉价地和安全可靠地存储,并且可以快速检索与分析,是当下研究的重要热点课题之一.本文首先分析了电网大数据的产生及特征,然后综述了工业界大数据分布式文件存储技术,最后分析适合国网非结构化数据的分布式文件存储策略.     

大数据时代如何安全有效的管理和使用信息

大数据时代为企业带来了新的机遇和挑战:一方面,数据的不断更新扩张给数据存储、管理和分析利用带来了挑战;另一方面,这样的海量数据中,蕴含着大量有价值的信息,对企业经营、后续运营和发展至关重要。那么,如何更加安全、有效的使用和管理这些信息呢?     

物联网与大数据技术在油田中的融合应用

油气生产过程中产生了大量实时数据,逐年积累形成海量的历史数据,而这些海量数据的价值未能得到充分挖掘。为了充分挖掘这些信息,文章设计了油气生产物联网与大数据技术融合系统。该系统架构将分布式存储、分布式计算和数据智能分析等技术,应用在油气生产过程的数字孪生、注水系统智能化和油井工况实时诊断等。大港某油田的应用表明,设计系统的实际效果良好。     

大数据存储解决方案

正创新阵列解决方案将集成在分层存储和管理产品中,以期帮助客户以最优成本回报保留并重新利用大数据内容。数据保护和大数据管理领域厂商昆腾公司(NYSE:QTM)近日宣布,通过将优化目标存储技术集成到新的分层存储和管理产品中,它将扩大自身在管理大数据内容方面的领导地位。这些新产品将使关注大数据内容和分析的客户应对长期以来维护及保护其磁盘上数据的当前挑战。     

面向大数据的开源推荐系统分析

处在大数据时代,人们面临的重要问题是如何处理庞大的过量信息,并且如何应对数据惊人的增长趋势。论文重点讨论了面向大数据的推荐系统（RS）,分析现有的开源推荐系统,然后通过评价指标（以及大数据的4V定义）阐述开源推荐系统如何应对大数据时代不断变换的挑战。     

探讨大数据时代高校劳资工作面临的挑战及应对策略

大数据即信息时代所产生的的数据量。随着大数据时代的到来,数据作为一种资本受到越来越多的重视,且其已在商业、学术以及政府管理等领域发挥着极为重要的作用。大数据的快速发展给高校劳资工作不仅带来机遇,同时也使高校面临着新的挑战,只有充分了解大数据时代背景下,高校劳资工作面临的新挑战,才能做好充分的应对准备,从而促进高校劳资工作的稳定、有序开展。基于此,本文概述大数据时代对高校劳资工作的重要性及影响,分析大数据时代背景下,高校劳资工作面临的挑战,并探讨具体的应对策略,供同行们参考和借鉴。     

数据治理技术

随着信息技术的普及，人类产生的数据量正在以指数级的速度增长，如此海量的数据就要求利用新的方法来管理.数据治理是将一个机构（企业或政府部门）的数据作为战略资产来管理，需要从数据收集到处理应用的一套管理机制，以期提高数据质量，实现广泛的数据共享，最终实现数据价值最大化.目前，各行各业对大数据的研究比较火热，但对于大数据治理的研究还处于起步阶段，一个组织的正确决策离不开良好的数据治理.首先介绍数据治理和大数据治理的概念、发展以及应用的必要性；其次，对已有的数据治理技术——数据规范、数据清洗、数据交换和数据集成进行具体的分析，并介绍了数据治理成熟度和数据治理框架设计；在此基础上，提出了大数据HAO治理模型.该模型以支持人类智能（HI）、人工智能（AI）和组织智能（OI）的三者协同为目标，再以公安的数据治理为例介绍HAO治理的应用；最后是对数据治理的总结和展望.     

A spatiotemporal compression based approach for efficient big data processing on Cloud

It is well known that processing big graph data can be costly on Cloud. Processing big graph data introduces complex and multiple iterations that raise challenges such as parallel memory bottlenecks, deadlocks, and inefficiency. To tackle the challenges, we propose a novel technique for effectively processing big graph data on Cloud. Specifically, the big data will be compressed with its spatiotemporal features on Cloud. By exploring spatial data correlation, we partition a graph data set into clusters. In a cluster, the workload can be shared by the inference based on time series similarity. By exploiting temporal correlation, in each time series or a single graph edge, temporal data compression is conducted. A novel data driven scheduling is also developed for data processing optimisation. The experiment results demonstrate that the spatiotemporal compression and scheduling achieve significant performance gains in terms of data size and data fidelity loss.     

External integrity verification for outsourced big data in cloud and IoT: A big picture

As cloud computing is being widely adopted for big data processing, data security is becoming one of the major concerns of data owners. Data integrity is an important factor in almost any data and computation related context. It is not only one of the qualities of service, but also an important part of data security and privacy. With the proliferation of cloud computing and the increasing needs in analytics for big data such as data generated by the Internet of Things, verification of data integrity becomes increasingly important, especially on outsourced data. Therefore, research topics on external data integrity verification have attracted tremendous research interest in recent years. Among all the metrics, efficiency and security are two of the most concerned measurements. In this paper, we will bring forth a big picture through providing an analysis on authenticator-based data integrity verification techniques on cloud and Internet of Things data. We will analyze multiple aspects of the research problem. First, we illustrate the research problem by summarizing research motivations and methodologies. Second, we summarize and compare current achievements of several of the representative approaches. Finally, we introduce our view for possible future developments.     

State of the art in big data applications in microgrid: A review

The prospering Big data era is emerging in the power grid. Multiple world-wide studies are emphasizing the big data applications in the microgrid due to the huge amount of produced data. Big data analytics can impact the design and applications towards safer, better, more profitable, and effective power grid. This paper presents the recognition and challenges of the big data and the microgrid. The construction of big data analytics is introduced. The data sources, big data opportunities, and enhancement areas in the microgrid like stability improvement, asset management, renewable energy prediction, and decision-making support are summarized. Diverse case studies are presented including different planning, operation control, decision making, load forecasting, data attacks detection, and maintenance aspects of the microgrid. Finally, the open challenges of big data in the microgrid are discussed.     

Big Data 2.0 Processing Systems: Taxonomy and Open Challenges

Data is key resource in the modern world. Big data has become a popular term which is used to describe the exponential growth and availability of data. In practice, the growing demand for large-scale data processing and data analysis applications spurred the development of novel solutions from both the industry and academia. For a decade, the MapReduce framework, and its open source realization, Hadoop, has emerged as a highly successful framework that has created a lot of momentum in both the research and industrial communities such that it has become the defacto standard of big data processing platforms. However, in recent years, academia and industry have started to recognize the limitations of the Hadoop framework in several application domains and big data processing scenarios such as large scale processing of structured data, graph data and streaming data. Thus, we have witnessed an unprecedented interest to tackle these challenges with new solutions which constituted a new wave of mostly domain-specific, optimized big data processing platforms. In this article, we refer to this new wave of systems as Big Data 2.0 processing systems. To better understand the latest ongoing developments in the world of big data processing systems, we provide a taxonomy and detailed analysis of the state-of-the-art in this domain. In addition, we identify a set of the current open research challenges and discuss some promising directions for future research.     

Big data analytics for video surveillance

This article addresses the usage and scope of Big Data Analytics in video surveillance and its potential application areas. The current age of technology provides the users, ample opportunity to generate data at every instant of time. Thus in general, a tremendous amount of data is generated every instant throughout the world. Among them, amount of video data generated is having a major share. Education, healthcare, tours and travels, food and culture, geographical exploration, agriculture, safety and security, entertainment etc., are the key areas where a tremendous amount of video data is generated every day. A major share among it are taken by the daily used surveillance data captured from the security purpose camera and are recorded everyday. Storage, retrieval, processing, and analysis of such gigantic data require some specific platform. Big Data Analytics is such a platform, which eases this analysis task. The aim of this article is to investigate the current trends in video surveillance and its applications using Big Data Analytics. It also aims to focus on the research opportunities for visual surveillance in Big Data frameworks. We have reported here the state-of-the-art surveillance schemes for four different imaging modalities: conventional video scene, remotely sensed video, medical diagnostics, and underwater surveillance. Several works were reported in this research field over recent years and are categorized based on the challenges solved by the researchers. A list of tools used for video surveillance using Big Data framework is presented. Finally, research gaps in this domain are discussed.

     

关于大数据的几点思考——学会管理和利用信息资源

和物联网、云汁算、移动通信等技术进步一样,大数据对于经济的发展、科学的进步、人类生活水平和质量的提高一定会带来新的飞跃。然而,和面对任何科学进步一样,我们需要保持清醒的头脑,记住任何技术进步都是双刃剑,任何进步都会伴随着新的挑战和隐患。本文从信息资源特殊规律的角度,对于大数据带来的机遇和挑战进行了分析,并指出了需要注意防止的、忽视理论研究的、简单化的倾向。     

浅谈大数据时代信息技术的机遇与挑战

随着大数据时代的到来,信息技术问题越来越受到关注。数据作为企业重要的资源,其安全性也正受到越来越多国家、企业和个人的重视,许多部门为了保证数据的安全和保密性,积极采取了相关的信息安全策略。大数据时代的到来,给信息技术到来了机遇的同时也带来了挑战。本文在结合大数据时代的具体要求同时,结合现在信息技术发展的现实状况和实施过程中存在的问题进行了分析,并具有针对性地对未来信息技术的发展提出具体建议,希望能够对信息技术的起到实质性的帮助。