一种分布式数据存储方案设计与实现

传统的基于关系型数据库的分布式存储主要通过引入中间件对数据进行水平或垂直拆分来实现,这类中间件主要适用查询主键存在单调递增或单调递减的情况,针对查询主键不符合该要求的,该文设计并实现了一种分布式数据存储方案,基于数据库号段模式生成单调递增的分布式ID作为关系型数据库的拆分主键,借助MongoDB存储查询键值和拆分主键的关联信息。实验结果表明,该方法可以有效实现海量数据的分布式存储。     

海量图片的分布式存储及负载均衡研究

针对海量图片给网站带来的访问速度下降、性能压力增大和I/O瓶颈等问题,提出一种海量图片的分布式存储及负载均衡技术。通过把图片数据和网站内容分开部署、在数据库中记录和维护图片服务器状态信息等方法实现图片和页面数据的分离。实验结果表明,该技术能提高网站的访问速度和运行效率,并可动态增加图片服务器的数量满足日益增加的性能需求。     

海量视频数据分布式存储性能优化方法研究

针对文件系统存储效率低、多余副本导致空间浪费以及磁盘能源损耗严重的问题,提出一种新型分布式优化存储策略(distributed optimized storage strategy,DOSS).首先,引入Bcache混合存储技术在磁盘阵列前增设固态硬盘,作为高速缓冲区对多路视频流进行临时数据组织,变多线程并发写任务为单线程顺序写入任务,规避磁盘内部碎片产生,有效提高系统写入效率.其次,提出改进的liberation码对视频数据进行压缩存储,在保证系统可靠性的同时提高磁盘空间利用率.最后,基于ioctl系统调用编写盘片转速控制函数,实现磁盘多级休眠和低延迟唤醒,减低磁盘能耗,提高使用寿命.结果表明单台存储服务器在500路4 Mbps并发视频流下存储效率提高约36％,存储空间节省约40％,系统应对12000路并发视频流时仍存在休眠磁盘约10％.     

基于云计算技术的海量信息分布式存储研究

面对海量信息的有效存储,为了保证存储信息的抽取和查询的效率,研究基于云计算技术的海量信息分布式的存储方法。采用GFS作为分布式文件系统和HDFS管理节点/存储节点架构作为分布式存储技术的依据,形成极大存储容量的计算机群,对信息实行并行处理;生成事实表,分析和处理不同维度和粒度的情况下的信息后,对其实行数据聚集;采用基于云计算技术改进ETL处理算法实行海量信息抽取,存储在数据库中,用户即可根据需求实行数据库信息查询。实验结果表明,该方法的存储性能较好,物理节点的增加会提高信息的插入效率,并且抽取后的信息信噪比较高,信息查询速度较快。     

云环境下海量GPS、视频监控数据存储检索研究

监控数据在不断增大,且80%以上为非结构化数据,利用传统关系型数据库存储管理时,占用机器内存较大,检索数据需对数据库全表扫描,管理性能低。为解决海量监控数据的存储与管理,采用HDFS分布式文件系统存储视频非结构化数据,研究设计Hbase属性信息表,利用Hbase分布式数据库存储属性信息,研究Hbase全局检索信息行键及MapReduce工作流程,构建检索,实现视频监控数据的检索。最终基于非结构化数据的检索技术以及分布式数据处理框架的研究,得到的海量监控数据存储与检索方案。     

基于Hadoop的大数据存储与检索性能优化

为实现大数据的安全存储和高效检索,本文提出基于Hadoop（分布式系统基础架构）的大数据存储与检索性能优化方法。以Hadoop架构为核心,构建分布式存储系统,利用Hadoop分布式集群优势,实现大数据的分块处理和存储,并引入分布式区域检索算法在各个区域中进行所需数据检索。测试结果显示：分布式存储能够更高效地完成大数据存储,各个类别数据的检索结果误差均低于2%。     

基于云计算的图书馆海量数据存储研究

基于云计算的数据存储技术改变了现代图书馆的管理模式,也更新了传统的信息服务方式。现代数字图书馆系统的构建将沿着网络信息服务产业的发展方向搭设,对以往的数据存储模式形成较大的冲击。该文就云计算环境下的图书海量数据存储技术进行研究,剖析现代数字图书馆的主要特征,以及对于云计算技术手段在图书馆数据管理中的应用现状做出分析,并且提出相应的指导性意见。     

基于云计算的图书馆海量数据存储研究

基于云计算的数据存储技术改变了现代图书馆的管理模式,也更新了传统的信息服务方式。现代数字图书馆系统的构建将沿着网络信息服务产业的发展方向搭设,对以往的数据存储模式形成较大的冲击。该文就云计算环境下的图书海量数据存储技术进行研究,剖析现代数字图书馆的主要特征,以及对于云计算技术手段在图书馆数据管理中的应用现状做出分析,并且提出相应的指导性意见。     

分布式海量遥感影像编目检索机制

为了高效地组织、检索分布式海量遥感数据,研究FGDC制定的遥感元数据标准草案,提出一种具有良好可扩展性的遥感编目数据模型。基于该编目数据模型,提出一种基于编目数据的分布式影像检索算法,设计并实现一个遥感数据编目检索系统,实验结果表明,该系统是有效的。     

高性能海量数据存储模拟器设计

介绍了一种高速海量数据存储系统的模拟器,即基于NAND Flash阵列的数据存储器模型,该模型采用多通道和多Interleave管理模式,形成以数据存储、传输为核心的数据存储SoC平台。主要完成对FTL算法和系统结构的有效性能测试,目前该模型具有一定的有效性,速度快,稳定性高。     

Efficient selection, storage, and retrieval of irregularly distributed elevation data

The advantages and disadvantages of various digital terrain models are discussed briefly and a new model for triangulating a set of nonuniformly distributed three-dimensional surface observations is described. An algorithm for hierarchical subdivision of a set of data points into nested triangles is proposed. The algorithm selects a subset of the data points which reduce the maximum error between a piecewise linear approximation of the surface using only the selected points and the elevations of the points not selected. The data structure used is such that for any given degree of approximation (in the maximum-error sense) only the necessary points need to be stored. Furthermore, an efficient method is available to approximate the elevation of the surface at any point not included in the original data. The performance of the algorithm is demonstrated experimentally.     

A real-time distributed cluster storage optimization for massive data in internet of multimedia things

With the rapid growth of massive data in the Internet of Multimedia Things, there are some problems of insufficient storage space and unbalanced load in the current methods. For the problem of massive real-time data storage, a distributed cluster storage optimization method is proposed. Considering the impact of replica cost and the generation of intermediate data on the replica layout, a replica generation and storage strategy is given with consideration of cost and storage space. In the data center, the data sensitivity and data access frequency is used as migration factors to achieve massive data migration. The improved collaborative evolution method is used to code the task scheduling particle swarm in massive data storage to obtain the optimal solution, and achieve massive real-time data distributed cluster storage for the Internet of things. The experimental results showed that the cost of data management by this method was only between 10 and 15, which showed that this method can effectively improve data access speed, reduce storage space, lower cost and better load balancing.

     

Issues in storage and retrieval of multimedia data

As the proliferation of multimedia systems continues in diverse application areas, it is becoming increasingly apparent that the performance of the I/O subsystem is a critical limiting factor in the usefulness of such systems. This has spurred extensive research to discover and design efficient and robust I/O systems for the storage and retrieval of multimedia data. To mitigate the effects of the I/O bottleneck in multimedia application environments, we must employ novel technologies and efficient algorithms, and we must use available resources carefully. This paper identifies some significant issues involved and presents a survey of the techniques developed or proposed during recent years to make multimedia I/O more efficient. e-mail: ashfaq@eecis.udel.edu     

A scalable storage supporting multistream real-time data retrieval

This paper presents a new data placement scheme for continuous-media playback via a scalable storage system. Multimedia contents are segmented into data blocks for the purpose of being stored, retrieved, and manipulated. If these data blocks belong to some continuous media, then they must be handled in a timely manner, for example, being retrieved before some deadline. One of the main challenges in implementing the above system is the simultaneous retrieval of a great number of different media streams from a very large storage system. The proposed scheme efficiently reduces the seeking delay by a very simple placement method and a retrieval scheduler. Thus, both the storage capacity and the number of concurrent accesses to the storage are scalable. The performance of the proposed scheme is evaluated through a simple analytical model and a practical prototype implementation.     

基于网格的分布式虚拟环境仿真海量数据管理

分析了分布式虚拟环境仿真的特点,提出了基于网格的分布式虚拟环境仿真的海量数据管理框架.该框架结构采用分层结构,自底向上依次为网格节点、高性能通信系统、数据存储与处理系统和计算系统.给出了一个基于上述体系结构的原型系统.对该原型系统的仿真结果表明,该海量数据管理体系结构设计符合虚拟环境仿真实时性、稳定性和高可靠性的要求.     

Optimising context data dissemination and storage in distributed pervasive computing systems

Context management systems are expected to administrate large volumes of spatial and non-spatial information in geographical disperse domains. In particular, when these systems cover wide areas such as cities, countries or even the entire planet, the design of scalable storage, retrieval and propagation mechanisms is paramount. This paper elaborates on mechanisms that address advanced requirements, including support for distributed context databases management; efficient query handling; innovative management of mobile physical objects and optimization strategies for distributed context data dissemination. These mechanisms establish a robust spatially-enhanced distributed context management framework that has already been designed and carefully implemented and thoroughly evaluated.     

基于甲骨文数据库管理系统的化学结构检索数据库的设计与实现

设计了一种基于大型数据库管理系统Oracle的化学结构数据存储模式,并实现了相应于此模式的高效化学结构检索方法。结构检索方法建立在图子图匹配算法VF2的基础上,对其进行了必要的改造和扩展,使之适合于化学结构检索。在此基础上,针对美国NCl(National Cancer Institute)25万个化合物的2D结构建立了数据库,成功进行了结构检索试验。结果表明,这种实现方法不仅能高效存储并准确检索化合物的结构信息,而且也容易实现与药物研发过程中所产生的大量其它数据(如生物筛选数据和DNA芯片基因表达数据等)进行高效整合。这个设计的改进版已经集成于微芯公司的药物研发生化信息学软件系统——TASS(Target Activity Structure System)中。