Managing deadline miss ratio and sensor data freshness in real-time databases

The demand for real-time data services is increasing in many applications including e-commerce, agile manufacturing, and telecommunications network management. In these applications, it is desirable to execute transactions within their deadlines, i.e., before the real-world status changes, using fresh (temporally consistent) data. However, meeting these fundamental requirements is challenging due to dynamic workloads and data access patterns in these applications. Further, transaction timeliness and data freshness requirements may conflict. We define average/transient deadline miss ratio and new data freshness metrics to let a database administrator specify the desired quality of real-time data services for a specific application. We also present a novel QoS management architecture for real-time databases to support the desired QoS even in the presence of unpredictable workloads and access patterns. To prevent overload and support the desired QoS, the presented architecture applies feedback control, admission control, and flexible freshness management schemes. A simulation study shows that our QoS-aware approach can achieve a near zero miss ratio and perfect freshness, meeting basic requirements for real-time transaction processing. In contrast, baseline approaches fail to support the desired miss ratio and/or freshness in the presence of unpredictable workloads and data access patterns.     

Data-Intensive Cloud Computing: Requirements, Expectations, Challenges, and Solutions

Data-intensive systems encompass terabytes to petabytes of data. Such systems require massive storage and intensive computational power in order to execute complex queries and generate timely results. Further, the rate at which this data is being generated induces extensive challenges of data storage, linking, and processing. A data-intensive cloud provides an abstraction of high availability, usability, and efficiency to users. However, underlying this abstraction, there are stringent requirements and challenges to facilitate scalable and resourceful services through effective physical infrastructure, smart networking solutions, intelligent software tools, and useful software approaches. This paper analyzes the extensive requirements which exist in data-intensive clouds, describes various challenges related to the paradigm, and assess numerous solutions in meeting these requirements and challenges. It provides a detailed study of the solutions and analyzes their capabilities in meeting emerging needs of widespread applications.     

Video data storage policies: an access frequency based approach

Video applications are characterized by their increased requirements for huge storage spaces and timing synchronization. Video data storage is a critical issue due to the so-called I/O bottleneck problem in relation to the quality of service while accessing video applications. The main contribution of the paper is that it considers video data dependencies, access frequencies and timing constraints in order to introduce a video data representation model which guides the storage policies. Two video data representation levels are considered to capture the frequencies of accesses at external (video objects) and internal (video clips) levels. A simulation model has been developed in order to evaluate the placement strategies. Video data placement is performed on a tertiary storage subsystem by both constructive and iterative improvement policies. Iterative improvement placement has been proven to outperform the other video data placement approaches.     

Improving performance by creating a native join-index for OLAP

The performance of online analytical processing (OLAP) is critical for meeting the increasing requirements of massive volume analytical applications. Typical techniques, such as in-memory processing, column-storage, and join indexes focus on high performance storage media, efficient storage models, and reduced query processing. While they effectively perform OLAP applications, there is a vital limitation: mainmemory database based OLAP (MMOLAP) cannot provide high performance for a large size data set. In this paper, we propose a novel memory dimension table model, in which the primary keys of the dimension table can be directly mapped to dimensional tuple addresses. To achieve higher performance of dimensional tuple access, we optimize our storage model for dimension tables based on OLAP query workload features. We present directly dimensional tuple accessing (DDTA) based join (DDTAJOIN), a technique to optimize query processing on the memory dimension table by direct dimensional tuple access. We also contribute by proposing an optimization of the predicate tree to shorten predicate operation length by pruning useless predicate processing. Our experimental results show that the DDTA-JOIN algorithm is superior to both simulated row-store main memory query processing and the open-source column-store main memory database MonetDB, thanks to the reduced join cost and simple yet efficient query processing.     

基于服务质量需求的存储系统动态优先权调度策略

多媒体应用要求存储系统提供保证服务质量的数据访问服务，响应时间超过延迟上限的访问请求必须小于一定的比例．基于服务质量需求的动态优先权调度策略（QADPS）根据多媒体应用接受存储系统服务的历史信息，计算多媒体应用的服务质量失败距离；并为失败距离较小的多媒体应用分配较高的调度优先权．QADPS根据多媒体应用的不同服务质量需求，实现了有差别的调度：多媒体应用的服务质量需求越严格，它的访问请求被调度的机会越高．QADPS能够在保证服务质量的前提下，支持更多的并发多媒体应用．     

A neuro-emulator with embedded capabilities for generalized learning

Artificial neural networks (ANN) are being used as one of the prime computing tool for an increasing number of applications. This is due in part to the ANN's ability to adapt to changes via learning. The dynamic nature of many applications as well as the computational and storage requirements of current learning algorithms creates a need for high performance neuro-architectures with learning capabilities. In this paper we identify a set of computational, communication and storage requirements for learning in ANNs. These requirements are representative of a wide variety of algorithms for different learning approaches. We propose a novel neuro-emulator that provides the computational ability for the stated requirements. While meeting all the identified requirements the new architecture maintains a high performance during learning. To show the capabilities of the proposed machine we present four diverse learning algorithms and step through the execution of each using the proposed architecture. We include an evaluation of the machine performance as well as a comparison with other architectures. It is shown that with a modest amount of hardware the proposed architecture yields an extremely high number of connections per second.     

曙光星云分布式文件系统:海量小文件存取

随着互联网应用的发展和云计算的兴起,在线图片、音频、视频以及微博等服务逐渐广泛发展,这些应用展示了与传统应用截然不同的数据访问和存储模式.数据中心内每秒钟都有大量较小文件的生成、分析和返回,这些应用对高并发海量文件的高吞吐、低延迟读写提出了新的挑战.提出基于分布式表存储的全新的分布式文件系统HVFS来管理数以十亿计的文件,并同时支持高吞吐和低延迟的文件访问.HVFS通过改进分布式可扩展哈希来管理元数据、日志结构的格式和列存储来利用时空局部性.本文描述了HVFS的设计和实现并进行了中等规模的实验.实验显示HVFS的表存储结构能够线性的扩展,并在82个结点上提供超过240,000次/秒、100,000次/秒的数据(<1KB)写和读;基于FUSE的实现在32个节点上提供超过180,000个/秒的文件创建速度.     

Enabling Grid Access to Mass Storage

The purpose of the EDG Storage Element (SE) is to provide a uniform Grid interface to mass storage systems. We needed an extensible and flexible architecture, and chose one based on message processing. Not only has this architecture provided a solid framework for meeting changing requirements, it also provides scope for building an extensible and scalable storage management system. While distributed SEs in a Grid are managed by higher level replication services, our request processing model opens possibilities for distributing a single SE across several nodes, allowing further opportunities for optimisation. In this paper, we discuss applications of distributed, transactional processing in mass storage, and its implications for optimisations based on access patterns. This work is funded by EU contract IST-2000-25182.     

数值模拟网格数据模型设计与实现

在等离子体动力学、电磁学理论等物理问题的数值模拟中,各类数值模拟程序产生了大量复杂结构的科学数据.一方面,计算程序需要以高效率的I/O方式存储数据,另一方面,数据需要在各类程序间很容易地交换与共享.随着数据的规模与复杂度不断增加,传统数据管理方式的局限性日益突出.为此,设计了面向计算物理领域的数据存储模型--数值模拟网格数据模型(JAD),引入元数据管理机制,对数值模拟程序数据对象进行抽象与封装,在HDF5软件库基础上实现了高层I/O函数库(JADLib),集成先进的数据存储技术,提供直观、易用的应用程序编程接口(API),使得数值模拟数据以统一格式高效率地存储.目前,JADLib已推广应用于高功率微波、惯性约束聚变等领域多个数值模拟程序中,与元数据管理系统(JADIS)、并行可视化系统(JaVis)建立了耦合,使得用户可以直接利用这些系统进行数据的浏览、分析及可视化,促进了应用程序间的数据共享.     

Storage allocation policies for time-dependent multimedia data

Multimedia computing requires support for heterogeneous data types with differing storage, communication and delivery requirements. Continuous media data types, such as audio and video, impose delivery requirements that are not satisfied by conventional physical storage organizations. In this paper, we describe a physical organization for multimedia data based on the need to support the delivery of multiple playout sessions from a single rotating-disk storage device. Our model relates disk characteristics to the different media recording and playback rates and derives their storage pattern. This storage organization guarantees that as long as a multimedia delivery process is running, starvation will never occur. Furthermore, we derive bandwidth and buffer constraints for disk access and present an approach to minimize latencies for non-continuous media stored on the same device. The analysis and numerical results indicate the feasibility of using conventional rotating magnetic disk storage devices to support multiple sessions for video-on-demand applications     

融合通信系统中分布式存储引擎的研究

融合通信是当今计算机应用领域研究热点之一,人们对融合通信系统中应用服务的要求也越来越高.在数据存取方面,基于传统关系型数据库或者基于传统文件系统的存储方式已经越来越不能满足应用的需求.随着Hadoop技术以及相关子系统的发展,分布式存储的优势日渐明显.因此,本文在分析HBase、Hive各自特点及其体系结构的基础上,结合融合通信具体项目提出了基于HBase-Hive集成设计的存储引擎设计方法,以此来解决融合通信系统中数据安全性、数据获取效率等方面不满足的情况.通过对比实验表明,该设计方案提高系统数据查询获取效率,也为后续数据挖掘方面的开发做好准备.     

An intelligent memory caching architecture for data-intensive multimedia applications

With the rapid developments in cloud computing and mobile networks, multimedia content can be accessed conveniently. Recently, some novel intelligent caching-based approaches have been proposed to improve the memory architectures for multimedia applications. These applications often face bottleneck related challenges which result in performance degradation and service delay issues. Intelligent multimedia network applications access the shared data by using a specific network file system. This results in answering the processing related constraints on hard-drive storage and might result in bringing bottleneck issues. Therefore, to improve the performance of these multimedia network applications, we present an intelligent distributed memory caching system. We integrate the multimedia application message passing interface in a multi-threaded environment and propose an algorithm which can handle concurrent response behavior for different multimedia applications. Results demonstrate that our proposed scheme outperforms traditional approaches in terms of throughput and file read access features.

     

高性能分布计算的数据管理与优化

在高性能计算程序对海量分布存储数据的操控中,有效的数据管理很重要。该文提出一个新的高性能分布计算的数据管理与优化系统,它包含一个元数据管理系统和存储系统,提供一个容易使用且能自动进行存储访问优化的平台。该平台采用的多存储资源体系结构能够满足性能和存储容量需求,并能自适应地利用当前的I/O优化方法。     

基于FPGA的DDR SDRAM控制器设计与实现

随着DDR SDRAM的广泛应用,为满足不同平台下的访存需求,文章设计并实现了一种基于FPGA的DDRSDRAM控制器,通过分级流水结构提高了系统性能,并通过参数的在线配置满足不同内存颗粒的参数需求,保证了控制器的灵活性和可扩展性。     

The basics of reliable distributed storage networks

Because of storage protocols that operate over extended distances, various distributed storage applications that improve the efficiency and reliability of data storage are now possible. Distributed storage applications improve efficiency by allowing any network server to transparently consolidate and access data stored in multiple physical locations. Remote backup and mirroring improve the system's reliability by copying critical data. These processes improve efficiency by eliminating backup downtime and manual backup operations. Business continuity and disaster recovery capabilities enable enterprises to recover quickly and transparently from system failure or data loss. Storage protocols and gateway devices enable rapid and transparent data transfer between mainframe applications and open-systems applications. NAS applications provide shared file access for clients using standard LAN-based technology, and can integrate with SAN architectures to provide truly distributed network capabilities. All these distributed storage network applications enable IT managers to improve data availability and reliability while minimizing management overhead and costs.     

Improving Data Accessibility with File Area Networks

Many companies have complex information systems with a growing amount of unstructured data - information that isn't organized into fixed categories. Databases have built-in tools for understanding and managing structured data. However, managing unstructured data - including migrating it to new storage equipment, backing it up, maintaining user access to material, and keeping information to satisfy governmental regulatory requirements - is a challenge. The current approaches - which entail manually moving files and examining the information they contain to determine how best to handle them - are not adequate for coping with the exploding quantity of unstructured data. Because of this, companies are turning to a new file-management approach: the file area network. A FAN is a set of technologies that organize, route, switch, replicate, and otherwise handle files over networks, all without interrupting user access to information, thereby providing a flexible, intelligent, cost-effective platform to move and manage data.     

面向边缘计算的属性加密方案

传统云环境下的属性加密方案在判定用户访问权限时通常仅依据年龄和职业等用户常规属性,而忽视了访问时间和位置的约束问题。为较好满足边缘计算的实时性和移动性需求,提出一种支持时间与位置约束的多授权外包属性加密方案。通过将时间域与位置域信息同时引入属性加密过程,实现更细粒度的访问控制。采用多授权机构共同管理属性信息,解决单授权机构的性能瓶颈问题,满足用户跨域访问需求。针对边缘计算中移动终端资源受限问题,将大部分解密计算外包至边缘节点,减轻移动终端设备负担。分析结果表明,在边缘计算环境下,该方案以较低的计算和存储开销实现了具有时间和位置约束的访问控制,并且可有效保障用户数据安全。     

树结构在几种特殊签名中的应用研究

树结构的层次性和链式的存储方式使其在数据查找的实现中体现出很高的效率,在数据插入和删除等操作的实现中无需移动大量节点,因而不仅在计算机科学领域具有广泛的应用,在实现特殊的信息安全方面也具有独特的作用。文章研究如何将树结构应用于仅增签名、前向安全签名、递增签名、可修订签名、内容抽取签名等特殊签名方案的问题,比较分析基于树结构的方案与普通方案在功能和效率上的差异。     

Adaptive prefetching and storage reorganization in a log-structuredstorage system

We present a storage management system that has the ability to adapt to the data access characteristics of the application that uses it based on collection and analysis of runtime statistics. This feature is especially useful in the storage management layer of database systems, where applications exhibit relatively predictable access patterns. Adaptive reorganization is performed by the storage management system in a manner that optimizes the access patterns of the system for which it is used. We enhance the log-structured storage system that naturally caters for write optimization, with the addition of a statistics collection mechanism to determine data access patterns of applications. The storage system can serve as a testbed for a variety of statistics analysis and clustering mechanisms. Higher level application-specific data clustering mechanisms can be used to override the storage system's low-level clustering mechanisms. In addition, the analysis techniques and reorganization scheme can be used in other storage systems. Performance results from our prototype show potential response time speedups of up to 83 percent over the basic log-structured file system in the best case, using a combination of storage reorganization and prefetching     

工程数据库管理系统设计、实现的几个关键问题

本文针对工程领域的特点，提出基于关系／网状的语义超图模型描述，以表达工程中不同语义类的多对多复杂对象关系；将该模型描述和表达的管理机制和基本数据在内存中统一管理和运行，使工程数据的存取效率大为提高；增加了大量通用和特殊功能，形成了功能更强、使用方便的Ｃ语言界面．在上述基础上自行研制并有自主版权的工程数据库管理系统ＥＤＢＭＳ（ｅｎｇｉｎｅｅｒｉｎｇｄａｔａｂａｓｅｍａｎａｇｅｍｅｎｔｓｙｓｔｅｍ）已在建筑ＣＡＤ领域中得到实际应用.