内存数据库关键技术研究

随着存储器价格的下降、64位系统的引入和广泛使用,内存数据库（MMDB）技术得到了更广泛的研究,并出现了一些原型系统和商业系统。首先对比了内存数据库与磁盘数据库在查询处理、索引技术等几个方面的差异,总结分析了内存数据库的关键技术和研究内容,内存数据库需要研究的问题,最后展望了未来的研究方向。     

Merging radiometric grids using histogram matching

Large-scale compilations of airborne geophysical data are an important part of regional-scale mineral exploration and in most cases, the compilation involves merging datasets of varying quality, acquired over a long period of time. Merging radiometric datasets involves the additional complication that many older surveys were flown with uncalibrated spectrometer systems. Conventional grid merging is based on calculation of base-level shift and scaling factors for each grid with the grid edges feathered. However, even after these corrections, soil moisture differences between surveys acquired over a long period of time and at different seasons when soil moisture content varied introduce scaling errors into the estimated radioelement concentrations. An alternative method based on histogram matching is suggested as an improvement on existing methods. Source code in Matlab format is available from the server at www.iamg.org.     

内存数据库的一种数据组织方式

近年来,电信和金融领域的主要应用已经变成数据密集型应用,数据库系统在其中扮演了关键的角色.但传统的基于磁盘的关系数据库系统却不能满足上述应用高性能、实时/近实时数据访问的要求,内存数据库系统则可以很好地满足各种应用系统的实时数据管理需求.介绍了内存数据库的一种内部数据组织方式,并对各种核心对象的存储及数据结构进行了分析.     

Merging local patterns using an evolutionary approach

This paper describes a Decentralized Agent-based model for Theory Synthesis (DATS) implemented by MASETS, a Multi-Agent System for Evolutionary Theory Synthesis. The main contributions are the following: first, a method for the synthesis of a global theory from distributed local theories. Second, a conflict resolution mechanism, based on genetic algorithms, that deals with collision/contradictions in the knowledge discovered by different agents at their corresponding locations. Third, a system-level classification procedure that improves the results obtained from both: the monolithic classifier and the best local classifier. And fourth, a method for mining very large datasets that allows for divide-and-conquer mining followed by merging of discoveries. The model is validated with an experimental application run on 15 datasets. Results show that the global theory outperforms all the local theories, and the monolithic theory (obtained from mining the concatenation of all the available distributed data), in a statistically significant way.     

Incremental recovery in main memory database systems

Recovery activities, like checkpointing and restart, in traditional database management systems are performed in a quiescent state where no transactions are active. This approach impairs the performance of online transaction processing systems, especially when a large volatile memory is used. An incremental scheme for performing recovery in main memory database systems (MMDBs), in parallel with transaction execution, is presented. A page-based incremental restart algorithm that enables the resumption of transaction processing as soon as the system is up is proposed. Pages are recovered individually and according to the demands of the post-crash transactions. A method for propagating updates from main memory to the backup database on disk is also provided. The emphasis is on decoupling the I/O activities related to the propagation to disk from the forward transaction execution in memory. The authors also construct a high-level recovery manager based on operation logging on top of the page-based algorithms. The proposed algorithms are motivated by the characteristics of large MMDBs, and exploit the technology of nonvolatile RAM     

Sorting unsorted and partially sorted lists using the natural merge sort

A version of the natural merge sort is shown to be an efficient method for both nearly sorted data and for random data. This method is compared with quickersort, 2-way merge sort, and Cook and Kim's method for various sortedness ratios.     

Time-shift replacement algorithm for main memory performance optimization

Page replacement algorithms of main memory in modern operating systems are crucial in system performance. When memory is full, a page replacement algorithm exploits temporal locality and frequency of page references to evict the page that is least likely to be accessed in the near future. Subsequently, loading the majority of data directly from memory improves performance by reducing I/O waits of accessing slow storage. Research of replacement algorithms that maximizes hit ratio while incurring as less overhead as possible has been constantly studied. In this paper, we propose a time-shift least recently used (TSLRU) algorithm that converts frequency information of page references into temporal locality. Frequent accesses of a page are thus recognized and accumulated in terms of time. Moreover, pages being loaded into memory for the first time are not necessarily the most recently used pages. As a result, one-pass pages are evicted sooner in our algorithm than in traditional LRU algorithm. Our performance evaluations show that the TSLRU outperforms conventional page replacement algorithms on both artificial and real application traces. For example, hit ratio of TSLRU advances ARC by \(4.17\%\) and LRU by \(5.91\%\) on normal distributed workloads. Moreover, TSLRU outperforms ARC by over \(2\%\) on half of the application traces tested.     

Suffix trees for inputs larger than main memory

A suffix tree is a fundamental data structure for string searching algorithms. Unfortunately, when it comes to the use of suffix trees in real-life applications, the current methods for constructing suffix trees do not scale for large inputs. As suffix trees are larger than the input sequences and quickly outgrow the main memory, the first attempts at building large suffix trees focused on algorithms which avoid massive random access to the trees being built. However, all the existing practical algorithms perform random access to the input string, thus requiring in essence that the input be small enough to be kept in main memory. The constantly growing pool of string data, especially biological sequences, requires us to build suffix trees for much larger strings.     

Road grade estimation for look-ahead vehicle control using multiple measurement runs

Look-ahead cruise controllers and other advanced driver assistance systems for heavy duty vehicles require high precision digital topographic road maps. This paper presents a road grade estimation algorithm for creation of such maps based on Kalman filter fusion of vehicle sensor data and GPS positioning information. The algorithm uses data from multiple passes over the same road to improve previously stored road grade estimates. Measurement data from three test vehicles and six experiments have been used to evaluate the quality of the obtained road grade estimate compared to a known reference. The obtained final grade estimate compares favorably to one acquired from a specialized road grade measurement vehicle with a DGPS receiver and inertial measurement unit, with an average root mean square error of 0.17% grade.     

Dynamic recognition prefetch engine for DRAM-PCM hybrid main memory

The Journal of Supercomputing - This research is to design an effective prefetching method required for hybrid main memory systems consisting of dynamic random-access memory (DRAM) and phase-change...     

Effects of update techniques on main memory database systemperformance

Update technique is an important issue related to database recovery. In a main memory database environment, transaction execution can be processed without any I/O, and all I/O operations involved are for recovery purposes. The efficiency of update techniques therefore has an important impact on the performance of main memory database systems. We compared the techniques of immediate and deferred update based on a database machine, MARS. The simulation results showed that immediate update outperforms deferred update unless system failure is a frequent occurrence     

Distributed load balancing for parallel main memory hash join

Parallel joins have been widely studied during the past decade and a number of efficient algorithms were presented. While it is known that the performance of these algorithms may suffer greatly in the presence of skewed input data, the work on load balancing schemes for parallel join has been limited. The main contribution of this paper is the development and analysis of a new distributed data structure and an effective load balancing scheme for parallel main memory hash join on NUMA architecture. Multiprocessors based on this architecture are scalable in both size of main memory and number of processors, and provide very high memory bandwidth. The load balancing scheme is based on random probing to avoid the hot spot problems caused by probing sequentially. We have modeled this load balancing scheme both analytically and experimentally. The experiments were run on a BBN TC2000 multiprocessor system     

基于SRAM和PRAM混合主存设计

由于DRAM芯片超高的静态功耗,使得利用DRAM构建高性能计算机系统中的大容量主存遇到能耗过大问题,这激发了对新型大容量主存结构的研究。针对上述问题,设计了一种基于SRAM和PRAM的混合主存系统,该系统将SRAM作为PRAM的专用写缓存,并将改进后的LRFU算法应用到SRAM写缓存,从而在对主存系统性能影响不大的前提下,有效降低主存系统的能耗和延长PRAM的可用时间。仿真结果显示,所设计的混合存储结构的能耗-延时积（EDP）为纯DRAM存储结构的40%;此外,与纯PRAM存储结构相比,可使PRAM的写操作次数下降28.5%,与将SRAM作为Cache相比,PRAM写次数下降13%。     

Direct RAMbus technology: the new main memory standard

Providing three times the memory bandwidth of the 66-MHz SDRAM subsystem, Direct RDRAM modules fit seamlessly into the existing mechanical space and airflow environment of the industry-standard PC chassis     

空间实时内存数据库恢复机制研究与实现

为了满足空间应用对数据管理的实时性和可靠性的要求,针对Swallow空间实时内存数据库设计了一个内存数据库的恢复机制,结合日志驱动修改、影子页面、模糊检查点的思想实现了内存数据库数据和日志的快速备份以及在内存数据库出现事务故障和系统故障时的恢复策略.实验表明该策略保证了事务的并发度,并且在事务故障和系统故障时都有着较高的恢复效率.     

Energy efficient task allocation for hybrid main memory architecture

Compared with the conventional dynamic random access memory (DRAM), emerging non-volatile memory technologies provide better density and energy efficiency. However, current NVM devices typically suffer from high write power, long write latency and low write endurance. In this paper, we study the task allocation problem for the hybrid main memory architecture with both DRAM and PRAM, in order to leverage system performance and the energy consumption of the memory subsystem via assigning different memory devices for each individual task. For an embedded system with a static set of periodical tasks, we design an integer linear programming (ILP) based offline adaptive space allocation (offline-ASA) algorithm to obtain the optimal task allocation. Furthermore, we propose an online adaptive space allocation (online-ASA) algorithm for dynamic task set where arrivals of tasks are not known in advance. Experimental results show that our proposed schemes achieve 27.01% energy saving on average, with additional performance cost of 13.6%.     

Distributed shared memory for roaming large volumes

We present a cluster-based volume rendering system for roaming very large volumes. This system allows to move a gigabyte-sized probe inside a total volume of several tens or hundreds of gigabytes in real-time. While the size of the probe is limited by the total amount of texture memory on the cluster, the size of the total data set has no theoretical limit. The cluster is used as a distributed graphics processing unit that both aggregates graphics power and graphics memory. A hardware-accelerated volume renderer runs in parallel on the cluster nodes and the final image compositing is implemented using a pipelined sort-last rendering algorithm. Meanwhile, volume bricking and volume paging allow efficient data caching. On each rendering node, a distributed hierarchical cache system implements a global software-based distributed shared memory on the cluster. In case of a cache miss, this system first checks page residency on the other cluster nodes instead of directly accessing local disks. Using two Gigabit Ethernet network interfaces per node, we accelerate data fetching by a factor of 4 compared to directly accessing local disks. The system also implements asynchronous disk access and texture loading, which makes it possible to overlap data loading, volume slicing and rendering for optimal volume roaming.