Linux系统下实现网络内存共享关键技术探析

网络内存共享的出现主要借鉴了传统的网格计算技术和集群内存共享技术。论文从动态函数截获,缓存数据组织与管理和异步缓存数据写入三方面探讨了网络内存共享的关键技术。     

一种内存网格的数据预取算法

内存网格(RAM(random access memory) grid)是一种面向广域网上内存资源共享的新型网格系统.它的主要目标是在物理内存不足的情况下,提高内存密集型应用或IO密集型应用的系统性能.内存网格的应用效果取决于网络通信开销.在减少或隐藏网络通信开销的情况下,其性能可以进一步提高.通过对内存网格的分析,设计了一种基于"推"数据的内存网格预取机制.借助数据挖掘领域中序列模式挖掘的方法,提出了相应的预取算法.通过基于真实运行状态的模拟,对预取算法进行了评估和验证.     

基于内存网格的磁盘缓存设计与实现

内存对计算机系统的性能具有重要影响,内存网格能够共享跨域的开放网络环境中的内存资源,以磁盘缓存的形式提高系统性能.为实现缓存对应用的透明性,提出了动态修改操作系统内核的二进制代码.实现文件系统读写流程的截获和重定向;并提出了基于内核线程的异步缓存写入方法.提高写缓存的效率.通过原型系统及实验,说明上述方法既不需要修改鹰用程序、也不需要修改操作系统源代码,并且能充分利用共享的内存资源+提高系统的I/O性能.     

PAMM:一种面向基于内存共享的域间通信的优化模型

云计算平台和虚拟化技术的结合为虚拟机域间通信带来了新的需求,基于内存共享的域间通信可以提高运行在同一物理机上的虚拟机间的通信效率。但是,基于内存共享的域间过程中产生的上下文状态切换限制了其优化能力。引入一种新的内存共享模型PAMM,即通过添加一个管理模块对内存共享过程中所传递的内存页进行聚合管理,减少申请超级调用的次数,以达到减少状态切换的目的。实验表明,PAMM能够提升基于内存共享的域间通信的通信效率。     

基于分布式构件服务器性能的内存消耗技术

针对分布式构件服务器集群系统性能管理问题,研究分布式构件系统的内存消耗技术,在运行时内存消耗研究的基础上,设计了内存消耗模型,进一步分析降低物理内存消耗的关键因素,最后提出了内存优化策略。     

集群上软件事务内存的层次化冲突检测

大多数事务内存研究都在多核处理器上进行,只有少数研究针对集群计算环境。现有的集群事务内存性能较差,因为事务内存的运行需要进行大量的远程内存访问,而集群的内存分布性使得远程内存访问的性能较差。事务内存运行中的冲突检测是进行远程内存访问最多的操作。我们提出了一种层次化的冲突检测方法,将其分为两个步骤进行,这种层次化的策略可以使集群上的软件事务内存获得高性能。     

内存网格中的自主协同缓存技术研究

本文基于原有工作提出了利用内存网格进行磁盘缓存的技术,从而组织和共享网络上分布的空闲内存资源,提高具有大量磁盘IO的节点的系统性能;给出了一种基本设计方案,并提出了缓存节点的时间分布和空间合并等自主协同机制,进一步优化了系统性能;最后通过基于实际采集数据的模拟,对方案进行了评价。     

LNMS网络内存系统及性能优化技术

在Linux环境下设计并实现了一个两级服务结构的网络内存系统(LNMS)。LNMS较传统网络内存系统具有更好的可扩展性。对LNMS提出了两种性能优化技术：预取和主动内存技术。在预取优化方面,针对并发应用提出一种M-PPM算法。主动内存技术则发掘了内存服务器的计算能力。实验表明,两种优化技术可有效地提升网络内存系统的性能。     

一种面向Xen虚拟计算环境的运行时内存泄漏检测方法

虚拟计算环境中系统性能的稳定性问题研究对于云计算相关技术的研究和应用具有重要的理论和实际意义.长时间不停机系统的内存泄漏可能给实际应用带来严重后果,在虚拟计算环境中检测运行时内存泄漏是一个极具挑战性的问题.针对该问题,对内存泄漏的现象进行了分类.基于Xen虚拟机构建并实现了一种面向Xen虚拟计算环境的虚拟化内存泄漏检测(virtualization memory leak detection, VMLD)的方法,提出了相应的检测规则.通过修改虚拟机管理器,设计超级调用,实现了内部缓冲区维护、控制、拦截、监视等模块.实验结果表明,VMLD方法能有效地检测出运行时内存泄漏,并且具有较好的性能.     

基于内存映射文件的复杂对象快速读取方法

当前内存数据库(NoSQL)、嵌入式数据库技术在高并发高性能系统中得到了广泛的应用,但对于复杂对象数据的读取效率仍然低下,研究发现主要性能瓶颈有两个:一是内核态与用户态间的内存拷贝,拷贝消耗时间与复杂对象的数据量成线性增长;二是从数据库数据到运行时数据对象的格式转化操作,不但需要开辟新的内存空间存储运行时数据对象,而且还需要解析原始数据并拷贝至新对象之中。为此,提出了一种基于内存映射(memory mapping)文件的复杂对象共享读取方法。借助内存映射文件与自定义内存分配器,实现了结构复杂的C++标准模板库容器对象跨进程无拷贝、无格式转化的共享,有效降低了数据读取延时。通过性能的分析比较表明,与NoSQL内存数据库、嵌入式数据库比,读取性能效率提升10倍以上。再加上底层技术成熟稳定,复用了标准模板库,具有开发成本低、可维护性强、实用性高等优点,因此,适用于高并发高性能的高可用后台服务系统。     

访存密集型应用在SMP机群系统中的性能分析

SMP机群系统因其良好的性价比、卓越的可扩展性与可用性，逐渐成为当前高性能计算机领域的主流结构．这种结点内共享存储、结点间消息传递的两级混合结构是目前并行计算研究的热点,在单个SMP结点中，总线和内存带宽是否满足CPU和I／O的需求对于访存密集型应用的性能影响很大。本文针对访存密集型应用的特点测试分析了在SMP机群中访存冲突对系统性能的影响，结果表明我们的SMP结点存在性能瓶颈，这种量化分析对于设计大规模的基于SMP的机群系统有很好的指导意义．     

基于共享物理存储空间的松耦合存储管理机制

引入一种NUMA多处理器原型系统,分析该系统上的操作系统对物理内存管理的特点。基于该系统设计和实现了一个全局共享内存系统,使操作系统充分利用整个系统上的物理内存,减少应用程序的执行时间。实验结果表明,该系统能够更好地支持存储密集型 应用。     

一种主动存储计算机模型

提出了一种新型的主动存储计算机模型，其特征是存储器单元不仅可以有传统的读写操作，还可以有执行操作，这个模型扩展了传统的随机访问计算机（RAM）模型，可以用来研究一些新型计算模式的体系结构机制和算法设计与分析，如SoC、可重构计算、并行计算、网格计算、万维网服务等，对于网格计算，这个模型可用于研究网格协议与执行机制、网格算法的正确性和复杂度以及网格客户端与服务器端的有效交互。     

Effectively utilizing global cluster memory for large data-intensive parallel programs

Large scientific parallel applications demand large amounts of memory space. Current parallel computing platforms schedule jobs without fully knowing their memory requirements. This leads to uneven memory allocation in which some nodes are overloaded. This, in turn, leads to disk paging, which is extremely expensive in the context of scientific parallel computing. To solve this problem, we propose a new peer-to-peer solution called parallel network RAM. This approach avoids the use of disk, better utilizes available RAM resources, and will allow larger problems to be solved while reducing the computational, communication, and synchronization overhead typically involved in parallel applications. We proposed several different parallel network RAM designs and evaluated the performance of each under different conditions. We discovered that different designs are appropriate in different situations.     

元计算环境下的目录服务机制研究

In Grid Computing environments, in order to reasonablely avail the super-speed host, great capacity storage,network bindwidth etc grid resources, and coordination use of the protocols and applications algorithms, the directory service system mechanism must be sufficiency considered. Validate selection of the available resources and related and latest information status. Discuss meta directory service mechanism based on LDAP for grid computing,point out the directory service system‘s major technique. At last, point out the related research contents of directory service.     

Robust and efficient memory management in Apache AsterixDB

Traditional relational database systems handle data by dividing their memory into sections such as a buffer cache and working memory, assigning a memory budget to each section to efficiently manage a limited amount of overall memory. They also assign memory budgets to memory-intensive operators such as sorts and joins and control the allocation of memory to these operators; each memory-intensive operator attempts to maximize its memory usage to reduce disk I/O cost. Implementing such memory-intensive operators requires a careful design and application of appropriate algorithms that properly utilize memory. Today's Big Data management systems need the ability to handle large amounts of data similarly, as it is unrealistic to assume that truly big data will fit into memory. In this article, we share our memory management experiences in Apache AsterixDB, an open-source Big Data management software platform that scales out horizontally on shared-nothing commodity computing clusters. We describe the implementation of AsterixDB's memory-intensive operators and their designs related to memory management. We also discuss memory management at the global (cluster) level. We conducted an experimental study using several synthetic and real datasets to explore the impact of this work. We believe that future Big Data management system builders can benefit from these experiences.     

Memory channel network for PCI

A memory-based networking approach provides clusters of computers up to 1,000 times the communication performance of conventional networks, with no compromise in cost or reliability. The memory channel for PCI's performance gains are the result of a system design approach that exploits natural cluster constraints to define a memory-based network. MC implements a form of virtual shared memory that permits applications to completely bypass the operating system and perform cluster communication directly from the user level. The hardware's simple and powerful communication model supports error handling at almost no cost or complexity to the application; guaranteed ordering under errors is the key innovation. The end result: Real-world cluster communication latency dropped by up to two orders of magnitude, and overhead by up to three orders of magnitude. These improvements elevate a lowly set of standard PCI computers running Unix into an impressive, highly available, parallel computing system     

Adaptive and Reliable Paging to Remote Main Memory

Workstation clusters provide significant aggregate amounts of resources, including processing power and main memory. In this paper we explore the collective use of main memory in a workstation cluster to boost the performance of applications that require more memory than a single workstation can provide. We describe the design, simulation, implementation, and evaluation of a pager that uses main memory of remote workstations in a workstation cluster as a faster-than-disk paging device and provides reliability in case of single workstation failures and adaptivity in network and disk load variations. Our pager has been implemented as a block device driver linked to the Digital UNIX operating system, without any modifications to the kernel code. Using several test applications we measure the performance of remote memory paging over an Ethernet interconnection network and find it to be up to twice as fast as traditional disk paging. We also evaluate the performance of various reliability policies and demonstrate their feasibility even over low bandwith networks such as Ethernet. We conclude that the benefits of reliable remote memory paging in workstation clusters are significant today and are likely to increase in the near future.     

网格计算技术及应用综述

Grid computing is a new kind of distributed computing technology and computing environment ,and also an attentive hot point of information technology in the world. It launches a space for new generation internal application,due to its supporting the complicated service efficiently and useable resource in the Internet. This paper sums up the results and applications instance in the grid computing in the world during recent years ,and then analyzes emphatically the system architecture ,components ,working principle in the grid computing and some typical grid systems ,and then discusses the problems of the large scale science computing and network service in grid computing in China,and alsopoints out the future trends.