A Dynamic Memory Allocation Optimization Mechanism Based on Spark

Spark is a distributed data processing framework based on memory. Memory allocation is a focus question of Spark research. A good memory allocation scheme can effectively improve the efficiency of task execution and memory resource utilization of the Spark. Aiming at the memory allocation problem in the Spark2.x version, this paper optimizes the memory allocation strategy by analyzing the Spark memory model, the existing cache replacement algorithms and the memory allocation methods, which is on the basis of minimizing the storage area and allocating the execution area according to the demand. It mainly including two parts: cache replacement optimization and memory allocation optimization. Firstly, in the storage area, the cache replacement algorithm is optimized according to the characteristics of RDD Partition, which is combined with PCA dimension. In this section, the four features of RDD Partition are selected. When the RDD cache is replaced, only two most important features are selected by PCA dimension reduction method each time, thereby ensuring the generalization of the cache replacement strategy. Secondly, the memory allocation strategy of the execution area is optimized according to the memory requirement of Task and the memory space of storage area. In this paper, a series of experiments in Spark on Yarn mode are carried out to verify the effectiveness of the optimization algorithm and improve the cluster performance.     

Extending SSD Lifespan with Comprehensive Non-Volatile Memory-Based Write Buffers

Journal of Computer Science and Technology - New non-volatile memory (NVM) technologies are expected to replace main memory DRAM (dynamic random access memory) in the near future. NAND flash...     

基于NB分类器重访概率预测的Web缓存替换策略

Web缓存是用来解决网络访问延迟和网络拥塞问题，缓存替换策略直接影响缓存的命中率。为此，提出一种朴素贝叶斯（NB）分类器重访概率预测的Web缓存替换策略；根据用户之前访问日志，通过分区操作提取多项特征来表示每次访问的对象，并构建特征数据集；训练NB分类器，用来确定缓存中对象被再次访问的概率，为对象分配权重；结合LRU策略来合理删除一些对象。仿真结果表明，提出的策略在保证较高命中率的同时有效降低了执行时间。     

一体化标识网络映射缓存DoS攻击防范方法研究

为了抵御一体化标识网络中接入路由器可能遭受的映射缓存DoS攻击,本文提出了一种基于双门限机制的映射缓存DoS攻击防范方法.该方法设计了一种基于迭代思想的谜题机制降低映射缓存中映射信息条目的增加速率,并采用了映射信息可信度算法识别和过滤映射缓存中恶意的映射信息条目.仿真实验与性能分析表明,该方法能够有效地抵御映射缓存DoS攻击,防止映射缓存溢出.     

Optimised memory allocation for less false abortion and better performance in hardware transactional memory

ABSTRACT

This paper introduces and tackles a special performance hazard in Hardware Transactional Memory (HTM): false abortion. False abortion causes many unnecessary transaction abortions in HTM and can greatly impact the performance, making HTM not that useful when it is adopted as a fast path for Software Transactional Memory. By introducing a new memory allocator design, we are able to put objects that are likely to be accessed together from different threads into different cache lines and thus avoid conflicts of hardware transactions in different threads. Experiments show that our method can reduce 47% of transaction abortion and achieve a speedup of up to 1.67× (averagely 22%), yet only consume 14% more memory, showing great potential to enhance current HTM technology.     

Research on HTML5 application cache poison attack

HTML5 application cache (AppCache) allowed Web browser to access Web offline.But it also brought a new method of cache poisoning attack that was more persisting.As for websites which used the AppCache,a novel poisoning method RFTM (replace file twice method),in which the attacker replaced the manifest file twice to poison the client’s AppCache,was proposed.Compared with the original attack,the legal server would not receive abnormal HTTP requests from the client in the attack.Therefore,changing the server configuration could not prevent the client from the RFTM AppCache poisoning.To avoid the attack mentioned above,a lightweight signature defense scheme Sec-Cache in application layer was designed.Furthermore,experiments show that it has good performance and compatibility.     

A soft‐cache strategy for pathologist's navigation in virtual microscopy

Navigation through large microscopical images demands special characteristics like flexibility to access image data and progressive recovery or relevant information, i.e., several qualities, magnifications, and random accesses to any desired Window of Interest in the Virtual Slide (VS). Nowadays there exist systems that allow such interaction, but with a certain delay, which is dependent on the application. It has been shown that caching or prefetching policies can speed up interaction with these systems. This article presents an optimal soft‐cache strategy, which improves the navigation times in virtual microscopy. The entire method includes an optimal soft‐cache strategy and a dynamical probabilistic model of a pathologist's navigation. This strategy was implemented as a Client‐Server application, using the JPEG2000‐JPIP standard and evaluated using different navigation patterns, namely, four different pathologists exploring 10 VS, stained with different dyes. The present approach was compared with a conventional soft‐cache method and the cache performance improved, in average, in about a 10%. Microsc. Res. Tech., 2011. © 2010 Wiley‐Liss, Inc.     

异质存储系统中的高速缓存机制研究

存储子系统是嵌入式系统的重要组成部分.由于传统存储系统的设计已经无法满足容量日益增长的需求,固态存储器的应用越来越广泛.针对固态存储系统的存取速率慢的问题,目前常用的优化技术主要有缓存和并行读写技术.然而,在大多数应用和研究中,如何将这两类技术进行融合是目前遇到的一个重大挑战.对此,本文提出一种融合了缓存技术和并行读写...     

云计算分布式缓存技术及其在物联网中的应用

物联网是信息技术发展到一定阶段的产物,而云计算平台是物联网应用的基础。文章从当前云计算应用所面临的问题和缺陷出发,介绍了云计算分布式缓存的部署方式、功能架构及关键技术,并说明了分布式缓存高性能、高吞吐、高可靠性、高扩展性等优势和特性。文章解决了物联网应用普遍面临的数据可靠性、大容量内存共享、多模块数据一致保障、线性扩容等难题,为物联网平台云化架构的底层支撑奠定了基础。