Improving energy efficiency for flash memory based embedded applications

The JFFS2 file system for flash memory compresses files before actually writing them into flash memory. Because of this, multimedia files, for instance, which are already compressed in the application level go through an unnecessary and time-consuming compression stage and cause energy waste. Also, when reading such multimedia files, the default use of disk cache results in unnecessary main memory access, hence an energy waste, due to the low cache hit ratio. This paper presents two techniques to reduce the energy consumption of the JFFS2 flash file system for power-aware applications. One is to avoid data compression selectively when writing files, and the other is to bypass the page caching when reading sequential files. The modified file system is implemented on a PDA running Linux and the experiment results show that the proposed mechanism effectively reduces the overall energy consumption when accessing continuous and large files.     

一种基于YAFFS2的静态磨损均衡算法设计

针对已有算法大多基于FTL(Flash Translation Layer)闪存存储体系设计,无法直接应用到闪存文件系统的情况,基于YAFFS2闪存文件系统设计了一种静态磨损均衡算法。考虑不同数据冷热属性的差异,设计了两种不同的块分配策略,在减缓最大擦除次数增长速度的同时,加强对冷数据占用块的利用;利用闪存块擦除次数在物理空间上分布的随机性,简化了块分配策略中的搜索过程,降低了算法复杂度;采用不均衡度阈值作为算法触发条件,保证良好的均衡效果。实验结果表明,算法降低了擦除次数方差,提高了闪存磨损均衡度,可有效延长闪存寿命,提高闪存存储系统的可靠性。     

基于重复数据消除的差异备份方法

为消除重复数据对数据传输和存储产生的影响,提出一种基于重复数据消除的差异备份方法。通过将文件的块按照一定区间划分固定大小并采用Hash表对文件块进行唯一性标识,使Rsync算法能检测不同文件之间的重复数据,通过分割Hash表,使块实现局部匹配,并利用校验和文件实现文件不同版本的差异传输。实验结果表明,与Rsync算法相比,该方法能有效减少传输的数据量,降低备份中心的存储量,提高块查找的效率。     

基于文件分割的二进制大对象存取算法

海量媒体数据存储系统中,文件过大或多用户并发访问等原因会造成二进制大对象(BLOB)存储过程失效。为提高存储系统存取非结构二进制文件的可靠性和稳定性,提出并建立了BLOB存储可靠性概率模型,证明了使用数量较多的文件分块和较小的分块大小,可提高BLOB存储成功的概率。根据该模型,改进了关系数据库存储环境下的BLOB数据存取技术。通过建立基于文件分块的存储结构,提出了BLOB数据分块存储和重新装配新算法并进行了仿真实验。BLOB存取算法的仿真实验和实际应用表明,该存取算法有效降低了媒体数据存储失效的概率。     

Companson and Diagnosis of Large Replicated Files

This paper examines the problem of comparing large replicated files in a context in which communication dominates the cost of comparison. A low-cost checking matrix is proposed for comparison of these replicated files. The checking matrix is composed of check symbols generated by a divide-and-conquer encoding algorithm. The matrix allows for detection and diagnosis of disagreeing pages with very little communication overhead. In contrast to a previous O(N) proposal, the storage requirement for the checking matrix is O(log N), where N is the number of pages in the file. The matrix can be stored in main memory without the need for extra accesses to disk during normal updates of pages.     

针对Flash存储介质的数据恢复技术研究

针对Flash存储介质的数据恢复技术是信息安全领域中一个非常重要的研究课题。Flash存储介质有其独特的、不同于磁存储介质的存储特性,所以在数据存储方式上也与磁存储介质有所不同。针对Flash存储介质的存储特性而设计的闪存文件系统通过闪存转换层来实现数据的读写和芯片的管理。因此从文件系统级和芯片级两个方面对Flash存储介质的数据恢复技术进行了研究,并提出了下一步工作方向。     

多模态医疗数据中海量小文件存储优化方法

Hadoop分布式文件系统(HDFS)通常用于大文件的存储和管理,当进行海量小文件的存储和计算时,会消耗大量的NameNode内存和访问时间,成为制约HDFS性能的一个重要因素.针对多模态医疗数据中海量小文件问题,提出一种基于双层哈希编码和HBase的海量小文件存储优化方法.在小文件合并时,使用可扩展哈希函数构建索引文件存储桶,使索引文件可以根据需要进行动态扩展,实现文件追加功能.在每个存储桶中,使用MWHC哈希函数存储每个文件索引信息在索引文件中的位置,当访问文件时,无须读取所有文件的索引信息,只需读取相应存储桶中的索引信息即可,从而能够在O(1)的时间复杂度内读取文件,提高文件查找效率.为了满足多模态医疗数据的存储需求,使用HBase存储文件索引信息,并设置标识列用于标识不同模态的医疗数据,便于对不同模态数据的存储管理,并提高文件的读取速度.为了进一步优化存储性能,建立了基于LRU的元数据预取机制,并采用LZ4压缩算法对合并文件进行压缩存储.通过对比文件存取性能、NameNode内存使用率,实验结果表明,所提出的算法与原始HDFS、HAR、MapFile、TypeStorage以及...     

TridentFS: a hybrid file system for non‐volatile RAM,flash memory and magnetic disk

A hybrid file system with high flexibility and performance, called Trident file system (TridentFS), is proposed to manage three types of storage with different performance characteristics, that is, Non‐Volatile RAM (NVRAM), flash memory and magnetic disk. Unlike previous NVRAM‐based hybrid file systems, novel techniques are used in TridentFS to improve the flexibility and performance. TridentFS is flexible by the support of various forms of flash memory and a wide range of NVRAM size. The former is achieved on the basis of the concept of stackable file systems, and the latter is achieved by allowing data eviction from the NVRAM. TridentFS achieves high performance by keeping hot data in the NVRAM and allowing data evicted from the NVRAM to be parallel distributed to the flash memory and disk. A data eviction policy is proposed to determine the data to be evicted from the NVRAM. Moreover, a data distribution algorithm is proposed to effectively leverage the parallelism between flash memory and disk during data distribution. TridentFS is implemented as a loadable module on Linux 2.6.29. The performance results show that it works well for both small‐sized and large‐sized NVRAM, and the proposed eviction policy outperforms LRU by 27%. Moreover, by effectively leveraging the parallelism between flash memory and disk, the proposed data distribution algorithm outperforms the RAID‐0 and a size‐based distribution method by up to 471.6% and 82.6%, respectively. By considering the data size and performance characteristics of the storage, the proposed data distribution algorithm outperforms the greedy algorithm by up to 15.5%. Copyright © 2014 John Wiley & Sons, Ltd.     

国产化环境下的海量小文件数据分布式存储技术

为缓解单一存储设备存储海量小文件的压力,提出了一种国产化环境下的海量小文件数据分布式存储技术。利用聚类算法实现海量小文件合并。以达到最大均衡度为目标,在多项约束条件下利用人工鱼群算法求解分布式存储方案。按照分布式存储方案将海量小文件数据迁移到存储节点及其存储设备上,完成海量小文件数据分布式存储。结果表明：14个存储节点和28个存储设备的内存占用较为均衡,内存资源利用率较高。将小文件样本迁移并存储到节点的过程中,分布式存储均衡度整体波动均超过设定的阈值1.0,说明分布式存储均衡度较好,证明了所提存储技术的有效性。     

基于小文件的内存云存储优化策略

由于内存云RAMCloud采用日志段的方式存储数据,因此当大量小文件存储于RAMCloud集群时,每个小文件独占整个段,会产生较多的段内碎片,从而导致内存的有效利用率较低以及大量的内存空间浪费。为了解决这个问题,提出基于文件分类的RAMCloud小文件存储优化策略。该策略首先根据文件的相关特性将小文件分为结构相关文件、逻辑相关文件以及相互独立文件三类;然后在存储时对结构相关的文件使用文件合并算法,逻辑相关和相互独立的小文件则使用分组算法。实验结果表明:同未进行优化的RAMCloud存储策略相比,该策略能有效提高集群内存利用率。     

SBBS: A sliding blocking algorithm with backtracking sub-blocks for duplicate data detection

With the explosive growth of data, storage systems are facing huge storage pressure due to a mass of redundant data caused by the duplicate copies or regions of files. Data deduplication is a storage-optimization technique that reduces the data footprint by eliminating multiple copies of redundant data and storing only unique data. The basis of data deduplication is duplicate data detection techniques, which divide files into a number of parts, compare corresponding parts between files via hash techniques and find out redundant data. This paper proposes an efficient sliding blocking algorithm with backtracking sub-blocks called SBBS for duplicate data detection. SBBS improves the duplicate data detection precision of the traditional sliding blocking (SB) algorithm via backtracking the left/right 1/4 and 1/2 sub-blocks in matching-failed segments. Experimental results show that SBBS averagely improves the duplicate detection precision by 6.5% compared with the traditional SB algorithm and by 16.5% compared with content-defined chunking (CDC) algorithm, and it does not increase much extra storage overhead when SBBS divides the files into equal chunks of size 8 kB.     

基于闪存的混合存储系统缓冲区管理算法

由于基于闪存的混合存储系统充分利用了闪存的高速随机读和磁盘的快速顺序写的特性,近年来已经成为了数据库管理系统的二级存储层的高效存储模式,但其I/O访问开销是一个继续提高存储性能的瓶颈.为了降低混合存储系统的I/O访问开销,提出了一种自适应缓冲区管理算法DLSB.该算法根据数据页的逻辑代价和物理代价进行自适应的数据域选择;并在选择的数据域中,比较闪存队列和磁盘队列容量的实际值与理想值来确定数据页的置换,达到了提高I/O访问效率的目的.实验结果表明,该算法有效且可行,显著降低了混合存储系统的I/O访问开销.     

嵌入式Flash文件系统的设计与实现

Flash存储器是一种在嵌入式系统中日益普及的存储介质,它提供了高密度且成本相对较低的固态存储。使用Flash存储器需要很多技巧来确保数据可靠性并延长Flash器件的使用寿命。根据Flash存储器的特点提出了一种文件系统模型,可以合理且有效地利用Flash存储器,同时提供日志特性以增强使用该文件系统的嵌入式系统的可靠性。     

基于分数阶Fourier变换的云存储系统重复数据删除算法

云存储系统的重复数据作为大量冗余数据的一种,对其有效及时地删除能保证云存储系统的稳定与运行。由于云存储系统中的干扰数据较多,信噪比较低,传统的重删算法会在分数阶Fourier域出现伪峰峰值,不能有效地对重复数据进行检测滤波和删除处理,因此提出一种改进的基于分数阶Fourier变换累积量检测的云存储系统重复数据删除算法。首先分析云存储系统重复数据删除机制体系架构,定义数据存储点的适应度函数,得到云存储节点的系统子集随机概率分布;采用经验约束函数对存储节点中的校验数据块分存,通过分数阶Fourier变换对云存储系统中的幅度调制分量进行残差信号滤波预处理。采用4阶累积量切片后置算子,把每个文件分为若干个块,针对每个文件块进行重删,进行重复数据检测后置滤波处理,实现存储资源上的重复数据检测及其删除。仿真实验表明,该算法能提高集群云存储系统计算资源的利用率,重复数据准确删除率较高,有效避免了数据信息流的干扰特征造成的误删和漏删,性能优越。     

A high performance NAND array file system based on multiple NAND flash memories

The existing NAND flash memory file systems have not taken into account multiple NAND flash memories for large-capacity storage. In addition, since large-capacity NAND flash memory is much more expensive than the same capacity hard disk drive, it is cost wise infeasible to build large-capacity flash drives. To resolve these problems, this paper suggests a new file system called NAFS for large-capacity storage with multiple small-capacity and low-cost NAND flash memories. It adopts a new cache policy, mount scheme, and garbage collection scheme in order to improve read and write performance, to reduce the mount time, and to improve the wear-leveling effectiveness. Our performance results show that NAFS is more suitable for large-capacity storage than conventional NAND file systems such as YAFFS2 and JFFS2 and a disk-based file system for Linux such as HDD-RAID5-EXT3 in terms of the read and write transfer rate using a double cache policy and the mount time using metadata stored on a separate partition. We also demonstrate that the wear-leveling effectiveness of NAFS can be improved by our adaptive garbage collection scheme.     

Flash memory efficient LTL model checking

As the capacity and speed of flash memories in form of solid state disks grow, they are becoming a practical alternative for standard magnetic drives. Currently, most solid-state disks are based on NAND technology and much faster than magnetic disks in random reads, while in random writes they are generally not.So far, large-scale LTL model checking algorithms have been designed to employ external memory optimized for magnetic disks. We propose algorithms optimized for flash memory access. In contrast to approaches relying on the delayed detection of duplicate states, in this work, we design and exploit appropriate hash functions to re-invent immediate duplicate detection.For flash memory efficient on-the-fly LTL model checking, which aims at finding any counter-example to the specified LTL property, we study hash functions adapted to the two-level hierarchy of RAM and flash memory. For flash memory efficient off-line LTL model checking, which aims at generating a minimal counterexample and scans the entire state space at least once, we analyze the effect of outsourcing a memory-based perfect hash function from RAM to flash memory.Since the characteristics of flash memories are different to magnetic hard disks, the existing I/O complexity model is no longer sufficient. Therefore, we provide an extended model for the computation of the I/O complexity adapted to flash memories that has a better fit to the observed behavior of our algorithms.     

基于HDFS的小文件存储与读取优化策略

本文对HDFS分布式文件系统进行了深入的研究,在HDFS中以流式的方式访问大文件时效率很高但是对海量小文件的存取效率比较低. 本文针对这个问题提出了一个基于关系数据库的小文件合并策略,首先为每个用户建立一个用户文件,其次当用户上传小文件时把文件的元数据信息存入到关系数据库中并将文件追加写入到用户文件中,最后用户读取小文件时通过元数据信息直接以流式方式进行读取. 此外当用户读取小于一个文件块大小的文件时还采取了数据节点负载均衡策略,直接由存储数据的DataNode传送给客户端从而减轻主服务器压力提高文件传送效率. 实验结果表明通过此方案很好地解决了HDFS对大量小文件存取支持不足的缺点,提高了HDFS文件系统对海量小文件的读写性能,此方案适用于具有海量小文件的云存储系统,可以降低NameNode内存消耗提高文件读写效率.     

基于闪存的星载存储数据管理研究

由于闪存介质在航空航天数据存储中的优势,逐渐成为DRAM、SDRAM之后的下一代卫星存储解决方案。但由于闪存的存储特性,目前缺乏对数据存储的有效管理,星载闪存文件系统面临诸多有待研究的问题。通过分析星载环境下卫星存储特征,给出星载环境下的数据管理需求,并实验模拟发现在 IOPS（每秒进行读写的次数）、元数据、吞吐量、随机寻址四种性能指标上,F2FS优于BTRFS、exFAT、NILFS这三种类闪存文件系统;结合具体测试,分析了闪存内部并行对于文件系统的影响。论证得到F2FS较适合对基于闪存的星载存储进行数据管理,并结合数据管理需求提出了适用于星载环境下的几点改进方案,为星载环境下闪存文件系统设计和实现提供了研究基础。     

海量样本数据集中小文件的存取优化研究

针对Hadoop分布式文件系统（Hadoop Distributed File System,HDFS）在海量样本数据集存储方面存在内存占用多、读取效率低的问题,以及分布式数据库HBase在存储文件名重复度和类似度高时产生访问热点的问题,结合样本数据集的特点、类型,提出一种面向样本数据集存取优化方案,优化样本数据集中小文件的写入、读取、添加、删除和替换策略。该方案根据硬件配置测得大、小文件的分界点,通过变尺度堆栈算法按样本数据集的目录结构将小文件合并存储至HDFS;结合行键优化策略将文件索引存储在HBase数据表中;搭建基于Ehcache缓存框架的预取机制。实验结果表明,该方案降低了主节点的内存消耗,提高了文件的读取效率,实现了对海量样本数据集中小文件的高效存取。     

NAND Flash文件系统方案及其可靠性设计

NAND Flash具有高存储密度和高存储速率的特点,在嵌入式系统领域得到了广泛应用,如何使其文件系统的性能最优化成为本文讨论的主题。本文首先提出文件系统的设计方法,主要阐述面向嵌入式应用的FAT文件系统;接着,针对NAND Flash存在固有的坏块这一弱点,为了提高整个系统的可靠性,提出一种利用嵌入式文件系统进行坏块处理的策略,核心思想为利用Flash存储器上未使用空间来代替坏块,同时,在FAT表中标记出损坏的坏的信息,避免以后对坏块进行读写。本策略经项目的具体应用证明,具有较高可靠性。