某存储系统高效FAT32文件系统实现技术研究

在某存储系统中,设计了一种高效的FAT32文件系统,其系统的实现采用了层次化设计思想,实现了物理实现层、缓冲层和文件实现层的数据交换,设计了服务函数接口,编写了服务函数实现流程图.该设计对硬盘的缓冲区采取较为先进的访问策略,这种策略非常有效,大大提高了文件系统的读写效率.最后对缓冲层和文件层进行了测试,测试结果证明访问效率有所提高,创建、查找、写文件和读文件功能正常,满足设计要求,为系统的后续研究打下了基础.     

分级存储系统中一种数据自动迁移方法

分级存储系统通过将数据在不同性能设备间动态迁移以达到高性能.已有分级存储系统未能充分利用负载信息导致数据迁移严重影响应用性能.提出了一种分级存储系统中的数据自动迁移方法AutoMig,目标是提高前台应用的I?O性能.AutoMig综合文件访问历史、文件大小、设备利用情况等参数,对文件进行动态分级,并使用LRU队列维护快速存储设备中的文件状态;挖掘关联文件用于自动预取;针对不同文件迁移操作采取不同的速率控制策略.对降级操作,根据负载变化动态调整迁移速率,对回迁操作则采取尽力而为的策略.在分级存储系统中的应用表明,与已有方法相比,AutoMig有效缩短了前台I?O响应时间.     

某存储系统高效FAT32文件系统实现技术研究

在某存储系统中,设计了一种高效的FAT32文件系统,其系统的实现采用了层次化设计思想,实现了物理实现层、缓冲层和文件实现层的数据交换,设计了服务函数接口,编写了服务函数实现流程图。该设计对硬盘的缓冲区采取较为先进的访问策略,这种策略非常有效,大大提高了文件系统的读写效率。最后对缓冲层和文件层进行了测试,测试结果证明访问效率有所提高,创建、查找、写文件和读文件功能正常,满足设计要求,为系统的后续研究打下了基础。     

分布式文件存储系统性能测试问题研究

分布式文件存储系统的性能测试不同于普通的文件系统性能测试。首先,需要根据系统支持的操作确定测试的benchmark,然后,需要根据系统的设计规模对多用户并发访问系统的情形进行模拟。在“燕星”分布式文件存储系统的性能测试中,对Andrewbenchmark进行改造,确定了测试的benchmark;基于RMIcallback机制,模拟了500个用户并发访问系统的情形。并将测试结果与NFS进行了对比,结果表明,“燕星”系统具有良好的实用性能。     

一种细粒度高效多版本文件系统

提出了基于快照的细粒度版本技术,能够克服已有多版本文件系统无法仅对系统局部目录或文件保留版本的缺点,增加了系统的灵活性;提出了版本空间的反向继承寻径,使用名字与版本独立的检索方案,可以充分利用版本间的相关性,优化数据物理布局,建立版本间的层级结构,既便于管理,又提高系统性能;设计了分别针对目录版本和文件版本的快速索引结构.评测结果表明,THVFS 的历史数据访问性能较著名的多版本文件系统ext3cow 提高了34.4%;Trace 实验中,相对于ext3,THVFS 的读性能提高了12%,同时,在每72 分钟生成一次快照的高频率下,维护所有历史版本仅需要80%的额外空间.     

云存储日志文件系统中快照的设计与实现

在逐步分析云存储日志文件系统中快照机制的基础上,设计并实现了基于HLFS(Hadoop DFS log-structured File-system)云存储日志文件系统的快照系统。通过在HLFS中添加新的快照,从快照系统中就可以随时获取该快照的信息,并恢复该快照时的所有数据,使云存储系统能够及时地进行数据恢复,从而避免云存储中核心数据的丢失,保证云存储中数据的安全性。为尽可能提高云存储日志文件系统中快照系统的可扩展性与实用性,详细分析快照在云存储日志文件系统中的设计与实现,并结合其他快照,分析了HLFS快照的实现性能。     

集群文件系统的性能最优化配置

集群文件系统是当前存储系统的研究热点.在资源一定的条件下,存储系统中元数据服务器和数据存储服务器节点及客户端节点数量之间的配置比例会对系统性能产生较大的影响.分析了lustre集群文件系统的参数配置,针对两种典型应用环境:文件服务和Web服务,测试了不同节点数和不同lustre条块大小配置下的系统性能;通过对比分析,得出lustre集群文件系统最优化配置,为提高集群文件系统性能提供参考.测试结果显示.当lustre文件系统的OST节点数与client节点数相当时系统性能最好.     

基于机器学习的动态分区并行文件系统性能优化

近年来,随着大数据、云计算技术的发展,应用系统越来越集中,规模亦越来越大,使得存 储系统的性能问题越来越突出。为应对其性能要求,并行文件系统得到了大量的应用。然而现有的并 行文件系统优化方法,大多只考虑应用系统或并行文件系统本身,较少考虑两者之间的协同。该文基 于应用系统在并行文件系统上的访问模式对存储系统的性能有显著影响这一特点,提出基于动态分区 的并行文件系统优化方法。首先,利用机器学习技术来分析挖掘各个性能影响因素和性能指标之间的 关系和规律,生成优化模型。其次,以优化模型为基础,辅助并行文件系统的参数调优工作。最后, 基于 Ceph 存储系统进行原型实现,并设计了三层架构应用系统进行了性能测试,最终达到优化并行 文件系统访问性能的目的。实验结果表明,所提出方法可以达到 85% 的预测优化准确率;在所提出模 型的辅助优化下,并行文件系统的吞吐量性能得到约 3.6 倍的提升。     

基于数据挖掘的文件元数据预取探究

在文件存储系统中,文件系统整体性能的提升对于保证文件的安全性和可靠性具有重要意义,而在此过程中,元数据访问性能与文件系统性能有密切关系,要想进一步满足大规模文件存储系统需要,就必须建立相应的文件元数据预取模型。本文通过对基于数据挖掘的文件元数据预取进行分析,以期满足文件数据的大量存取访问需求。     

基于海量数据存储系统多级存储介质的热点数据区分方法

海量数据的应用导致文件读写压力变大,必然需要考虑文件I/O带来的系统性能瓶颈.考虑到不同存储设备性能和成本不同,而且数据访问具有时间和空间局部性,因此需要进行分级存储.考虑到数据存在着周期性的变化规律,数据访问的热度是变化的,海量存储系统中相当大比例的数据静止不动,而且高性能存储设备有限,因此基于分级存储技术进行数据迁移,并且对常规价值评定模型进行改进后提出它所适用的基于文件级的数据迁移算法.针对并发访问读写压力变大之后对数据进行迁移的场景,用Web并发访问测试工具来对数据迁移之后的硬件读写进行评估,经过对Web服务器并发访问响应时间的测试验证,与常规迁移方法相比该算法的确改进了系统的性能.     

File-access characteristics of parallel scientific workloads

Phenomenal improvements in the computational performance of multiprocessors have not been matched by comparable gains in I/O system performance. This imbalance has resulted in I/O becoming a significant bottleneck for many scientific applications. One key to overcoming this bottleneck is improving the performance of multiprocessor file systems. The design of a high-performance multiprocessor file system requires a comprehensive understanding of the expected workload. Unfortunately, until recently, no general workload studies of multiprocessor file systems have been conducted. The goal of the CHARISMA project was to remedy this problem by characterizing the behavior of several production workloads, on different machines, at the level of individual reads and writes. The first set of results from the CHARISMA project describe the workloads observed on an Intel iPSC/860 and a Thinking Machines CM-5. This paper is intended to compare and contrast these two workloads for an understanding of their essential similarities and differences, isolating common trends and platform-dependent variances. Using this comparison, we are able to gain more insight into the general principles that should guide multiprocessor file-system design     

A measurement-based model to predict the performance impact ofsystem modifications: a case study

The paper presents a performance case study of parallel jobs executing in real multi user workloads. The study is based on a measurement based model capable of predicting the completion time distribution of the jobs executing under real workloads. The model constructed is also capable of predicting the effects of system design changes on application performance. The model is a finite state, discrete time Markov model with rewards and costs associated with each state. The Markov states are defined from real measurements and represent system/workload states in which the machine has operated. The paper places special emphasis on choosing the correct number of states to represent the workload measured. Specifically, the performance of computationally bound, parallel applications executing in real workloads on an Alliant FX/80 is evaluated. The constructed model is used to evaluate scheduling policies, the performance effects of multiprogramming overhead, and the scalability of the Alliant FX/8O in real workloads. The model identifies a number of available scheduling policies which would improve the response time of parallel jobs. In addition, the model predicts that doubling the number of processors in the current configuration would only improve response time for a typical parallel application by 25%. The model recommends a different processor configuration to more fully utilize extra processors. The paper also presents empirical results which validate the model created     

Evaluating the Performance of Cache-Affinity Scheduling in Shared-Memory Multiprocessors

As a process executes on a processor, it builds up state in that processor′s cache. In multiprogrammed workloads, the opportunity to reuse this state may be lost when a process gets rescheduled, either because intervening processes destroy its cache state or because the process may migrate to another processor. In this paper, we explore affinity scheduling, a technique that helps reduce cache misses by preferentially scheduling a process on a processor where it has run recently. Our study focuses on a bus-based multiprocessor executing a variety of workloads, including mixes of scientific, software development, and database applications. In addition to quantifying the performance benefits of exploiting affinity, our study is distinctive in that it provides low-level data from a hardware performance monitor that details why the workloads perform as they do. Overall, for the workloads studied, we show that affinity scheduling reduces the number of cache misses by 7-36%, resulting in execution time improvements of up to 10%. Although the overall improvements are small, modifying the operating system scheduler to exploit affinity appears worthwhile-affinity has no negative impact on the workloads and we show that it is extremely simple to add to existing schedulers.     

Pmfs中目录项索引的实现

可字节寻址的非易失存储介质,如相变存储器等,使数据可以在内存级别持久化。由于非易失存储器（NVM）本身的读写延时非常低,系统软件开销成为了决定整个持久化内存系统性能的主要因素。Pmfs是一个专门为持久化内存所设计的文件系统,然而,Pmfs下的每个目录操作（打开、创建或删除）都会遍历目录下的所有目录项,导致了随文件数增长而线性增长的目录项查找开销。通过测试发现,在特定类型负载下这种开销成为了整个文件系统的瓶颈。针对该问题,在Pmfs中实现了持久化的目录项索引来加速目录操作。测试结果显示,基于单目录下100000文件的负载,该优化使得文件创建速度提高了12倍,带宽增加了27.3%。     

Markov-chain based reliability analysis for distributed systems

In a typical distributed computing system (DCS), nodes consist of processing elements, memory units, shared resources, data files, and programs. For a distributed application, programs and data files are distributed among many processing elements that may exchange data and control information via communication link. The reliability of DCS can be expressed by the analysis of distributed program reliability (DPR) and distributed system reliability (DSR). In this paper, two reliability measures are introduced which are Markov-chain distributed program reliability (MDPR) and Markov-chain distributed system reliability (MDSR) to accurately model the reliability of DCS. A discrete time Markov chain with one absorbing state is constructed for this problem. The transition probability matrix is employed to represent the transition probability from one state to another state in a unit of time. In addition to mathematical method to evaluate the MDPR and MDSR, a simulation result is also presented to prove its correction.     

Procedures for managing extendible array files

Varieties of files are reviewed and the notion of a file space is discussed. A number of properties which characterize file spaces are listed. This modest theoretical framework has a clarifying influence on the subject of files. Certain file-system primitives are introduced which are similar to those in most commonly-known operating systems, and then these primitives are used in constructing a collection of routines for reading and writing extedible arrays. Extendible arrays are based on fixed page size randomly-accessible files and they are a useful universal input/output (I/O) structure for many different storage and retrieval schemes. The routines for managing such files are similar to the mechanisms used in handling a paged address space in a virtual memory system. The contribution of this paper lies in the detailed implementation which is presented; these routines are well understood in principle, but tricky to construct in practice. Finally, these routines are used in a practical example which provides for managing a variable blocksize free space list within a extendible array.     

HSM2: A New Heuristic State Minimization Algorithm for Finite State Machine

This paper proposes a heuristic state minimization algorithm (HSM2) for finite state machines (FSM). HSM2 focuses on the generation and adjustment of the closed cover. First an initial closed cover is generated by heuristically selecting proper maximal compatibles to satisfy all the covering and closure conditions, and then it is adjusted to be a minimal or near minimal closed cover by heuristically removing repeated states. Experimental results show that the algorithm is faster and obtains better or the same solutions compared with conventional methods.     

Characterizing files in the modern Gnutella network

The Internet has witnessed an explosive increase in the popularity of Peer-to-Peer (P2P) file-sharing applications during the past few years. As these applications become more popular, it becomes increasingly important to characterize their behavior in order to improve their performance and quantify their impact on the network. In this paper, we present a measurement study on characteristics of available files in the modern Gnutella system. We develop two new methodologies to capture accurate snapshots of available files in a large-scale P2P system. These methodologies were implemented in a parallel crawler that captures the entire overlay topology of the system where each peer in the overlay is annotated with its available files. We have captured more than 50 snapshots of the Gnutella system that span over 1 year period. Using these snapshots, we conduct three types of analysis on available files: (1) Static analysis, (2) Topological analysis, and (3) dynamic analysis. Our results reveal several interesting properties of available files in Gnutella that can be leveraged to improve the design and evaluation of P2P file-sharing applications. This paper extends and supplants the earlier version of this paper presented at MMCN 2006 [1]. This material is based upon work supported by the National Science Foundation (NSF) under Grant No. Nets-NBD-0627202, CAREER Award CNS-0448639, and an unrestricted gift from Cisco Systems. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or Cisco.     

面向流式数据处理系统的高效故障恢复方法

针对流式数据处理系统Flink无法高效处理单点故障的问题,提出了一种基于增量状态和备份的故障容错系统Flink+。首先,提前建立备份算子和数据通路;然后,对数据流图中的输出数据进行缓存,必要时使用磁盘;其次,在系统快照时进行任务状态同步;最后,在系统故障时使用备份任务和缓存的数据恢复计算。在系统实验测试中,Flink+在无故障运行时没有显著增加额外容错开销;而在单机和分布式环境下处理单点故障时,与Flink系统相比,所提系统在单机8任务并行度下故障恢复时间减少了96.98%,在分布式16任务并行度下故障恢复时间减少了88.75%。实验结果表明,增量状态和备份方法一起使用可以有效减少流式系统单点故障的恢复时间,增强系统的鲁棒性。