共查询到20条相似文献,搜索用时 15 毫秒
1.
Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. In a previous paper we showed that prefetching and caching have thepotential to deliver the performance benefits of parallel file systems to parallel applications. In this paper we describe experiments withpractical prefetching policies that base decisions only on on-line reference history, and that can be implemented efficiently. We also test the ability of those policies across a range of architectural parameters. 相似文献
2.
Data distribution in memory or on disks is an important factor influencing the performance of parallel applications. On the
other hand, programs or systems, like a parallel file system, frequently redistribute data between memory and disks.
This paper presents a generalization of previous approaches of the redistribution problem. We introduce algorithms for mapping
between two arbitrary distributions of a data set. The algorithms are optimized for multidimensional array partitions. We
motivate our approach and present potential utilizations. The paper also presents a case study, the employment of mapping
functions, and redistribution algorithms in a parallel file system.
相似文献
Walter F. TichyEmail: |
3.
One problem with data-intensive computing facilitating is how to effectively manage massive amounts of data stored in a parallel I/O system. The file assignment method plays a significant role in data management. However, in the context of a parallel I/O system, most existing file assignment approaches share the following two limitations. First, most existing methods are designed for a non-partitioned file, while the file in a parallel I/O system is generally partitioned to provide aggregated bandwidth. Second, the file allocation metric, e.g. service time, of most existing methods is difficult to determine in practice, and also these metrics only reflect the static property of the file. In this paper, a new metric, namely file access density is proposed to capture the dynamic property of file access, i.e. disk contention property. Based on file access density definition, this paper introduces a new static file assignment algorithm named MinCPP and its dynamic version DMinCPP, both of which aim at minimizing the disk contention property. Furthermore MinCPP and DMinCPP take the file partition property into consideration by trying to allocate the partitions belonging to the same file onto different disks. By assuming file request arrival follows the Poisson process, we prove the effectiveness of the proposed schemes both analytically and experimentally. The MinCPP presented in this study can be applied to reorganize the files stored in a large-scale parallel I/O system and the DMinCPP can be integrated into file systems which dynamically allocate files in a batch. 相似文献
4.
Nowadays, the rapid development of the internet calls for a high performance file system, and a lot of efforts have already been devoted to the issue of assigning nonpartitioned files in a parallel file system with the aim of pursuing a prompt response to requests. Yet most of the existing strategies still fail to bring about an optimal performance on system mean response time metrics, and new strategies which can achieve better performance in terms of mean response time become indispensable for parallel file systems. This paper, while addressing the issue of assigning nonpartitioned files in parallel file systems where the file accesses exhibit Poisson arrival rates and fixed service times, presents an on-line file assignment strategy, named prediction-based dynamic file assignment (PDFA), to minimize the mean response time among disks under different workload conditions, and a comparison of the PDFA with the well-known file assignment algorithms, such as HP and SOR. Comprehensive experimental results show that PDFA is able to improve the performance consistently in terms of mean response time among all algorithms for comparison. 相似文献
5.
基于集群的MPI主从式并行文件传输系统的研究与实现 总被引:1,自引:0,他引:1
研究并实现了一种基于PC集群系统的MPI主从式并行文件传输系统。通过引入一个监控节点实时监控当前各真实节点上的连接数、请求任务量以及各节点的实际处理能力,并衡量不同内容节点的负载,以实现集群内各节点负载均衡,从而更合理地实现并行文件I/O的任务分配。结果显示,该系统具有近似线性加速比,即文件I/O传输速度随着并行节点数量的增多而提高。 相似文献
6.
DanFeng HongJiang Yi-FengZhu 《计算机科学技术学报》2004,19(6):0-0
Without any additional cost, all the disks on the nodes of a cluster can be connected together through CEFT-PVFS, an RAID-10 style parallel file system, to provide a multi-GB/s parallel I/O performance.I/O response time is one of the most important measures of quality of service for a client. When multiple clients submit data-intensive jobs at the same time, the response time experienced by the user is an indicator of the power of the cluster. In this paper, a queuing model is used to analyze in detail the average response time when multiple clients access CEFT-PVFS. The results reveal that response time is with a function of several operational parameters. The results show that I/O response time decreases with the increases in I/O buffer hit rate for read requests, write buffer size for write requests and the number of server nodes in the parallel file system, while the higher the I/O requests arrival rate, the longer the I/O response time. On the other hand, the collective power of a large cluster supported by CEFT-PVFS is shown to be able to sustain a steady and stable I/O response time for a relatively large range of the request arrival rate. 相似文献
7.
Xubin Li Christian Xin Stephen L. 《Journal of Parallel and Distributed Computing》2009,69(12):961-973
High availability data storage systems are critical for many applications as research and business become more data driven. Since metadata management is essential to system availability, multiple metadata services are used to improve the availability of distributed storage systems. Past research has focused on the active/standby model, where each active service has at least one redundant idle backup. However, interruption of service and even some loss of service state may occur during a fail-over depending on the replication technique used. In addition, the replication overhead for multiple metadata services can be very high. The research in this paper targets the symmetric active/active replication model, which uses multiple redundant service nodes running in virtual synchrony. In this model, service node failures do not cause a fail-over to a backup and there is no disruption of service or loss of service state. A fast delivery protocol is further discussed to reduce the latency of the total order broadcast needed. The prototype implementation shows that metadata service high availability can be achieved with an acceptable performance trade-off using the symmetric active/active metadata service solution. 相似文献
8.
Data I/O has become a major bottleneck of computational performance of geospatial analysis and modeling. In this study, a parallel GeoTIFF I/O library (pGTIOL) was developed. Through the storage mapping and data arrangement techniques, pGTIOL can operate on files in either strip or tile storage mode, read/write any sub-domain of data within the raster dataset. pGTIOL enables asynchronized I/O, which means a process can read/write its own sub-domains of data when necessary without synchronizing with other processes. pGTIOL was integrated into the parallel raster processing library (pRPL). Several pGTIOL-based data I/O functions and options were added to pRPL, while the existing functions of pRPL stay intact. Experiments showed that the integration of pRPL and pGTIOL achieved higher performance than the original pRPL that uses GDAL as the I/O interface. Therefore, pRPL + pGTIOL enables transparent parallelism for high-performance raster processing with the capability of true parallel I/O of massive raster datasets. 相似文献
9.
为了观察和分析进化算法的执行情况,往往需要将算法执行过程中产生的大量数据存储在磁盘文件中。用于进化算法的嵌入式数据存储引擎EADB(Evolutionary Algorithm Database)提供了简便灵活的数据存储接口,通过使用内存映射文件技术来实现数据的快速和大量存储。相较于传统文件I/O存储方式和一般的通用数据存储引擎,EADB大大加快了存储速度。 相似文献
10.
In this paper, we study I/O server placement for optimizing parallel I/O performance on switch-based clusters, which typically
adopt irregular network topologies to allow construction of scalable systems with incremental expansion capability. Finding
optimal solution to this problem is computationally intractable. We quantified the number of messages travelling through each
network link by a workload function, and developed three heuristic algorithms to find good solutions based on the values of the workload function. The maximum-workload-based
heuristic chooses the locations for I/O nodes in order to minimize the maximum value of the workload function. The distance-based
heuristic aims to minimize the average distance between the compute nodes and I/O nodes, which is equivalent to minimizing
average workload on the network links. The load-balance-based heuristic balances the workload on the links based on a recursive
traversal of the routing tree for the network.
Our simulation results demonstrate performance advantage of our algorithms over a number of algorithms commonly used in existing
parallel systems. In particular, the load-balance-based algorithm is superior to the other algorithms in most cases, with
improvement ratio of 10 to 95% in terms of parallel I/O throughput. 相似文献
11.
并行文件系统Lustre粗粒度I/O性能良好,细粒度I/O性能相对粗粒度I/O比较低下,因此优化细粒度I/O性能成为提高系统整体I/O性能的关键问题。在研究和分析了Lustre的I/O访问模式、细粒度I/O服务流程和页面替换算法等方面后,提出了细粒度优先(Fine Grained First,FGF)LRU算法。在OST端及Client端的页高速缓存中最大程度地保留细粒度I/O的页面,降低细粒度I/O引起的页面下沉速度,延长细粒度I/O页面在主存中的时间,进而减少对磁盘的访问次数,降低磁盘访问开销。通过对实验数据的对比和分析,验证了FGF-LRU算法的有效性。在不影响粗粒度I/O性能的情况下,提高了细粒度I/O性能,最终实现提高系统整体I/O性能。 相似文献
12.
《Parallel Computing》2014,40(10):697-709
In order to run tasks in a parallel and load-balanced fashion, existing scientific parallel applications such as mpiBLAST introduce a data-initializing stage to move database fragments from shared storage to local cluster nodes. Unfortunately, with the exponentially increasing size of sequence databases in today’s big data era, such an approach is inefficient.In this paper, we develop a scalable data access framework to solve the data movement problem for scientific applications that are dominated by “read” operation for data analysis. SDAFT employs a distributed file system (DFS) to provide scalable data access for parallel sequence searches. SDAFT consists of two interlocked components: (1) a data centric load-balanced scheduler (DC-scheduler) to enforce data-process locality and (2) a translation layer to translate conventional parallel I/O operations into HDFS I/O. By experimenting our SDAFT prototype system with real-world database and queries at a wide variety of computing platforms, we found that SDAFT can reduce I/O cost by a factor of 4–10 and double the overall execution performance as compared with existing schemes. 相似文献
13.
14.
针对传统MPI集群并行I/O效率不高的问题,通过分析Lustre并行文件系统的特点和MPI-IO集中式I/O操作的算法,提出了一种基于MPI-IO编程接口的改进方案,用以改善集群I/O瓶颈,提高I/O并行效率,并通过实验验证了该方案的可行性。 相似文献
15.
16.
高性能计算系统需要一个可靠高效的并行文件系统.Lustre集群文件系统是典型的基于对象存储的集群文件系统,它适合大数据量聚合I/O操作.大文件I/O操作能够达到很高的带宽,但是小文件I/O性能低下.针对导致Lustre的设计中不利于小文件I/O操作的两个方面,提出了Filter Cache方法.在Lustre的OST组件中设计一个存放小文件I/O数据的Cache,让OST端的小文件I/O操作异步进行,以此来减少用户感知的小文件I/O操作完成的时间,提高小文件I/O操作的性能. 相似文献
17.
David E. Singh Florin Isaila Juan C. Pichel Jesús Carretero 《The Journal of supercomputing》2009,47(1):53-75
In this paper, we present a novel multiple phase I/O collective technique for generic block-cyclic distributions. The I/O
technique is divided into two stages: inspector and executor. During the inspector stage, the communication pattern is computed
and the required datatypes are automatically generated. This information is used during the executor stage in performing the
communication and file accesses. The two stages are decoupled, so that for repetitive file access patterns, the computations
from the inspector stage can be performed once and reused several times by the executor. This strategy allows to amortize
the inspector cost over several I/O operations. In this paper, we evaluate the performance of multiple phase I/O collective
technique and we compare it with other state of the art approaches. Experimental results show that for small access granularities,
our method outperforms in the large majority of cases other parallel I/O optimizations techniques.
相似文献
Jesús CarreteroEmail: |
18.
NVM存储设备系统具备提供高吞吐的潜质,包括接近内存的读写速度、字节寻址特性和支持多路转发等优势。但现有的系统软件栈并没有针对NVM去设计,使得系统软件栈存在许多影响系统访问性能的因素。通过分析发现文件系统的锁机制具有较大的开销,这使得数据的并发访问在多核心环境下成为一个难题。为了缓解这些问题,设计了无锁的文件读写机制以及基于字节的读写接口。通过取消基于文件的锁机制改变了粗粒度的访问控制,利用自主管理请求提高了进程的并发度;在设计能够利用字节寻址的新的文件访问接口时,不仅考虑了NVM存储设备的读写非对称,还考虑了其读写操作的不同特性。这些设计减少了软件栈的开销,有利于发挥NVM特性来提供一个高并发、高吞吐和耐久的存储系统。最后利用开源NVM模拟器PMEM实现了FPMRW原型系统,使用Filebench通用测试工具对FPMRW进行测试与分析,结果显示,FPMRW相对EXT+PMEM和XFS+PMEM能提高3%~40%的系统吞吐率。 相似文献
19.
Sudip K. Seal Kalyan S. Perumalla Steven P. Hirshman 《Journal of Parallel and Distributed Computing》2013
Direct solvers based on prefix computation and cyclic reduction algorithms exploit the special structure of tridiagonal systems of equations to deliver better parallel performance compared to those designed for more general systems of equations. This performance advantage is even more pronounced for block tridiagonal systems. In this paper, we re-examine the performances of these two algorithms taking the effects of block size into account. Depending on the block size, the parameter space spanned by the number of block rows, size of the blocks and the processor count is shown to favor one or the other of the two algorithms. A critical block size that separates these two regions is shown to emerge and its dependence both on problem dependent parameters and on machine-specific constants is established. Empirical verification of these analytical findings is carried out on up to 2048 cores of a Cray XT4 system. 相似文献
20.
本文介绍了并行打印机端口在主机与非打印机外设之间进行数据传输中的应用。给出了接口电路设计的方框图,和数据输入输出的子过程,并指出了进一步的工作方向。 相似文献