DotDFS: A Grid-based high-throughput file transfer system

DotGrid platform is a Grid infrastructure integrated with a set of open and standard protocols recently implemented on the top of Microsoft .NET in Windows and MONO .NET in UNIX/Linux. DotGrid infrastructure along with its proposed protocols provides a right and solid approach to targeting other platforms, e.g., the native C/C++ runtime. In this paper, we propose a new concurrent file transfer protocol called DotDFS as a high-throughput distributed file transfer component for DotGrid. DotDFS introduces some open binary protocols for efficient file transfers on current Grid infrastructures. DotDFS protocol also provides mechanisms for multiple file streams to gain high-throughput file transfer similar to GridFTP protocol, but by proposing and implementing a new parallel TCP connection-oriented paradigm. Almost no research work has been conducted to suggest a concurrent file transfer protocol that simultaneously employs threaded and event-driven models in the protocol level. Due to our knowledge, DotDFS is the first concurrent file transfer protocol that, from this viewpoint, presents a new computing paradigm in the field of data transmission protocols. In our LAN tests, we have achieved better results than Globus GridFTP implementation particularly in multiple TCP streams and directory tree transfers. Our LAN experiences in memory-to-memory tests show that DotDFS accesses to the 94% bottleneck bandwidth while GridFTP is accessing 91%. In LAN disk-to-disk tests, comparing DotDFS protocol with GridFTP protocol unveils a set of interesting and technical problems in GridFTP for both the nature of the protocol and its implementation by Globus. In the WAN experimental studies, we propose a new idea for analytical modeling of file transfer protocols like DotDFS inspired by sampling, experimentation and mathematical interpolation approaches. The cross-platform and open standard-based features of DotDFS provide a substantial framework for unifying data access and resource sharing in real heterogeneous Grid environments.     

基于GridFTP的并行数据传输性能分析与研究

分析论述了广域网格网络中数据传输的需求以及网格数据传输协议—GridFTP的性能特点,对GridFTP中最重要的并行传输机制进行了探讨。通过大量实验测试对不同并行度下数据传输的时间、带宽、吞吐量、数据传输总量等性能参数进行了比较,讨论了并行度对传输性能的影响,提出了并行传输机制在提高传输性能方面的局限和值得注意的问题。     

数据网格的动态复制选址策略及应用研究

动态复制技术对于提高数据网格的性能是非常重要的。对目前的动态复制策略进行了综述,鉴于目前效果较好的动态复制策略均为单选址算法,对于延迟较大、分布较广的网格存在很大局限性。提出了三种多选址的动态复制策略,并将它们转化为经典的数学问题进行求解。在给出了多选址动态复制策略在远程教育资源管理中的应用后,在欧洲数据网格试验床1拓扑上进行了仿真实验,实验结果表明:与目前效果较好的选址策略相比,所提出的多选址策略对于减少网络负载和网络延迟效果显著。     

分布式并行文件系统中锁管理的研究

分析了传统的分布式锁管理中范围锁的实现及其局限,给出了一种用于支持交叉访问模式的新的分布式锁管理算法--DBM算法.分布式锁管理是分布式并行文件系统的关键组成部分之一.软件仿真实验表明新算法在交叉访问模式下大大提高了系统并发度.     

基于Agent联盟机制的网格资源协同分配

在网格计算环境,许多应用的执行经常需要同时协同分配多个资源以满足性能需求。文中提出了一种新的基于Agent联盟机制的资源协同分配方法。首先,对Globus和Legion的资源协同分配的方法进行了比较。然后,描述了一个基于Agent的网格资源管理结构。最后,设计了一个Agent联盟机制用于有效的资源协同分配。     

On‐demand data co‐allocation with user‐level cache for grids

Conventional remote data access middlewares usually provide client applications with either a pre‐staging scheme or an on‐demand access scheme to fetch data. The pre‐staging scheme uses parallel downloads to fetch a completed input file from multiple data sources, even when only a tiny file fragment is required. Such a transfer scheme consumes unnecessary data transmission time and storage space. In contrast, the on‐demand scheme downloads only the required data blocks from a single data source and does not fully utilize the downstream bandwidth of the computing nodes. This paper presents a middleware called ‘Spigot’ that facilitates legacy (grid‐unaware) applications to transparently access remote data by using native I/O function calls. Spigot uses the on‐demand concept to avoid unnecessary data transfer and adopts a co‐allocation download algorithm to improve the data transfer performance. Moreover, it uses the pre‐fetching strategy to reduce the data waiting time by overlapping data acquisition and data processing. It also provides the client application with its own user‐level cache, which is advantageous since a larger cache space is available in comparison with the kernel‐level cache. Further, it is easy to maintain data consistency between Spigot nodes. The experimental results indicate that Spigot achieves superior performance in reducing the data waiting time than the pre‐staging and the on‐demand access schemes. Copyright © 2010 John Wiley & Sons, Ltd.     

基于数据流的启发式文件传输识别系统设计

为了防止文件通过网络泄露,研究了网络传输文件的格式、传送形式以及通信行为特征,提出一种基于网络数据流的文件传输识别方法。通过引入文件数据流的概念,构建了具有特异性的文件特征指纹库,采用自适应深度协议分析技术,对文件传输在通信协议中表现出的内容和行为特征信息进行多层次启发式的检测。测试结果表明,该系统运行高效,检测结果准确。     

DES加密算法在保护文件传输中数据安全的应用

随着Intemet的广泛应用,网络中文件传输的数据安全性越来越受到人们的重视。原始的DES加密算法一次加解密64位,本文介绍了如何使用C语言实现改进的DES数据加密算法,使其可以方便的加解密任意格式任意大小的文件,确保文件传输中的数据安全。最后针对DES加密算法存在的安全隐患给予了简单说明,并提出了DES加密算法的使用误区。     

iPACS: Power-aware covering sets for energy proportionality and performance in data parallel computing clusters

Energy consumption in datacenters has recently become a major concern due to the rising operational costs and scalability issues. Recent solutions to this problem propose the principle of energy proportionality, i.e., the amount of energy consumed by the server nodes must be proportional to the amount of work performed. For data parallelism and fault tolerance purposes, most common file systems used in MapReduce-type clusters maintain a set of replicas for each data block. A covering subset is a group of nodes that together contain at least one replica of the data blocks needed for performing computing tasks. In this work, we develop and analyze algorithms to maintain energy proportionality by discovering a covering subset that minimizes energy consumption while placing the remaining nodes in low-power standby mode in a data parallel computing cluster. Our algorithms can also discover covering subset in heterogeneous   computing environments. In order to allow more data parallelism, we generalize our algorithms so that it can discover k$k$-covering subset, i.e., a set of nodes that contain at least k$k$ replicas of the data blocks. Our experimental results show that we can achieve substantial energy saving without significant performance loss in diverse cluster configurations and working environments.     

A Bee Colony based optimization approach for simultaneous job scheduling and data replication in grid environments

This paper presents a novel Bee Colony based optimization algorithm, named Job Data Scheduling using Bee Colony (JDS-BC). JDS-BC consists of two collaborating mechanisms to efficiently schedule jobs onto computational nodes and replicate datafiles on storage nodes in a system so that the two independent, and in many cases conflicting, objectives (i.e., makespan and total datafile transfer time) of such heterogeneous systems are concurrently minimized. Three benchmarks – varying from small- to large-sized instances – are used to test the performance of JDS-BC. Results are compared against other algorithms to show JDS-BC's superiority under different operating scenarios. These results also provide invaluable insights into data-centric job scheduling for grid environments.     

An enhanced 3D data transfer method for fluid–structure interface by ISOMAP nonlinear space dimension reduction

With the accuracy limitation of some transfer methods in the self-parallel fluid–structure interfaces, a data transfer method is proposed by an ISOMAP (Isometric Mapping) nonlinear dimensionality reduction. Through this new method, the data transfer problem of the self-parallel interfaces is solved. Example of a 3D turbine blade shows that the proposed method can improve the transfer accuracy in the non-matching meshes.