A survey of dynamic replication strategies for improving data availability in data grids

Data grid is a distributed collection of storage and computational resources that are not bounded within a geophysical location. It is a fast growing area of research and providing efficient data access and maximum data availability is a challenging task. To achieve this task, data is replicated to different sites. A number of data replication techniques have been presented for data grids. All replication techniques address some attributes like fault tolerance, scalability, improved bandwidth consumption, performance, storage consumption, data access time etc. In this paper, different issues involved in data replication are identified and different replication techniques are studied to find out which attributes are addressed in a given technique and which are ignored. A tabular representation of all those parameters is presented to facilitate the future comparison of dynamic replication techniques. The paper also includes some discussion about future work in this direction by identifying some open research problems.     

PDDRA: A new pre-fetching based dynamic data replication algorithm in data grids

In recent years, grid technology has had such a fast growth that it has been used in many scientific experiments and research centers. A large number of storage elements and computational resources are combined to generate a grid which gives us shared access to extra computing power. In particular, data grid deals with data intensive applications and provides intensive resources across widely distributed communities. Data replication is an efficient way for distributing replicas among the data grids, making it possible to access similar data in different locations of the data grid. Replication reduces data access time and improves the performance of the system. In this paper, we propose a new dynamic data replication algorithm named PDDRA that optimizes the traditional algorithms. Our proposed algorithm is based on an assumption: members in a VO (Virtual Organization) have similar interests in files. Based on this assumption and also file access history, PDDRA predicts future needs of grid sites and pre-fetches a sequence of files to the requester grid site, so the next time that this site needs a file, it will be locally available. This will considerably reduce access latency, response time and bandwidth consumption. PDDRA consists of three phases: storing file access patterns, requesting a file and performing replication and pre-fetching and replacement. The algorithm was tested using a grid simulator, OptorSim developed by European Data Grid projects. The simulation results show that our proposed algorithm has better performance in comparison with other algorithms in terms of job execution time, effective network usage, total number of replications, hit ratio and percentage of storage filled.     

基于层次化调度策略和动态数据复制的网格调度方法

针对在网格中如何有效地进行任务调度和数据复制, 以便减少任务执行时间等问题, 提出了任务调度算法（ISS）和优化动态数据复制算法（ODHRA）, 并构建一个方案将两种算法进行了有效结合。该方案采用ISS算法综合考虑任务等待队列的数量、任务需求数据的位置和站点的计算容量, 采用网络结构分级调度的方式, 配以适当的权重系数计算综合任务成本, 搜索出最佳计算节点区域; 采用ODHRA算法分析数据传输时间、存储访问延迟、等待在存储队列中的副本请求和节点间的距离, 在众多的副本中选取出最佳副本位置, 再结合副本放置和副本管理, 从而降低了文件访问时间。仿真结果表明, 提出的方案在平均任务执行时间方面, 与其他算法相比表现出了更好的性能。     

A bandwidth and effective hit optimal cache scheme for wireless data access networks with client injected updates

In this paper, we propose an optimal cache replacement policy for data access applications in wireless networks where data updates are injected from all the clients. The goal of the policy is to increase effective hits in the client caches and in turn, make efficient use of the network bandwidth in wireless environment. To serve the applications with the most updated data, we also propose two enhanced cache access policies making copies of data objects strongly consistent. We analytically prove that a cache system, with a combination of our cache access and replacement policy, guarantees the optimal number of effective cache hits and optimal cost (in terms of network bandwidth) per data object access. Results from both analysis and extensive simulations demonstrate that the proposed policies outperform the popular Least Frequently Used (LFU) scheme in terms of both effective hits and bandwidth consumption. Our flexible system model makes the proposed policies equally applicable to applications for the existing 3G, as well as upcoming LTE, LTE Advanced and WiMAX wireless data access networks.     

数据网格的动态复制选址策略及应用研究

动态复制技术对于提高数据网格的性能是非常重要的。对目前的动态复制策略进行了综述,鉴于目前效果较好的动态复制策略均为单选址算法,对于延迟较大、分布较广的网格存在很大局限性。提出了三种多选址的动态复制策略,并将它们转化为经典的数学问题进行求解。在给出了多选址动态复制策略在远程教育资源管理中的应用后,在欧洲数据网格试验床1拓扑上进行了仿真实验,实验结果表明:与目前效果较好的选址策略相比,所提出的多选址策略对于减少网络负载和网络延迟效果显著。     

Improved version of parallel programming interface for distributed data with multiple helper servers

We present an improved version of the Parallel Programming Interface for Distributed Data with Multiple Helper Servers (PPIDDv2) library, which provides a common application programming interface that is based on the most frequently used functionality of both MPI-2 and GA. Compared with the previous version, the PPIDDv2 library introduces multiple helper servers to facilitate global data structures, and allows programmers to make heavy use of large global data structures efficiently.

Program summary

Program title: PPIDDv2Catalogue identifier: AEEF_v2_0Program summary URL:http://cpc.cs.qub.ac.uk/summaries/AEEF_v2_0.htmlProgram obtainable from: CPC Program Library, Queen?s University, Belfast, N. IrelandLicensing provisions: Standard CPC licence, http://cpc.cs.qub.ac.uk/licence/licence.htmlNo. of lines in distributed program, including test data, etc.: 22 997No. of bytes in distributed program, including test data, etc.: 184 477Distribution format: tar.gzProgramming language: Fortran, CComputer: Many parallel systemsOperating system: VariousHas the code been vectorised or parallelised?: Yes. 2–1024 processors usedRAM: 50 MbytesClassification: 6.5External routines: Global Arrays or MPI-2Catalogue identifier of previous version: AEEF_v1_0Journal reference of previous version: Comput. Phys. Comm. 180 (2009) 2673Does the new version supersede the previous version?: YesNature of problem: Many scientific applications require management and communication of data that is global, and the standard MPI-2 protocol provides only low-level methods for the required one-sided remote memory access.Solution method: The Parallel Programming Interface for Distributed Data (PPIDD) library provides an interface, suitable for use in parallel scientific applications, that delivers communications and global data management. The library can be built either using the Global Arrays (GA) toolkit, or a standard MPI-2 library. This abstraction allows the programmer to write portable parallel codes that can utilise the best, or only, communications library that is available on a particular computing platform.Reasons for new version: In the previous version, functionality in global data structure was mainly implemented by MPI-2 passive one-sided operations. In real applications which make heavy use of global data structures, very poor performance was observed.Summary of revisions: Multiple helper servers are introduced to facilitate the manipulation and management of global data structure. Mutual exclusion is also implemented by the help of a data server, and becomes much more robust and efficient. In addition, flexible options are provided to choose different settings for helper servers. Significant improvement has been seen in performance tests.Running time: Problem-dependent. The test provided with the distribution takes only a few seconds to run.     

The cgmCUBE project: Optimizing parallel data cube generation for ROLAP

On-line Analytical Processing (OLAP) has become one of the most powerful and prominent technologies for knowledge discovery in VLDB (Very Large Database) environments. Central to the OLAP paradigm is the data cube, a multi-dimensional hierarchy of aggregate values that provides a rich analytical model for decision support. Various sequential algorithms for the efficient generation of the data cube have appeared in the literature. However, given the size of contemporary data warehousing repositories, multi-processor solutions are crucial for the massive computational demands of current and future OLAP systems. In this paper we discuss the cgmCUBE Project, a multi-year effort to design and implement a multi-processor platform for data cube generation that targets the relational database model (ROLAP). More specifically, we discuss new algorithmic and system optimizations relating to (1) a thorough optimization of the underlying sequential cube construction method and (2) a detailed and carefully engineered cost model for improved parallel load balancing and faster sequential cube construction. These optimizations were key in allowing us to build a prototype that is able to produce data cube output at a rate of over one TeraByte per hour. Research supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).     

A branch-and-price algorithm for the general case of scheduling parallel machines with a single server

We consider the strongly NP-hard problem of scheduling two-operation non-preemptable jobs on two identical parallel machines. A single server, that can handle at most one job at a time, is available to carry out the first (or setup) operation. The second operation, to be carried out on the same machine but without the server, must be executed immediately after the setup. The objective is to minimize the makespan. We apply a column generation method to a population of partial schedules, in turn generated by some well known heuristics, to achieve effective and efficient solutions. We compare the performance of this method with those proposed earlier and also suggest future work.     

Dynamic replica placement and selection strategies in data grids— A comprehensive survey

Data replication techniques are used in data grid to reduce makespan, storage consumption, access latency and network bandwidth. Data replication enhances data availability and thereby increases the system reliability. There are two steps involved in data replication, namely, replica placement and replica selection. Replica placement involves identifying the best possible node to duplicate data based on network latency and user request. Replica selection involves selecting the best replica location to access the data for job execution in the data grid. Various replica placement and selection algorithms are available in the literature. These algorithms measure and analyze different parameters such as bandwidth consumption, access cost, scalability, execution time, storage consumption and makespan. In this paper, various replica placement and selection strategies along with their merits and demerits are discussed. This paper also analyses the performance of various strategies with respect to the parameters mentioned above. In particular, this paper focuses on the dynamic replica placement and selection strategies in the data grid environment.     

Privacy-preserving smart metering with multiple data Consumers

The increasing diffusion of Automatic Meter Reading (AMR) and the possibility to open the system to third party services has raised many concerns about the protection of personal data related to energy, water or gas consumption, from which details about the habits of the users can be inferred.This paper proposes an infrastructure and a communication protocol for allowing utilities and third parties (data Consumers) to collect measurement data with different levels of spatial and temporal aggregation from smart meters without revealing the individual measurements to any single node of the architecture.The proposed infrastructure introduces a set of functional nodes in the smart grid, namely the Privacy Preserving Nodes (PPNs), which collect customer data encrypted by means of Shamir’s Secret Sharing Scheme, and are supposed to be controlled by independent parties. By exploiting the homomorphic properties of the sharing scheme, the measurements can be aggregated directly in the encrypted domain. Therefore, an honest-but-curious attacker can obtain neither disaggregated nor aggregated data. The PPNs perform different spatial and temporal aggregation for each Consumer according to its needs and access rights. The information Consumers recover the aggregated data by collecting multiple shares from the PPNs.The paper also discusses the problem of deploying the information flows from the customers to the PPNs and, then, to the information Consumers in a resource constrained environment. We prove that minimizing the number of PPNs is a NP-hard problem and propose a fast greedy algorithm. The scalability of the infrastructure is first analyzed under the assumption that the communication network is reliable and timely, then in presence of communication errors and node failures. The paper also evaluates the anonymity of external attackers.     

数据网格的多选址动态复制策略及其应用研究

动态复制技术对于提高数据网格的性能非常重要。对目前的动态复制策略进行了综述,鉴于目前效果较好的动态复制策略均为单选址算法,对于延迟较大、分布较广的网格存在很大局限性,提出三种多选址的动态复制策略,并将其转化为经典的数学问题进行求解。然后,给出了多选址动态复制策略在大规模分布式资源管理中的应用。     

An effective parallel approach for genetic-fuzzy data mining

Data mining is most commonly used in attempts to induce association rules from transaction data. In the past, we used the fuzzy and GA concepts to discover both useful fuzzy association rules and suitable membership functions from quantitative values. The evaluation for fitness values was, however, quite time-consuming. Due to dramatic increases in available computing power and concomitant decreases in computing costs over the last decade, learning or mining by applying parallel processing techniques has become a feasible way to overcome the slow-learning problem. In this paper, we thus propose a parallel genetic-fuzzy mining algorithm based on the master–slave architecture to extract both association rules and membership functions from quantitative transactions. The master processor uses a single population as a simple genetic algorithm does, and distributes the tasks of fitness evaluation to slave processors. The evolutionary processes, such as crossover, mutation and production are performed by the master processor. It is very natural and efficient to run the proposed algorithm on the master–slave architecture. The time complexities for both sequential and parallel genetic-fuzzy mining algorithms have also been analyzed, with results showing the good effect of the proposed one. When the number of generations is large, the speed-up can be nearly linear. The experimental results also show this point. Applying the master–slave parallel architecture to speed up the genetic-fuzzy data mining algorithm is thus a feasible way to overcome the low-speed fitness evaluation problem of the original algorithm.     

An Anticipative Recursively Adjusting Mechanism for parallel file transfer in data grids

Data Grids enable the sharing, selection, and connection of a wide variety of geographically distributed computational and storage resources for content needed by large‐scale data‐intensive applications such as high‐energy physics, bioinformatics, and virtual astrophysical observatories. In Data Grids, co‐allocation architectures were developed to enable parallel downloads of data sets from selected replica servers. As Internet is usually the underlying network of a grid, network bandwidth plays as the main factor affecting file transfers between clients and servers. In this paradigm, there are still some challenges that need to be solved, such as to reduce differences in finish times between selected replica servers, to avoid traffic congestion resulting from transferring the same blocks in different links among servers and clients, and to manage network performance variations among parallel transfers. In this paper, we propose the Anticipative Recursively Adjusting Mechanism (ARAM) scheme to adjust the workloads on selected replica servers and handle unpredictable variations in network performance by those servers. Our algorithm is based on using the finish rates for previously assigned transfers to anticipate the bandwidth status for the next section to adjust workloads, and to reduce file transfer times in grid environments. Our approach is useful in grid environments with unstable network link. It not only reduces idle time wasted waiting for the slowest server, but also decreases file transfer completion times. Copyright © 2010 John Wiley & Sons, Ltd.     

Block models for scheduling jobs on two parallel machines with a single server

We consider the problem of scheduling a set of non-preemptable jobs on two identical parallel machines such that the makespan is minimized. Before processing, each job must be loaded on a machine, which takes a given setup time. All these setups have to be done by a single server which can handle at most one job at a time. For this problem, we propose a mixed integer linear programming formulation based on the idea of decomposing a schedule into a set of blocks. We compare the results obtained by the model suggested with known heuristics from the literature.     

A novel dynamic load balancing scheme for parallel systems

Adaptive mesh refinement (AMR) is a type of multiscale algorithm that achieves high resolution in localized regions of dynamic, multidimensional numerical simulations. One of the key issues related to AMR is dynamic load balancing (DLB), which allows large-scale adaptive applications to run efficiently on parallel systems. In this paper, we present an efficient DLB scheme for structured AMR (SAMR) applications. This scheme interleaves a grid-splitting technique with direct grid movements (e.g., direct movement from an overloaded processor to an underloaded processor), for which the objective is to efficiently redistribute workload among all the processors so as to reduce the parallel execution time. The potential benefits of our DLB scheme are examined by incorporating our techniques into a SAMR cosmology application, the ENZO code. Experiments show that by using our scheme, the parallel execution time can be reduced by up to 57% and the quality of load balancing can be improved by a factor of six, as compared to the original DLB scheme used in ENZO.     

Generating data transfers for distributed GPU parallel programs

Nowadays, high performance applications exploit multiple level architectures, due to the presence of hardware accelerators like GPUs inside each computing node. Data transfers occur at two different levels: inside the computing node between the CPU and the accelerators and between computing nodes. We consider the case where the intra-node parallelism is handled with HMPP compiler directives and message-passing programming with MPI is used to program the inter-node communications. This way of programming on such an heterogeneous architecture is costly and error-prone. In this paper, we specifically demonstrate the transformation of HMPP programs designed to exploit a single computing node equipped with a GPU into an heterogeneous HMPP + MPI exploiting multiple GPUs located on different computing nodes.     

A throughput maximised parallel architecture for 2D fast Discrete Pascal Transform

In this paper, we present a fully pipelined parallel implementation of a two dimensional (2D) Discrete Pascal Transform (DPT). Our approach first makes use of the properties of the Kronecker product and the vec operation on matrices to form an alternate 2D DPT representation suitable for column parallel computation. Next, we lend ourselves to the results from Skodras’ work in 1D DPT to achieve the final architecture for fast 2D DPT. With a fully pipelined implementation, the architecture possesses an initial latency of 2(N-1) clock cycles and a maximum throughput of one complete two dimensional transform every clock cycle, given any input matrix of size N×N. To evaluate our work, our results obtained from actual FPGA implementation were benchmarked against results from other previous works.     

CEA: A Cyclic Expansion Algorithm for data migration in parallel video servers

Parallel video servers can achieve highly storage-saving and granularly load-balancing, but they suffer from a system expansion problem. As the number of users continuously increases, the system inevitably needs to expand the number of video servers. However, the expansion of a parallel video server system is not as simple as that of a replicated video server system. Hence, this work develops an efficient expansion algorithm, called the Cyclic Expansion Algorithm (CEA), for parallel video servers. The proposed CEA algorithm has several good features. First, the data layout of each video content exhibits periodicity. Consequently, the meta-data size of each video and the complexity of the CEA algorithm are reduced. Second, the number of required data movements during a system expansion is optimized. Third, the total number of required XOR recomputations for updating parity blocks during an expansion is also minimized. Additionally, the new CEA can be applied to a variety of distributed storage systems, such as the cloud-based storage systems using striping and parity check techniques.     

Near real-time parallel processing and advanced data management of SAR images in grid environments

In this paper, we describe the process of parallelizing an existing, production level, sequential Synthetic Aperture Radar (SAR) processor based on the Range-Doppler algorithmic approach. We show how, taking into account the constraints imposed by the software architecture and related software engineering costs, it is still possible with a moderate programming effort to parallelize the software and present an message-passing interface (MPI) implementation whose speedup is about 8 on 9 processors, achieving near real-time processing of raw SAR data even on a moderately aged parallel platform. Moreover, we discuss a hybrid two-level parallelization approach that involves the use of both MPI and OpenMP. We also present GridStore, a novel data grid service to manage raw, focused and post-processed SAR data in a grid environment. Indeed, another aim of this work is to show how the processed data can be made available in a grid environment to a wide scientific community, through the adoption of a data grid service providing both metadata and data management functionalities. In this way, along with near real-time processing of SAR images, we provide a data grid-oriented system for data storing, publishing, management, etc.

Increasing throughput for robotic cells with parallel Machines and multiple robots

We address the problem of scheduling robots' moves in a robotic cell that is used by a Dallas-area semiconductor equipment manufacturer. The cell has parallel machines, multiple robots, and Euclidean travel times. We describe a plan of operation that allows the robots to operate concurrently, efficiently, and with no risk of colliding. We propose a set of sequences of robot moves, analytically determine this scheme's throughput, and determine problem instances for which it is optimal. Through simulation, we demonstrate that our scheme is superior to the heuristic dispatching rule currently in use by the manufacturer.

Note to Practitioners-Efficient scheduling of a robotic cell can greatly increase productivity and revenue for manufacturers in many different industries. This increase becomes more pronounced for larger cells that employ multiple robots and parallel machines at various production stages. This paper describes a schedule of robotic actions that is optimal under a common set of conditions for such large cells, in addition to many other types of cells. When this set of conditions does not hold, even though optimality could not be proven, this schedule is shown to be superior to one currently in use by some semiconductor manufacturers. We also present a scheme that allows the robots to operate concurrently, efficiently, and with no risk of colliding. Additionally, an approximation to the improvement in revenues realized by using this schedule is provided.