A Meta-Brokering Framework for Science Gateways

Recently scientific communities produce a growing number of computation-intensive applications, which calls for the interoperation of distributed infrastructures including Clouds, Grids and private clusters. The European SHIWA and ER-flow projects have enabled the combination of heterogeneous scientific workflows, and their execution in a large-scale system consisting of multiple Distributed Computing Infrastructures. One of the resource management challenges of these projects is called parameter study job scheduling. A parameter study job of a workflow generally has a large number of input files to be consumed by independent job instances. In this paper we propose a meta-brokering framework for science gateways to support the execution of such workflows. In order to cope with the high uncertainty and unpredictable load of the utilized distributed infrastructures, we introduce the so called resource priority services. These tools are capable of determining and dynamically updating priorities of the available infrastructures to be selected for job instances. Our evaluations show that this approach implies an efficient distribution of job instances among the available computing resources resulting in shorter makespan for parameter study workflows.     

A Set of Successive Job Allocation Models in Distributed Computing Infrastructures

The growing number of scientific computation-intensive applications calls for an efficient utilization of large-scale, potentially interoperable distributed infrastructures. Parameter sweep applications represent a large body of workflows. While the principle of workflows is easy to conceive, their execution is very complex and no universally accepted solution exists. In this paper we focus on the resource allocation challenges of parameter study jobs in distributed computing infrastructures. To cope with this NP-hard problem and the high uncertainty present in these systems, we propose a series of job allocation models that helps refining and simplifying the problem complexity. In this way we present some special cases that are polynomial and show how more complex scenarios can be reduced to these models. It is known from practice that a small number of job sizes improves the result of job allocation, therefore we state a hypothesis relying on this fact in one of our models. Unfortunately, the reduction of the general problem (using K-means clustering) did not help, and thus the hypothesis has proved to be false. In the future, we shall look for clustering techniques which fit this goal better.     

Harnessing Cloud Technologies for a Virtualized Distributed Computing Infrastructure

The InterGrid system aims to provide an execution environment for running applications on top of interconnected infrastructures. The system uses virtual machines as building blocks to construct execution environments that span multiple computing sites. Such environments can be extended to operate on cloud infrastructures, such as Amazon EC2. This article provides an abstract view of the proposed architecture and its implementation; experiments show the scalability of an InterGrid-managed infrastructure and how the system can benefit from using the cloud.     

On Efficiency of Multi-job Grid Allocation Based on Statistical Trace Data

The ever growing number of computation-intensive applications calls for utilizing large-scale, potentially interoperable distributed infrastructures. Nowadays, such distributed systems enable the management of heterogeneous scientific workflows of considerable sizes, where job scheduling and resource management is a crucial issue. In this paper we focus on the challenges of scheduling parameter sweep applications, a specific and commonly used type of workflows where ordering of job executions is irrelevant. A parameter sweep has a large set of independent job instances, called a multi-job, submitted for execution in a single step. In order to cope with the high uncertainty and unpredictable load of resources, and the simultaneous submissions of multi-job instances, we propose a statistics-based brokering approach for allocating jobs to resources so that the makespan is minimised. Earlier studies claim that users’ predictions on job runtime are inaccurate and unusable for scheduling. Our aim is to examine, whether statistical trace data for the same purpose is efficient compared to randomized allocation.     

GA-based cloud resource estimation for agent-based execution of bag-of-tasks applications

Executing bag-of-tasks applications in multiple Cloud environments while satisfying both consumers’ budgets and deadlines poses the following challenges: How many resources and how many hours should be allocated? What types of resources are required? How to coordinate the distributed execution of bag-of-tasks applications in resources composed from multiple Cloud providers?. This work proposes a genetic algorithm for estimating suboptimal sets of resources and an agent-based approach for executing bag-of-tasks applications simultaneously constrained by budgets and deadlines. Agents (endowed with distributed algorithms) compose resources and coordinate the execution of bag-of-tasks applications. Empirical results demonstrate that the genetic algorithm can autonomously estimate sets of resources to execute budget-constrained and deadline-constrained bag-of-tasks applications composed of more economical (but slower) resources in the presence of loose deadlines, and more powerful (but more expensive) resources in the presence of large budgets. Furthermore, agents can efficiently and successfully execute randomly generated bag-of-tasks applications in multi-Cloud environments.     

Cost–Performance Optimization of Application- and Context-Aware Distributed Infrastructures

Application- and context-aware infrastructures involve the network in the execution of distributed applications through special devices, namely, the application cards, placed in network nodes. The sharp separation of applications and network is smoothed, and by performing part of the distributed application inside the network, it is possible to reduce costs and improve performance with a better optimization of the whole distributed information and communication technology (ICT) infrastructure. This optimization is allowed by the additional degrees of freedom of placing cards in nodes and of assigning applications to such cards. In this paper, we provide an optimization algorithm that minimizes the total cost of the entire distributed ICT infrastructure, given a target performance objective defined as the end-to-end delay for the completion of the distributed application tasks. We focus on two sample applications that are well suited for application- and context-aware infrastructures: caching and protocol translation. The joint optimization of computing and communication resources is an innovative contribution of this paper, as, in the literature, hardware and network components are typically optimized separately. Results show that the total infrastructural cost savings are in the range of 15%-20%. However, savings can be obtained only if cards satisfy a cost-performance curve that is also analyzed.     

A self-adaptive resources selection model through a small-world based heuristic

The small-world phenomenon is a principle in which seemingly distant nodes are linked by short chains of acquaintances. This property is found in a wide range of biological, social or natural networks. We proposed a self-adaptive model for solving the grid computing resources selection problem. A heuristic based on small-world concepts is defined within this model. Grid computing infrastructures are distributed systems with heterogeneous and geographically distributed resources. The present approach selects the most efficient resources during the application execution for facing the environmental changes. The model is tested in a real European grid computing infrastructure. Finally, from the results that have been obtained during the evaluation phase it is possible to conclude that the model achieves a reduction in applications execution time as well as an increase in the successfully completed tasks rate.     

A concurrent rule scheduling algorithm for active rules

The use of rules in a distributed environment creates new challenges for the development of active rule execution models. In particular, since a single event can trigger multiple rules that execute over distributed sources of data, it is important to make use of concurrent rule execution whenever possible. This paper presents the details of the integration rule scheduling (IRS) algorithm. Integration rules are active database rules that are used for component integration in a distributed environment. The IRS algorithm identifies rule conflicts for multiple rules triggered by the same event through static, compile-time analysis of the read and write sets of each rule. A unique aspect of the algorithm is that the conflict analysis includes the effects of nested rule execution that occurs as a result of using an execution model with an immediate coupling mode. The algorithm therefore identifies conflicts that may occur as a result of the concurrent execution of different rule triggering sequences. The rules are then formed into a priority graph before execution, defining the order in which rules triggered by the same event should be processed. Rules with the same priority can be executed concurrently. The IRS algorithm guarantees confluence in the final state of the rule execution. The IRS algorithm is applicable for rule scheduling in both distributed and centralized rule execution environments.     

Planning virtual infrastructures for time critical applications with multiple deadline constraints

Executing time critical applications within cloud environments while satisfying execution deadlines and response time requirements is challenging due to the difficulty of securing guaranteed performance from the underlying virtual infrastructure. Cost-effective solutions for hosting such applications in the Cloud require careful selection of cloud resources and efficient scheduling of individual tasks. Existing solutions for provisioning infrastructures for time constrained applications are typically based on a single global deadline. Many time critical applications however have multiple internal time constraints when responding to new input. In this paper we propose a cloud infrastructure planning algorithm that accounts for multiple overlapping internal deadlines on sets of tasks within an application workflow. In order to better compare with existing work, we adapted the IC-PCP algorithm and then compared it with our own algorithm using a large set of workflows generated at different scales with different execution profiles and deadlines. Our results show that the proposed algorithm can satisfy all overlapping deadline constraints where possible given the resources available, and do so with consistently lower host cost in comparison with IC-PCP.     

Bundle and Pool Architecture for Multi-Language, Robust, Scalable Workflow Executions

In this paper, we leverage the previous work on the SHIWA bundling format and expand on this specification in order to facilitate workflow execution within a multi-workflow environment. We introduce a scalable and robust execution pool environment that supports workflows consisting of sub-workflows built upon a multitude of different workflow engines and environments, and also provide a common workflow representation for seamless connectivity through serialization to workflow bundles. We also present a meta-workflow scenario based upon this system. Workflow bundles employ the lightweight Open Archives Initiative Object Reuse and Exchange (ORE) Web-based standard, to provide a common format for representing and sharing workflows and the associated metadata required for their execution. This generalized bundling approach is already available within five workflow engines and has proven a useful environment for inter-workflow experimentation. The execution pool facilitates federated access to multiple distributed computing infrastructures supported by the underlying workflow engines subscribed to the pool. Workflow bundles are exposed using the eXtensible Messaging and Presence Protocol (XMPP), which provides the necessary communication backbone to enable multiple workflow engine agents to asynchronously publish and subscribe to bundles in meta-workflow pipelines. We present experiments showing the scalability and robustness of the pool execution approach with results showing that overheads remain controlled for up to 150 workflow agents, and that agent failures have very limited impact. We then demonstrate the applicability of our architecture by describing how a Java-based music analysis workflow can be distributed within such a multi-workflow environment consisting of the Triana and MOTEUR workflow engines.     

Designing and evaluating an energy efficient Cloud

Cloud infrastructures have recently become a center of attention. They can support dynamic operational infrastructures adapted to the requirements of distributed applications. As large-scale distributed systems reach enormous sizes in terms of equipment, the energy consumption issue becomes one of the main challenges for large-scale integration. Like any other large-scale distributed system, Clouds face an increasing demand in energy. In this paper, we explore the energy issue by analyzing how much energy virtualized environments cost. We provide an energy-efficient framework dedicated to Cloud architectures and we validate it through different experimentations on a modern multicore platform. We show on a realistic example that our infrastructure could save 25% of the Cloud nodes’ electrical consumption.     

Mobile-agent coordination models for Internet applications

Internet applications face challenges that mobile agents and the adoption of enhanced coordination models may overcome. Each year more applications shift from intranets to the Internet, and Internet-oriented applications become more popular. New design and programming paradigms call help harness the Web's potential. Traditional distributed applications assign a set of processes to a given execution environment that, acting as local-resource managers, cooperating a network-unaware fashion. In contrast, the mobile-agent paradigm defines applications as consisting of network-aware entities-agents-which can exhibit mobility by actively changing their execution environment, transferring themselves during execution. The authors propose a taxonomy of possible coordination models for mobile-agent applications, then use their taxonomy to survey and analyze resent mobile-agent coordination proposals. Their case study, which focuses on a Web-based information-retrieval application, helps show that the mobility of application components and the distribution area's breadth can create coordination problems different from those encountered in traditional distributed applications     

Biased random walks on resource network graphs for load balancing

The currently emerging large-scale complex networks and networks of networks are becoming apparent in the pervasive supply of seamless and transparent access to heterogeneous resources and services such as network domains, applications, services and storage owned by multiple organizations. The dynamics and heterogeneous environments involved, however, pose many challenges for controlling and balancing resource access, composition and deployment across complex grid and network infrastructures. In this paper, a scheme is proposed that gives a distributed load-balancing scheme by generating almost regular resource allocation networks. This network system is self-organized and depends only on local information for load distribution and resource discovery. The in-degree of each node refers to its free resources, and the job assignment and resource discovery processes required for load-balancing are accomplished by using fitted random sampling. Simulation results show that the generated network system provides an effective, scalable, and reliable load-balancing scheme for the distributed resources in grids and networks. The proposed solution is tested with real world data and the performance is tested against a recently reported distributed algorithm for load balancing.     

Overhead Analysis of Scientific Workflows in Grid Environments

Scientific workflows are a topic of great interest in the grid community that sees in the workflow model an attractive paradigm for programming distributed wide-area grid infrastructures. Traditionally, the grid workflow execution is approached as a pure best effort scheduling problem that maps the activities onto the grid processors based on appropriate optimization or local matchmaking heuristics such that the overall execution time is minimized. Even though such heuristics often deliver effective results, the execution in dynamic and unpredictable grid environments is prone to severe performance losses that must be understood for minimizing the completion time or for the efficient use of high-performance resources. In this paper, we propose a new systematic approach to help the scientists and middleware developers understand the most severe sources of performance losses that occur when executing scientific workflows in dynamic grid environments. We introduce an ideal model for the lowest execution time that can be achieved by a workflow and explain the difference to the real measured grid execution time based on a hierarchy of performance overheads for grid computing. We describe how to systematically measure and compute the overheads from individual activities to larger workflow regions and adjust well-known parallel processing metrics to the scope of grid computing, including speedup and efficiency. We present a distributed online tool for computing and analyzing the performance overheads in real time based on event correlation techniques and introduce several performance contracts as quality-of-service parameters to be enforced during the workflow execution beyond traditional best effort practices. We illustrate our method through postmortem and online performance analysis of two real-world workflow applications executed in the Austrian grid environment.     

Large-scale climate simulations harnessing clusters,grid and cloud infrastructures

The current availability of a variety of computing infrastructures including HPC, Grid and Cloud resources provides great computer power for many fields of science, but their common profit to accomplish large scientific experiments is still a challenge. In this work, we use the paradigm of climate modeling to present the key problems found by standard applications to be run in hybrid distributed computing infrastructures and propose a framework to allow a climate model to take advantage of these resources in a transparent and user-friendly way. Furthermore, an implementation of this framework, using the Weather Research and Forecasting system, is presented as a working example. In order to illustrate the usefulness of this framework, a realistic climate experiment leveraging Cluster, Grid and Cloud resources simultaneously has been performed. This test experiment saved more than 75% of the execution time, compared to local resources. The framework and tools introduced in this work can be easily ported to other models and are probably useful in other scientific areas employing data- and CPU-intensive applications.     

云环境下影响数据分布并行应用执行效率的因素分析

云环境下,类似MapReduce的数据分布并行应用被广泛运用。针对此类应用执行效率低、成本高的问题,以Hadoop为例,首先,分析该类应用的执行方式,发现数据量、节点数和任务数是影响其效率的主要因素;其次,探讨以上因素对应用效率的影响;最后,通过实验得出在数据量一定的情况下,增加节点数不会明显提高应用的执行效率,反而极大地增加执行成本;当任务数接近节点数时,应用的执行效率较高、成本较低。该结论为云环境中类似MapReduce的数据分布并行应用的效率优化提供借鉴,并为用户租用云资源提供参考。     

KOALA: a co‐allocating grid scheduler

In multicluster systems, and more generally in grids, jobs may require co‐allocation, that is, the simultaneous or coordinated access of single applications to resources of possibly multiple types in multiple locations managed by different resource managers. Co‐allocation presents new challenges to resource management in grids, such as locating sufficient resources in geographically distributed sites, allocating and managing resources in multiple, possibly heterogeneous sites for single applications, and coordinating the execution of single jobs at multiple sites. Moreover, as single jobs now may have to rely on multiple resource managers, co‐allocation introduces reliability problems. In this paper, we present the design and implementation of a co‐allocating grid scheduler named KOALA that meets these co‐allocation challenges. In addition, we report on the results of an analysis of the performance in our multicluster testbed of the co‐allocation policies built into KOALA . We also include the results of a performance and reliability test of KOALA while our testbed was unstable. Copyright © 2007 John Wiley & Sons, Ltd.     

An Integrated Approach to Designing and Evaluating Collaborative Applications and Infrastructures

Collaborative systems include both general infrastructures and specific applications for supporting collaboration. Because of the relative newness and complexity of these systems, it has been unclear what approach should be used to design and evaluate them. Based on the lessons learned from our work and that of others on collaborative systems, we have derived an integrated approach to researching collaborative applications and infrastructures. The approach can be described as a sequence of steps: We decompose the functionality of collaboration systems into smaller functions that can be researched more-or-less independently. For each of these functions, we adopt general (system-independent) principles regarding the design and implementation of the function, identify collaboration scenarios at multiple levels of abstraction, identify requirements based on the scenarios, adopt an interaction model to meet the requirements, realize the interaction model as a concrete user interface, develop a logical architecture of the system, identify a physical architecture for placing the logical components in a distributed system, develop infrastructure abstractions, use the abstractions to implement applications, and perform lab studies, field experiments, and simulations to evaluate the infrastructure and applications. As in other models with multiple phases, feedback from subsequent phases is used to modify the results from the previous phases. In this paper, we describe, illustrate and motivate this research plan.