Reputation-based dependable scheduling of workflow applications in Peer-to-Peer Grids

Grids facilitate creation of wide-area collaborative environment for sharing computing or storage resources and various applications. Inter-connecting distributed Grid sites through peer-to-peer routing and information dissemination structure (also known as Peer-to-Peer Grids) is essential to avoid the problems of scheduling efficiency bottleneck and single point of failure in the centralized or hierarchical scheduling approaches. On the other hand, uncertainty and unreliability are facts in distributed infrastructures such as Peer-to-Peer Grids, which are triggered by multiple factors including scale, dynamism, failures, and incomplete global knowledge.In this paper, a reputation-based Grid workflow scheduling technique is proposed to counter the effect of inherent unreliability and temporal characteristics of computing resources in large scale, decentralized Peer-to-Peer Grid environments. The proposed approach builds upon structured peer-to-peer indexing and networking techniques to create a scalable wide-area overlay of Grid sites for supporting dependable scheduling of applications. The scheduling algorithm considers reliability of a Grid resource as a statistical property, which is globally computed in the decentralized Grid overlay based on dynamic feedbacks or reputation scores assigned by individual service consumers mediated via Grid resource brokers. The proposed algorithm dynamically adapts to changing resource conditions and offers significant performance gains as compared to traditional approaches in the event of unsuccessful job execution or resource failure. The results evaluated through an extensive trace driven simulation show that our scheduling technique can reduce the makespan up to 50% and successfully isolate the failure-prone resources from the system.     

Towards supporting multiple virtual private computing environments on computational Grids

Grid computing now becomes a practical computing paradigm and solution for distributed systems and applications. Currently increasing resources are involved in Grid environments and a large number of applications are running on computational Grids. Unfortunately Grid computing technologies are still far away from reach of inexperienced application users, e.g., computational scientists and engineers. A software layer is required to provide an easy interface of Grids to end users.To meet this requirement HEAVEN (Hosting European Application Virtual ENvironment) upperware is proposed to build on top of Grid middleware. This paper presents HEAVEN philosophy of virtual computing for Grids – a combinational idea of simulation and emulation approaches. The concept of Virtual Private Computing Environment (VPCE) is thereafter proposed and defined. The design and current implementation of HEAVEN upperware are discussed in detail. Use case of Ag2D application justifies the philosophy of HEAVEN virtual computing methodology and the design/implementation of HEAVEN upperware.     

Grid programming: some indications where we are headed

Grid computing enables the development of large scientific applications on an unprecedented scale. Grid-aware applications, also called meta-applications or multi-disciplinary applications, make use of coupled computational resources that are not available at a single site. In this light, the Grids let scientists solve larger or new problems by pooling together resources that could not be coupled easily before. It is well known that the programmer’s productivity in designing and implementing efficient distributed/parallel applications on high-performance computers is still usually a very time-consuming task. Grid computing makes the situation worse. Consequently, the development of Grid programming environments that would enable programmers to efficiently exploit this technology is an important and hot research issue.

After an introduction on the main Grid programming issues, this paper will review the most important approaches/projects conducted in this field worldwide.     

Large scale distributed visualization on computational Grids: A review

Advances in science and engineering have put high demands on tools for high performance large-scale data exploration and analysis. Visualization is a powerful technology for analyzing data and presenting results. Todays science and engineering have benefited from state-of-the-art of Grid technologies and modern visualization systems. To visualize the large amount of data, rendering technologies are widely used to parallelize visualization tasks over distributed resources on computational Grids. It raises the necessity to balance the computational load and to minimize the network bandwidth requirements. This article explains in Grid environments how new approaches of visualization architecture and load-balancing algorithms address these challenges in a principled fashion. The Grid infrastructure that supports large scale distributed visualization is also introduced. Some typical visualization systems on Grids are referenced for discussions.     

A Taxonomy of Workflow Management Systems for Grid Computing

With the advent of Grid and application technologies, scientists and engineers are building more and more complex applications to manage and process large data sets, and execute scientific experiments on distributed resources. Such application scenarios require means for composing and executing complex workflows. Therefore, many efforts have been made towards the development of workflow management systems for Grid computing. In this paper, we propose a taxonomy that characterizes and classifies various approaches for building and executing workflows on Grids. We also survey several representative Grid workflow systems developed by various projects world-wide to demonstrate the comprehensiveness of the taxonomy. The taxonomy not only highlights the design and engineering similarities and differences of state-of-the-art in Grid workflow systems, but also identifies the areas that need further research.     

Characterizing Grids: Attributes, Definitions, and Formalisms

Grid systems and technologies have evolved over nearly a decade; yet, there is still no widely accepted definition for Grids. In particular, the essential attributes that distinguish Grids from other distributed computing environments have not been articulated. Most approaches to definition adopt a static view and consider only the properties and components of, or the applications supported by, Grids. The definition proposed in this paper is based on the runtime semantics of distributed systems. Rather than attempt to simply compare static characteristics of Grids and other distributed computing environments, this paper analyzes operational differences, from the viewpoint of an application executing in both environments. Our definition is expressed formally as an Abstract State Machine that facilitates the analysis of existing Grid systems or the design of new ones with rigor and precision. This new, semantical approach proposes an alternative to the currently accepted models for determining whether or not a distributed system is a Grid.     

The Virtual Laboratory: a toolset to enable distributed molecular modelling for drug design on the World‐Wide Grid

Computational Grids are emerging as a new paradigm for sharing and aggregation of geographically distributed resources for solving large‐scale compute and data intensive problems in science, engineering and commerce. However, application development, resource management and scheduling in these environments is a complex undertaking. In this paper, we illustrate the development of a Virtual Laboratory environment by leveraging existing Grid technologies to enable molecular modelling for drug design on geographically distributed resources. It involves screening millions of compounds in the chemical database (CDB) against a protein target to identify those with potential use for drug design. We have used the Nimrod‐G parameter specification language to transform the existing molecular docking application into a parameter sweep application for executing on distributed systems. We have developed new tools for enabling access to ligand records/molecules in the CDB from remote resources. The Nimrod‐G resource broker along with molecule CDB data broker is used for scheduling and on‐demand processing of docking jobs on the World‐Wide Grid (WWG) resources. The results demonstrate the ease of use and power of the Nimrod‐G and virtual laboratory tools for grid computing. Copyright © 2003 John Wiley & Sons, Ltd.     

关于网格及其它分布计算技术的若干问题的讨论

1.引言在“网格:面向虚拟组织的资源共享技术”一文中,我们主要给出了由Ian Foster等定义的网格及相关基本概念和研究领域,讨论了网格的基本理念和关键技术。在“网格体系结构详解”一文中,详述了Globus项目提出的网格体系结构的构成及功能。这些内容旨在说明网格是什么。实际上,我们也可以从另一方面,或不同的角度来观察和认识网格。比     

Entropic Grid Scheduling

Computational Grids (CGs) are large scale dynamical networks of geographically distributed peer resource clusters. These clusters are independent but cooperating computing systems bound by a management framework for the provision of computing services, called Grid Services. In its basic form, the Grid scheduling problem consists in finding at least one cluster that has the capacity to handle, within the constraints of a specified quality of service, a user service request submitted to the CG. Since CGs span distinct management domains, the scheduling process has to be decentralized. Furthermore, it has to account for the ubiquitous uncertainty on the state of the CG. In this paper, we propose a scalable distributed Entropy-based scheduling approach that utilizes a Markov chain model to capture the dynamics of the service capacity state. An entropy-based quantification of the uncertainty on the service capacity information is developed and explicitly integrated within the proposed Grid scheduling approach. The performance of the proposed scheduling strategy is validated, through simulation, against a random delegation scheme and a load balancing-based scheduling strategy with respect to throughput, exploitation and convergence speed, respectively.     

An improved load-balancing mechanism based on deadline failure recovery on GridSim

Grid computing has emerged a new field, distinguished from conventional distributed computing. It focuses on large-scale resource sharing, innovative applications and in some cases, high performance orientation. The Grid serves as a comprehensive and complete system for organizations by which the maximum utilization of resources is achieved. The load balancing is a process which involves the resource management and an effective load distribution among the resources. Therefore, it is considered to be very important in Grid systems. For a Grid, a dynamic, distributed load balancing scheme provides deadline control for tasks. Due to the condition of deadline failure, developing, deploying, and executing long running applications over the grid remains a challenge. So, deadline failure recovery is an essential factor for Grid computing. In this paper, we propose a dynamic distributed load-balancing technique called “Enhanced GridSim with Load balancing based on Deadline Failure Recovery” (EGDFR) for computational Grids with heterogeneous resources. The proposed algorithm EGDFR is an improved version of the existing EGDC in which we perform load balancing by providing a scheduling system which includes the mechanism of recovery from deadline failure of the Gridlets. Extensive simulation experiments are conducted to quantify the performance of the proposed load-balancing strategy on the GridSim platform. Experiments have shown that the proposed system can considerably improve Grid performance in terms of total execution time, percentage gain in execution time, average response time, resubmitted time and throughput. The proposed load-balancing technique gives 7 % better performance than EGDC in case of constant number of resources, whereas in case of constant number of Gridlets, it gives 11 % better performance than EGDC.     

一种基于CCM构件模型的计算网格平台

如何聚合网络中分布异构的计算资源来解决大规模的科学计算问题,和如何减少并行程序设计的复杂性,一直是网格计算研究的难点之一。文章提出了一种基于CORBA构件技术的计算网格新思想,构造了一个计算网格的模型(CCGM)。该模型能够充分地利用构件技术带来的可组装和易管理的特性来形成网格计算。并通过问题的抽象定义和使用ParIDL工具将问题的定义映射到CCGM之上,简化了计算网格应用的开发。通过测试和分析CCG(ComponentbasedComputationalGrid)系统,表明CCG系统具有较好的加速比。     

Peer-to-peer service provisioning in cloud computing environments

This paper aims to advance the management and delivery of services in large, heterogeneous, uncertain, and evolving cloud computing environments. The goal is important because such systems are becoming increasingly popular, yet existing service management methods do not scale well, and nor do they perform well under highly unpredictable conditions. If these problems can be solved, then Information Technology (IT) services can be made to operate in more scalable and reliable manner. In this paper, we present a peer-to-peer approach for managing services in large scale, dynamic, and evolving cloud computing environments. The system components such as virtualized services, computing servers, storage, and databases self-organize themselves using a peer-to-peer networking overlay. Inter-networking system components through peer-to-peer routing and information dissemination structure is essential to avoid the problems of management bottleneck and single point of failure that is predominantly associated with traditional centralized and hierarchical distributed (grids/clouds) system design approaches. We have validated our approach by conducting a set of rigorous performance evaluation study using the Amazon EC2 cloud computing environment. The results prove that managing services based on peer-to-peer routing and information dissemination structure is feasible and offers significant performance benefits as regards to overall system reliability, scalability, and self-management.     

Integration of task abortion and security requirements in GA-based meta-heuristics for independent batch grid scheduling

Scheduling and resource allocation in large scale distributed environments, such as Computational Grids (CGs), arise new requirements and challenges not considered in traditional distributed computing environments. Among these new requirements, task abortion and security become needful criteria for Grid schedulers. The former arises due to the dynamics of the Grid systems, in which resources are expected to enter and leave the system in an unpredictable way. The latter requirement appears crucial in Grid systems mainly due to a multi-domain nature of CGs. The main aim of this paper is to develop a scheduling model that enables the aggregation of task abortion and security requirements as additional, together with makespan and flowtime, scheduling criteria into a cumulative objective function. We demonstrate the high effectiveness of genetic-based schedulers in finding near-optimal solutions for multi-objective scheduling problem, where all criteria (objectives) are simultaneously optimized. The proposed meta-heuristics are experimentally evaluated in static and dynamic Grid scenarios by using a Grid simulator. The obtained results show the fast reduction of the values of basic scheduler performance metrics, especially in the dynamic case, that confirms the usefulness of the proposed approach in real-life scenarios.     

Towards a Powerful European DCI Based on Desktop Grids

Service Grids like the EGEE Grid can not provide the required number of resources for many VOs. Therefore extending the capacity of these VOs with volunteer or institutional desktop Grids would significantly increase the number of accessible computing resources that can particularly advantageously be exploited in case of parameter sweep applications. This objective has been achieved by the EDGeS project that built a production infrastructure enabling the extension of gLite VOs with several volunteer and institutional desktop Grids. The paper describes the technical solution of integrating service Grids and desktop Grids and, the actual EDGeS production infrastructure. The main objectives and current achievements of the follow-up EDGI project have also been described showing how the existing EDGeS infrastructure can be further extended with clouds.     

EDGeS: Bridging EGEE to BOINC and XtremWeb

Desktop Grids, such as XtremWeb and BOINC, and Service Grids, such as EGEE, are two different approaches for science communities to gather computing power from a large number of computing resources. Nevertheless, little work has been done to combine these two Grid technologies in order to establish a seamless and vast Grid resource pool. In this paper we present the EGEE Service Grid, the BOINC and XtremWeb Desktop Grids. Then, we present the EDGeS solution to bridge the EGEE Service Grid with the BOINC and XtremWeb Desktop Grids.     

基于运行时模型的混合云管理方法

随着云计算技术的普及,涌现出众多不同用途、不同类型的云计算平台.为了满足遗产系统整合和动态资源扩展等需求,常常需要构造混合云来统一管理不同云平台中的计算和存储资源.然而,不同云平台的管理接口和管理机制存在差异,使得开发混合云管理系统难度大、复杂度高.本文提出一种基于运行时模型的混合云管理方法：首先,在云平台管理接口基础上,构造单一云平台的运行时模型;其次,根据云平台领域知识,提出一种云平台软件体系结构的统一模型;最后,通过模型转换,实现云平台统一模型到运行时模型的映射.于是,管理程序可以建立在云平台统一模型的基础上,降低了混合云管理系统开发的难度和复杂度.本文还实现了基于运行时模型的CloudStack和亚马逊EC2混合云管理系统,并对方法的可行性和有效性进行了验证.     

Scalable dimensioning of resilient Lambda Grids

Grids consist of the aggregation of numerous dispersed computational, storage and network resources, able to satisfy even the most demanding computing jobs. Due to the data-intensive nature of Grid jobs, there is an increasing interest in Grids using optical transport networks as this technology allows for the timely delivery of large amounts of data. Such Grids are commonly referred to as Lambda Grids.

An important aspect of Grid deployment is the allocation and activation of installed network capacity, needed to transfer data and jobs to and from remote resources. However, the exact nature of a Grid’s network traffic depends on the way arriving workload is scheduled over the various Grid sites. As Grids possibly feature high numbers of resources, jobs and users, solving the combined Grid network dimensioning and workload scheduling problem requires the use of scalable mathematical methods such as Divisible Load Theory (DLT). Lambda Grids feature additional complexity such as wavelength granularity and continuity or conversion constraints must be enforced. Additionally, Grid resources cannot be expected to be available at all times. Therefore, the extra complexity of resilience against possible resource failures must be taken into account when modelling the combined Grid network dimensioning and workload scheduling problem, enforcing the need for scalable solution methods. In this work, we tackle the Lambda Grid combined dimensioning and workload scheduling problem and incorporate single-resource failure or unavailability scenarios. We use Divisible Load Theory to tackle the scalability problem and compare non-resilient lambda Grid dimensioning to the dimensions needed to survive single-resource failures. We distinguish three failure scenarios relevant to lambda Grid deployment: computational element, network link and optical cross-connect failure. Using regular network topologies, we derive analytical bounds on the dimensioning cost. To validate these bounds, we present comparisons for the resulting Grid dimensions assuming a 2-tier Grid operation as a function of varying wavelength granularity, fiber/wavelength cost models, traffic demand asymmetry and Grid scheduling strategy for a specific set of optical transport networks.     

Using a Simple Prioritisation Mechanism to Effectively Interoperate Service and Opportunistic Grids in the EELA-2 e-Infrastructure

Grids currently in production can be broadly classified as either service Grids, composed of dedicated resources, or opportunistic Grids that harvest the computing power of non-dedicated resources when they are idle. While a service Grid provides high and well defined levels of quality of service, an opportunistic Grid provides only a best-effort service. Nevertheless, since opportunistic Grids do not require resources to be fully dedicated to the Grid, they have the potential to assemble a much larger number of resources. Moreover, these Grids cater very well to the execution of the so-called embarrassingly parallel applications, a type of application that is frequently found in practice, and that comprises the largest portion of the typical workload processed in production Grid systems. The EELA-2 e-infrastructure is comprised of a service Grid and an opportunistic Grid that federates computing resources from scientific institutions in both Europe and Latin America. Due to the complementary characteristics of these two types of Grids, a lot of attention has recently been placed in how to interoperate them. In this paper we focus on the less studied problem of assessing the feasibility of such interoperation. We analyse different prioritisation policies that define when the resources of one Grid can be used to run jobs originating from the other. Our results show that in the absence of a suitable prioritisation policy, the benefits that the users of one Grid may have, frequently come with an important negative impact on the users of the other Grid. We also show that a simple reciprocation mechanism is capable of arbitrating the interoperation in such a way that, whenever possible, users profit from the interoperation and, in no case, this benefit leads to a noticeable reduction on the quality of service that the users would experience were the Grids not to interoperate. We conclude discussing how we have implemented, in the context of the EELA-2 project, this prioritisation mechanism, allowing the effective interoperation of a service Grid based on the gLite middleware with an opportunistic Grid that uses the OurGrid middleware.     

Achieving efficiency, quality of service and robustness in multi-organizational Grids

Scalability, flexibility, quality of service provisioning, efficiency and robustness are the desired characteristics of most computing systems. Although the emerging Grid computing paradigm is scalable and flexible, achieving both efficiency and quality of service provisioning in Grids is a challenging task but is necessary for the wide adoption of Grids. Grid middleware should also be robust to uncertainties such as those in user-estimated runtimes of Grid applications. In this paper, we present a complete middleware framework for Grids that achieves user satisfaction by providing QoS guarantees for Grid applications, cost effectiveness by efficiently utilizing resources and robustness by intelligently handling uncertain runtimes of applications.     

Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal

Computational Grids connect resources and users in a complex way in order to deliver nontrivial qualities of services. According to the current trend various communities build their own Grids and due to the lack of generally accepted standards these Grids are usually not interoperable. As a result, large scale sharing of resources is prevented by the isolation of Grid systems. Similarly, people are isolated, because the collaborative work of Grid users is not supported by current environments. Each user accesses Grids as an individual person without having the possibility of organizing teams that could overcome the difficulties of application development and execution more easily. The paper describes a new workflow-oriented portal concept that solves both problems. It enables the interoperability of various Grids during the execution of workflow applications, and supports users to develop and run their Grid workflows in a collaborative way. The paper also introduces a classification model that can be used to identify workflow-oriented Grid portals based on two general features: Ability to access multiple Grids, and support for collaborative problem solving. Using the approach the different potential portal types are introduced, their unique features are discussed and the portals and Problem Solving Environments (PSE) of our days are classified. The P-GRADE Portal as a Globus-based implementation for the classification model is also presented. The work described in this paper is supported by the Hungarian Grid project (IHM 4671/1/2003), by the Hungarian OTKA project (No. T042459) and a collaboration project with the University of Reading.