Using a Simple Prioritisation Mechanism to Effectively Interoperate Service and Opportunistic Grids in the EELA-2 e-Infrastructure

Grids currently in production can be broadly classified as either service Grids, composed of dedicated resources, or opportunistic Grids that harvest the computing power of non-dedicated resources when they are idle. While a service Grid provides high and well defined levels of quality of service, an opportunistic Grid provides only a best-effort service. Nevertheless, since opportunistic Grids do not require resources to be fully dedicated to the Grid, they have the potential to assemble a much larger number of resources. Moreover, these Grids cater very well to the execution of the so-called embarrassingly parallel applications, a type of application that is frequently found in practice, and that comprises the largest portion of the typical workload processed in production Grid systems. The EELA-2 e-infrastructure is comprised of a service Grid and an opportunistic Grid that federates computing resources from scientific institutions in both Europe and Latin America. Due to the complementary characteristics of these two types of Grids, a lot of attention has recently been placed in how to interoperate them. In this paper we focus on the less studied problem of assessing the feasibility of such interoperation. We analyse different prioritisation policies that define when the resources of one Grid can be used to run jobs originating from the other. Our results show that in the absence of a suitable prioritisation policy, the benefits that the users of one Grid may have, frequently come with an important negative impact on the users of the other Grid. We also show that a simple reciprocation mechanism is capable of arbitrating the interoperation in such a way that, whenever possible, users profit from the interoperation and, in no case, this benefit leads to a noticeable reduction on the quality of service that the users would experience were the Grids not to interoperate. We conclude discussing how we have implemented, in the context of the EELA-2 project, this prioritisation mechanism, allowing the effective interoperation of a service Grid based on the gLite middleware with an opportunistic Grid that uses the OurGrid middleware.     

Grid Economics: A Selective Discussion of Two Research Problems

The last 5 years have seen considerable discussion of various types of Grids—compute Grids, storage Grids, and data Grids. Using the checklist given in Foster (, 2002) to define a Grid, two important problems that arise in the context of resource sharing in Grid computing environments are discussed. First, the well documented problem in compute Grid environments that arises from the inability of consumers to accurately estimate their resource requirements is presented. This results in incorrect scheduling of requests for Grid resources and social welfare loss. To address this problem, two research proposals are briefly described. The first approach argues for the design of decision support tools to help users with resource estimation while the second approach studies the design of resource allocation mechanisms that can work with stochastic specifications of resource requirements. This is in contrast to the traditional point estimates of resource required by extant mechanisms. Next, resource provisioning and pricing problems that arise in data storage and retrieval Grids are described. These Grids differ fundamentally from compute Grids but share some economic characteristics with P2P file sharing networks. Drawing on this connection, pricing mechanisms and resource provisioning research is briefly discussed.     

Fault tolerant and prioritized scheduling in OGSA‐based mobile Grids

Grids and mobile Grids can form the basis and the enabling technology for pervasive and utility computing due to their ability to being open, highly heterogeneous and scalable. In this paper we present a scheme for advancing quality of service (QoS) attributes, such as fault tolerance and prioritized scheduling, in OGSA‐based mobile Grids. The fault tolerance is achieved by producing and managing sufficient replicas of tasks submitted for execution on the mobile Grid resources. We design a simple and efficient prioritization scheme, which allows the scheduling of the tasks submitted by the Grid users as distinguished priorities that can be managed and exploited as a QoS parameter by the Grid infrastructure operator. The results that are presented show the efficiency of the proposed scheme in being simple and additionally enriching with reliability and QoS features the applications that are built on the concept of mobile Grids. Copyright © 2008 John Wiley & Sons, Ltd.     

Towards supporting multiple virtual private computing environments on computational Grids

Grid computing now becomes a practical computing paradigm and solution for distributed systems and applications. Currently increasing resources are involved in Grid environments and a large number of applications are running on computational Grids. Unfortunately Grid computing technologies are still far away from reach of inexperienced application users, e.g., computational scientists and engineers. A software layer is required to provide an easy interface of Grids to end users.To meet this requirement HEAVEN (Hosting European Application Virtual ENvironment) upperware is proposed to build on top of Grid middleware. This paper presents HEAVEN philosophy of virtual computing for Grids – a combinational idea of simulation and emulation approaches. The concept of Virtual Private Computing Environment (VPCE) is thereafter proposed and defined. The design and current implementation of HEAVEN upperware are discussed in detail. Use case of Ag2D application justifies the philosophy of HEAVEN virtual computing methodology and the design/implementation of HEAVEN upperware.     

Flexible Grid service management through resource partitioning

In this paper, a distributed and scalable Grid service management architecture is presented. The proposed architecture is capable of monitoring task submission behaviour and deriving Grid service class characteristics, for use in performing automated computational, storage and network resource-to-service partitioning. This partitioning of Grid resources amongst service classes (each service class is assigned exclusive usage of a distinct subset of the available Grid resources), along with the dynamic deployment of Grid management components dedicated and tuned to the requirements of a particular service class introduces the concept of Virtual Private Grids. We present two distinct algorithmic approaches for the resource partitioning problem, the first based on Divisible Load Theory (DLT) and the second built on Genetic Algorithms (GA). The advantages and drawbacks of each approach are discussed and their performance is evaluated on a sample Grid topology using NSGrid, an ns-2 based Grid simulator. Results show that the use of this Service Management Architecture in combination with the proposed algorithms improves computational and network resource efficiency, simplifies schedule making decisions, reduces the overall complexity of managing the Grid system, and at the same time improves Grid QoS support (with regard to job response times) by automatically assigning Grid resources to the different service classes prior to scheduling.     

Workflow Concepts of the Java CoG Kit

Many scientific simulations and experiments require the coordination of numerous tasks posed by interdisciplinary research teams. Grids can provide access to the necessary high-end resources to conduct such tasks. The complex tasks and their interactions must be supported through convenient tools. To address this issue, we introduce a number of Grid abstractions that make the development of Grid middleware-independent tools possible and allow for the integration of a number of commodity tools. Our vision is implemented through an integrated approach based on a layered architecture that bridges the gap between Grid middleware and scientific applications. Our abstractions include specialized services, a Grid workflow engine and language, and Gridfaces – graphical abstractions that can be employed in science portals and standalone applications.     

GridBLAST: a Globus‐based high‐throughput implementation of BLAST in a Grid computing framework

Improvements in the performance of processors and networks have made it feasible to treat collections of workstations, servers, clusters and supercomputers as integrated computing resources or Grids. However, the very heterogeneity that is the strength of computational and data Grids can also make application development for such an environment extremely difficult. Application development in a Grid computing environment faces significant challenges in the form of problem granularity, latency and bandwidth issues as well as job scheduling. Currently existing Grid technologies limit the development of Grid applications to certain classes, namely, embarrassingly parallel, hierarchical parallelism, work flow and database applications. Of all these classes, embarrassingly parallel applications are the easiest to develop in a Grid computing framework. The work presented here deals with creating a Grid‐enabled, high‐throughput, standalone version of a bioinformatics application, BLAST, using Globus as the Grid middleware. BLAST is a sequence alignment and search technique that is embarrassingly parallel in nature and thus amenable to adaptation to a Grid environment. A detailed methodology for creating the Grid‐enabled application is presented, which can be used as a template for the development of similar applications. The application has been tested on a ‘mini‐Grid’ testbed and the results presented here show that for large problem sizes, a distributed, Grid‐enabled version can help in significantly reducing execution times. Copyright © 2005 John Wiley & Sons, Ltd.     

Autonomic QoS control in enterprise Grid environments using online simulation

As Grid Computing increasingly enters the commercial domain, performance and quality of service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. This paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.     

Grid programming: some indications where we are headed

Grid computing enables the development of large scientific applications on an unprecedented scale. Grid-aware applications, also called meta-applications or multi-disciplinary applications, make use of coupled computational resources that are not available at a single site. In this light, the Grids let scientists solve larger or new problems by pooling together resources that could not be coupled easily before. It is well known that the programmer’s productivity in designing and implementing efficient distributed/parallel applications on high-performance computers is still usually a very time-consuming task. Grid computing makes the situation worse. Consequently, the development of Grid programming environments that would enable programmers to efficiently exploit this technology is an important and hot research issue.

After an introduction on the main Grid programming issues, this paper will review the most important approaches/projects conducted in this field worldwide.     

Introduce: An Open Source Toolkit for Rapid Development of Strongly Typed Grid Services

Service-oriented architectures and applications have gained wide acceptance in the Grid computing community. A number of tools and middleware systems have been developed to support application development using Grid Services architectures. Most of these efforts, however, have focused on low-level support for management and execution of Grid services, management of Grid-enabled resources, and deployment and execution of applications that make use of Grid services. Simple-to-use service development tools, which would allow a Grid service developer to leverage Grid technologies without needing to know low-level details, are becoming increasingly important for wider application of the Grid. In this paper, we describe an open-source, extensible toolkit, called Introduce, that supports easy development and deployment of Web Services Resource Framework (WSRF) compliant services. Introduce is designed to reduce the service development and deployment effort by hiding low level details of the Globus Toolkit and to enable the implementation of strongly typed services. In strongly typed services, a service produces and consumes data types that are well-defined and published in the Grid. This enables data-level syntactic interoperability so that clients and services can access and consume data elements programmatically and correctly. We expect that enabling strongly typed Grid services while lowering the difficulty of entry to the Grid via toolkits like Introduce will have a major impact to the success of the Grid and its wider adoption as a viable technology of choice in the commercial sector as well as in academic, medical, and government research.     

Grid Computing in China

Grid computing presents a new trend to distributed computation and Internet applications, which can construct a virtual single image of heterogeneous resources, provide uniform application interface and integrate widespread computational resources into super, ubiquitous and transparent aggregation. In the adoption of Grid computing, China, who is facing more resource heterogeneity and other specific demands, has put much effort on both research and practical utilization. In this paper, we introduce the major China Grid research projects and their perspective applications. First we give the overview of the four government-sponsored programs in Grid, namely the China National Grid, ChinaGrid, NSFC Grid, and ShanghaiGrid. Then we present six representative ongoing Grid systems in details, which are categorized into Grid middleware and Grid application. This paper provides the general picture of Grid computing in China, and shows the great efforts, devotion and confidence in China to use Grid technology to boost the society, economics and scientific research.     

A Taxonomy of Workflow Management Systems for Grid Computing

With the advent of Grid and application technologies, scientists and engineers are building more and more complex applications to manage and process large data sets, and execute scientific experiments on distributed resources. Such application scenarios require means for composing and executing complex workflows. Therefore, many efforts have been made towards the development of workflow management systems for Grid computing. In this paper, we propose a taxonomy that characterizes and classifies various approaches for building and executing workflows on Grids. We also survey several representative Grid workflow systems developed by various projects world-wide to demonstrate the comprehensiveness of the taxonomy. The taxonomy not only highlights the design and engineering similarities and differences of state-of-the-art in Grid workflow systems, but also identifies the areas that need further research.     

The catallaxy approach for decentralized economic-based allocation in Grid resource and service markets

Efficient resource allocation in dynamic large-scale environments is one of the challenges of Grids. In centralized economic-based allocation approaches, the user requests can be matched to the fastest, cheapest or most available resource. This approach, however, shows limitations in scalability and in dynamic environments. In this paper, we explore a decentralized economic approach for resource allocation in Grid markets based on the Catallaxy paradigm. Catallactic agents discover selling nodes in the resource and service Grid markets, and negotiate with each other maximizing their utility by following a strategy. By means of simulations, we evaluate the behavior of the approach, its resource allocation efficiency and its performance with different demand loads in a number of Grid density and dynamic environments. Our results indicate that while the decentralized economic approach based on Catallaxy applied to Grid markets shows similar efficiency to a centralized system, its decentralized operation provides greater advantages: scalability to demand and offer, and robustness in dynamic environments.     

Modeling and Supporting Grid Scheduling

Grid resource management systems and schedulers are important components for building Grids. They are responsible for the selection and allocation of Grid resources to current and future applications. Thus, they are important building blocks for making Grids available to user communities. In this paper we briefly analyze the requirements of Grid resource management and provide a classification of schedulers. Then, we define an extensible formal model for Grid scheduling activities, and characterize the general Grid scheduling problem. Finally, we provide a reference architecture for the support of our model and discuss different aspects of architectural implementations.     

GMarte: Grid middleware to abstract remote task execution

Grid computing technologies are now being largely deployed with the widespread adoption of the Globus Toolkit as the industrial standard Grid middleware. However, its inherent steep learning curve discourages the use of these technologies for non‐experts. Therefore, to increase the use of Grid computing, it is important to have high‐level tools that simplify the process of remote task execution. In this paper we introduce a middleware, developed on top of the Java Commodity Grid, which offers an object‐oriented, user‐friendly application programming interface, from the Java language, which eases remote task execution for computationally intensive applications. Copyright © 2006 John Wiley & Sons, Ltd.     

Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal

Computational Grids connect resources and users in a complex way in order to deliver nontrivial qualities of services. According to the current trend various communities build their own Grids and due to the lack of generally accepted standards these Grids are usually not interoperable. As a result, large scale sharing of resources is prevented by the isolation of Grid systems. Similarly, people are isolated, because the collaborative work of Grid users is not supported by current environments. Each user accesses Grids as an individual person without having the possibility of organizing teams that could overcome the difficulties of application development and execution more easily. The paper describes a new workflow-oriented portal concept that solves both problems. It enables the interoperability of various Grids during the execution of workflow applications, and supports users to develop and run their Grid workflows in a collaborative way. The paper also introduces a classification model that can be used to identify workflow-oriented Grid portals based on two general features: Ability to access multiple Grids, and support for collaborative problem solving. Using the approach the different potential portal types are introduced, their unique features are discussed and the portals and Problem Solving Environments (PSE) of our days are classified. The P-GRADE Portal as a Globus-based implementation for the classification model is also presented. The work described in this paper is supported by the Hungarian Grid project (IHM 4671/1/2003), by the Hungarian OTKA project (No. T042459) and a collaboration project with the University of Reading.     

Grid-Controlled Lightpaths for High Performance Grid Applications

Grid applications call for high performance networking support. One attractive solution is to deploy Grids over optical networks. However, resource management in optical domains is traditionally very rigid and cannot successfully meet the requirements of Grid applications, such as flexible provisioning and configuration. In this paper, we present a customizable resource management solution for optical networks where users can create lightpaths on demand and manage their own network resources. Thanks to a Grid-centric system architecture, lightpath resources can be shared among users and easily integrated with data and computation Grids.     

ICENI: Optimisation of component applications within a Grid environment

Effective exploitation of Computational Grids can only be achieved when applications are fully integrated with the Grid middleware and the underlying computational resources. Fundamental to this exploitation is information. Information about the structure and behaviour of the application, the capability of the computational and networking resources, and the availability and access to these resources by an individual, a group or an organisation.

In this paper we describe Imperial College e-Science Networked Infrastructure (ICENI), a Grid middleware framework developed within the London e-Science Centre. ICENI is a platform-independent framework that uses open and extensible XML derived protocols, within a framework built using Java and Jini, to explore effective application execution upon distributed federated resources. We match a high-level application specification, defined as a network of components, to an optimal combination of the currently available component implementations within our Grid environment, by using composite performance models. We demonstrate the effectiveness of this architecture through the high-level specification and solution of a set of linear equations by automatic and selection of optimal resources and implementations.     

Performance analysis of allocation policies for interGrid resource provisioning

Several Grids have been established and used for varying science applications during the last years. Most of these Grids, however, work in isolation and with different utilisation levels. Previous work has introduced an architecture and a mechanism to enable resource sharing amongst Grids. It has demonstrated that there can be benefits for a Grid to offload requests or provide spare resources to another Grid. In this work, we address the problem of resource provisioning to Grid applications in multiple-Grid environments. The provisioning is carried out based on availability information obtained from queueing-based resource management systems deployed at the provider sites which are the participants of the Grids. We evaluate the performance of different allocation policies. In contrast to existing work on load sharing across Grids, the policies described here take into account the local load of resource providers, imprecise availability information and the compensation of providers for the resources offered to the Grid. In addition, we evaluate these policies along with a mechanism that allows resource sharing amongst Grids. Experimental results obtained through simulation show that the mechanism and policies are effective in redirecting requests thus improving the applications’ average weighted response time.