Achieving Interoperation of Grid Data Resources via Workflow Level Integration

Production Grids are becoming widely utilized by the e-Science community to run computation and data intensive experiments more efficiently. Unfortunately, different production Grid infrastructures are based on different middleware technologies, both for computation and for data access. Although there is significant effort from the Grid community to standardize the underlying middleware, solutions that allow existing non-standard tools to interoperate are one of the major concerns of Grid users today. This paper describes the generic requirements towards the interoperation of Grid data resources within computational workflows, and suggests integration techniques that allow workflow engines to access various heterogeneous data resources during workflow execution. Reference implementations of these techniques are presented and recommendations on their applicability and suitability are made.     

Achieving efficiency, quality of service and robustness in multi-organizational Grids

Scalability, flexibility, quality of service provisioning, efficiency and robustness are the desired characteristics of most computing systems. Although the emerging Grid computing paradigm is scalable and flexible, achieving both efficiency and quality of service provisioning in Grids is a challenging task but is necessary for the wide adoption of Grids. Grid middleware should also be robust to uncertainties such as those in user-estimated runtimes of Grid applications. In this paper, we present a complete middleware framework for Grids that achieves user satisfaction by providing QoS guarantees for Grid applications, cost effectiveness by efficiently utilizing resources and robustness by intelligently handling uncertain runtimes of applications.     

Provide Virtual Distributed Environments for Grid computing on demand

Grid users always expect to meet some challenges to employ Grid resources, such as customized computing environment and QoS support. In this paper, we propose a new methodology for Grid computing – to use virtual machines as computing resources and provide Virtual Distributed Environments (VDE) for Grid users. It is declared that employing virtual environment for Grid computing can bring various advantages, for instance, computing environment customization, QoS guarantee and easy management. A light weight Grid middleware, Grid Virtualization Engine, is developed accordingly to provide functions of building virtual environment for Grids. We also present a typical use case, on-demand build a virtual e-Science infrastructure to justify the methodology.     

Towards supporting multiple virtual private computing environments on computational Grids

Grid computing now becomes a practical computing paradigm and solution for distributed systems and applications. Currently increasing resources are involved in Grid environments and a large number of applications are running on computational Grids. Unfortunately Grid computing technologies are still far away from reach of inexperienced application users, e.g., computational scientists and engineers. A software layer is required to provide an easy interface of Grids to end users.To meet this requirement HEAVEN (Hosting European Application Virtual ENvironment) upperware is proposed to build on top of Grid middleware. This paper presents HEAVEN philosophy of virtual computing for Grids – a combinational idea of simulation and emulation approaches. The concept of Virtual Private Computing Environment (VPCE) is thereafter proposed and defined. The design and current implementation of HEAVEN upperware are discussed in detail. Use case of Ag2D application justifies the philosophy of HEAVEN virtual computing methodology and the design/implementation of HEAVEN upperware.     

An Interoperable Grid Information System for Integrated Resource Monitoring Based on Virtual Organizations

In many Grid infrastructures different kinds of information services are in use, which utilize different incompatible data structures and interfaces to encode and provide their data. Homogeneous monitoring of these infrastructures with the monitoring data being accessible everywhere independently of the middleware which provided it, is the basis for a consistent status reporting on the Grids’ resources and services. Thus, interoperability or interoperation between the different information services in a heterogeneous Grid infrastructure is required. Monitoring data must contain the identity of the affected Virtual Organization (VO) so that it can be related to the resources and services the VO has allocated to enable VO-specific information provision. This paper describes a distributed architecture for an interoperable information service, which combines data unification and categorization with policies for VO membership, VO resource management and data transformations. This service builds the basis for an integrated and interoperating monitoring of Grids, which provide their data to more than one VO and utilize heterogeneous information services.     

On-demand service hosting on production grid infrastructures

The Software as a Service (SaaS) methodology is a key paradigm of Cloud computing. In this paper, we focus on an interesting topic—to dynamically host services on existing production Grid infrastructures. In general, production Grids normally employ a Job-Submission-Execution (JSE) model with rigid access interfaces. In this paper, we implement the Cyberaide onServe, a lightweight middleware with a virtual appliance. The Cyberaide onServe implements the SaaS model on production Grids by translating the SaaS model to the JSE model. The Cyberaide onServe can be deployed on demand in a virtual appliance, host users’ software as a Web service, accept Web service invocations; finally, the Cyberaide onServe can execute them on production Grids. We have deployed the Cyberaide onServe on the TeraGrid and the test results show that the Cyberaide onServe can provide SaaS functionalities with a good performance.     

A standards-based interoperable single sign-on framework in ARC Grid middleware

Security infrastructure is one of the most challenging tasks in the development, integration and deployment of Grid middlewares. Even though the Grid community addresses the security issue through public key infrastructures (PKI) to support mutual authentication using X.509 certificates, maintaining X.509 credentials is not that easy for non-IT-experts, and has proved to be an obstacle for a more wide deployment of Grid technologies. The identity federation is an increasingly popular technology that can facilitate cross-domain single sign-on without requiring the users to maintain any credentials additional to their own institutional accounts. We believe that utilizing identity federation for Grid middlewares is a promising path for the Grid technology to get more widely used. This paper describes a single sign-on infrastructure developed as a part of the NorduGrid ARC (Advanced Resource Connector) Grid middleware. It adopts the identity federation standard (SAML), as well as other Web Service standards. It focuses on a single sign-on solution at the middleware level for users to access Grids by only using their frequently used accounts, without being bothered to maintain X.509 credentials. Users can use their username/password only to access Grids developed in ARC middleware, as well as access Grids developed in other middlewares that requires users to provide X.509 certificates. Moreover, the single sign-on for workflow-like Grid applications (in which intermediate entities act on behalf of users) is also supported. As an important aspect of single sign-on, authorization is also considered by implementing an attribute-based authorization using SAML standard. In addition, the performance of single sign-on solution is measured. We identify performance limitations of security-related services inside this solution, and analyse the ways to avoid the limitations. To our knowledge, the work presented in this paper is the first evaluated implementation that utilizes identity federation for Grid usage on the middleware level.     

Mobile agent middleware for mobile computing

Mobile computing requires an advanced infrastructure that integrates suitable support protocols, mechanisms, and tools. This mobility middleware should dynamically reallocate and trace mobile users and terminals and permit communication and coordination of mobile entities. In addition, open and untrusted environments must overcome system heterogeneity and grant the appropriate security level. Solutions to these issues require compliance with standards to interoperate with different systems and legacy components and a reliable security infrastructure based on standard cryptographic mechanisms and tools. Many proposals suggest using mobile agent technology middleware to address these issues. A mobile agent moves entities in execution together with code and achieved state, making it possible to upgrade distributed computing environments without suspending service. We propose three mobile computing services: user virtual environment (UVE), mobile virtual terminal (MVT), and virtual resource management (VRM). UVE provides users with a uniform view of their working environments independent of current locations and specific terminals. MVT extends traditional terminal mobility by preserving the terminal execution state for restoration at new locations, including active processes and subscribed services. VRM permits mobile users and terminals to maintain access to resources and services by automatically requalifying the bindings and moving specific resources or services to permit load balancing and replication     

Coordinated harnessing of the IRISGrid and EGEE testbeds with GridWay

Since the late 1990s, we have witnessed an extraordinary development of Grid technologies. Nowadays, different Grids are being deployed within the context of a growing number of national and transnational research projects. However, the coexistence of those different infrastructures involves two challenging issues, namely: (i) simultaneous and coordinated use of resources from different Grids, from the end user perspective; and (ii) the simultaneous contribution of resources to different Grids, from the resource owner perspective. In this paper, we demonstrate that a decentralized and “end-to-end” scheduling and execution system can efficiently interoperate different Grids. In particular, we evaluate the coordinated use of the EGEE and IRISGrid testbeds in the execution of a Bioinformatics application. Results show the feasibility of building loosely coupled computational Grid environments only based on Globus services, while obtaining non-trivial levels of quality of service, in terms of performance and reliability. Such approach allows a straightforward resource sharing since the resources are accessed by using de facto standard protocols and interfaces.     

Fault tolerant and prioritized scheduling in OGSA‐based mobile Grids

Grids and mobile Grids can form the basis and the enabling technology for pervasive and utility computing due to their ability to being open, highly heterogeneous and scalable. In this paper we present a scheme for advancing quality of service (QoS) attributes, such as fault tolerance and prioritized scheduling, in OGSA‐based mobile Grids. The fault tolerance is achieved by producing and managing sufficient replicas of tasks submitted for execution on the mobile Grid resources. We design a simple and efficient prioritization scheme, which allows the scheduling of the tasks submitted by the Grid users as distinguished priorities that can be managed and exploited as a QoS parameter by the Grid infrastructure operator. The results that are presented show the efficiency of the proposed scheme in being simple and additionally enriching with reliability and QoS features the applications that are built on the concept of mobile Grids. Copyright © 2008 John Wiley & Sons, Ltd.     

Validating and Scaling the MicroGrid: A Scientific Instrument for Grid Dynamics

Large-scale Grids that aggregate and share resources over wide-area networks present major challenges in understanding dynamic application and resource behavior for performance, stability, and reliability. Accurate study of the dynamic behavior of applications, middleware, resources, and networks depends on coordinated and accurate modeling of all four of these elements simultaneously. We have designed and implemented a tool called the MicroGrid which enables accurate and comprehensive study of the dynamic interaction of applications, middleware, resource, and networks. The MicroGrid creates a virtual Grid environment – accurately modeling networks, resources, the information services (resource and network metadata) transparently. Thus, the MicroGrid enables users, Grid researchers, or Grid operators to study arbitrary collections of resources and networks. The MicroGrid includes the MaSSF online network simulator which provides packet-level accurate, but scalable network modeling. We present experimental results with applications which validate the implementation of the MicroGrid, showing that it not only runs real Grid applications and middleware, but that it accurately models both their and underlying resource and network behavior. We also study a range of techniques for scaling a critical part of the online network simulator to the simulation of large networks. These techniques employ a sophisticated graph partitioner, and a range of edge and node weighting schemes exploiting a range of static network and dynamic application information. The best of these, profile-driven placement, scales well to online simulation of large networks of 6,000 nodes using 24 simulation engine nodes.     

Service level agreement based adaptive Grid superscheduling

Grid computing brings heterogeneity and decentralization to the world of science and technology. It leverages every bit of idle computing resources and provides a straightforward middleware for integrating cross-domain scientific devices and legacy systems. In a super big Grid, job scheduling is challenging specifically when it needs to have access to vast amount of resources. The process of mapping jobs onto Grid resources requires significant consideration in terms of Grid architecture design, consumer demands and provider revenues. In this paper, we simultaneously utilize the legacy architecture of superscheduling, forwarding strategy, service level, success rate, and service pricing strategies and finally propose a service level agreement based on adaptive superscheduling (SAS) algorithm. SAS algorithm presents unified connectivity via efficient diffusion of jobs through the Grid infrastructure that is fueled from the previous scheduling events across the Grid. Moreover, by enforcing the service level agreement terms from a rich set of ask and bid prices, system performance, and load statistics, SAS successfully boosts revenue and utilization statistics. We perform an extensive experimental analysis for different Grid scales. Based on our experimental result, the SAS algorithm maximizes revenue while guarantees quality of service. More specifically, the quality of service is achieved through a high ratio of completed jobs and remarkable utilization of resources.     

Virtual workflow system for distributed collaborative scientific applications on Grids

Grid computing has become an effective computing technique in recent years. This paper develops a virtual workflow system to construct distributed collaborative applications for Grid users. The virtual workflow system consists three levels: abstract workflow system, translator and concrete workflow system. The research highlight of the implementation is that this workflow system is developed based on CORBA and Unicore Grid middleware. Furthermore, this implementation can support legacy application developed with Parco and C++ codes. This virtual workflow system can provide efficient GUI for users to organize distributed scientific collaborative applications and execute them on Grid resources. We present the design, implementation, and evaluation of this virtual workflow system in the paper.     

EMPEROR: An OGSA Grid Meta-Scheduler Based on Dynamic Resource Predictions

Scheduling constitutes an integral feature of Grid computing infrastructures, being also a key to realizing several of the Grid promises. In particular, scheduling can maximize the resources available to end users, accelerate the execution of jobs, while also supporting scalable and autonomic management of the resources comprising a Grid. Grid scheduling functionality hinges on middleware components called meta-schedulers, which undertake to automatically distribute jobs across the dispersed heterogeneous resources of a Grid. In this paper we present the design and implementation of a Grid meta-scheduler, which we call EMPEROR. EMPEROR provides a framework for implementing scheduling algorithms based on performance criteria. In implementing a particular instantiation of this framework, we have devised models for predicting host load and memory resources, and accordingly for estimating the running time of a task. These models hinge on time series analysis techniques and take into account results of the cluster computing literature. Apart from incorporating these models, EMPEROR provides fully fledged Grid scheduling functionality, which complies with OGSA standards as the later are reflected in the Globus toolkit. Specifically, EMPEROR interfaces to Globus middleware services (i.e., GSI, MDS, GRAM) towards discovering resources, implementing the scheduling algorithm and ultimately submitting jobs to local scheduling systems. By and large, EMPEROR is one of the few standards based meta-schedulers making use of dynamic scheduling information.     

Flexible Grid service management through resource partitioning

In this paper, a distributed and scalable Grid service management architecture is presented. The proposed architecture is capable of monitoring task submission behaviour and deriving Grid service class characteristics, for use in performing automated computational, storage and network resource-to-service partitioning. This partitioning of Grid resources amongst service classes (each service class is assigned exclusive usage of a distinct subset of the available Grid resources), along with the dynamic deployment of Grid management components dedicated and tuned to the requirements of a particular service class introduces the concept of Virtual Private Grids. We present two distinct algorithmic approaches for the resource partitioning problem, the first based on Divisible Load Theory (DLT) and the second built on Genetic Algorithms (GA). The advantages and drawbacks of each approach are discussed and their performance is evaluated on a sample Grid topology using NSGrid, an ns-2 based Grid simulator. Results show that the use of this Service Management Architecture in combination with the proposed algorithms improves computational and network resource efficiency, simplifies schedule making decisions, reduces the overall complexity of managing the Grid system, and at the same time improves Grid QoS support (with regard to job response times) by automatically assigning Grid resources to the different service classes prior to scheduling.     

Scalable dimensioning of resilient Lambda Grids

Grids consist of the aggregation of numerous dispersed computational, storage and network resources, able to satisfy even the most demanding computing jobs. Due to the data-intensive nature of Grid jobs, there is an increasing interest in Grids using optical transport networks as this technology allows for the timely delivery of large amounts of data. Such Grids are commonly referred to as Lambda Grids.

An important aspect of Grid deployment is the allocation and activation of installed network capacity, needed to transfer data and jobs to and from remote resources. However, the exact nature of a Grid’s network traffic depends on the way arriving workload is scheduled over the various Grid sites. As Grids possibly feature high numbers of resources, jobs and users, solving the combined Grid network dimensioning and workload scheduling problem requires the use of scalable mathematical methods such as Divisible Load Theory (DLT). Lambda Grids feature additional complexity such as wavelength granularity and continuity or conversion constraints must be enforced. Additionally, Grid resources cannot be expected to be available at all times. Therefore, the extra complexity of resilience against possible resource failures must be taken into account when modelling the combined Grid network dimensioning and workload scheduling problem, enforcing the need for scalable solution methods. In this work, we tackle the Lambda Grid combined dimensioning and workload scheduling problem and incorporate single-resource failure or unavailability scenarios. We use Divisible Load Theory to tackle the scalability problem and compare non-resilient lambda Grid dimensioning to the dimensions needed to survive single-resource failures. We distinguish three failure scenarios relevant to lambda Grid deployment: computational element, network link and optical cross-connect failure. Using regular network topologies, we derive analytical bounds on the dimensioning cost. To validate these bounds, we present comparisons for the resulting Grid dimensions assuming a 2-tier Grid operation as a function of varying wavelength granularity, fiber/wavelength cost models, traffic demand asymmetry and Grid scheduling strategy for a specific set of optical transport networks.     

Reputation-based dependable scheduling of workflow applications in Peer-to-Peer Grids

Grids facilitate creation of wide-area collaborative environment for sharing computing or storage resources and various applications. Inter-connecting distributed Grid sites through peer-to-peer routing and information dissemination structure (also known as Peer-to-Peer Grids) is essential to avoid the problems of scheduling efficiency bottleneck and single point of failure in the centralized or hierarchical scheduling approaches. On the other hand, uncertainty and unreliability are facts in distributed infrastructures such as Peer-to-Peer Grids, which are triggered by multiple factors including scale, dynamism, failures, and incomplete global knowledge.In this paper, a reputation-based Grid workflow scheduling technique is proposed to counter the effect of inherent unreliability and temporal characteristics of computing resources in large scale, decentralized Peer-to-Peer Grid environments. The proposed approach builds upon structured peer-to-peer indexing and networking techniques to create a scalable wide-area overlay of Grid sites for supporting dependable scheduling of applications. The scheduling algorithm considers reliability of a Grid resource as a statistical property, which is globally computed in the decentralized Grid overlay based on dynamic feedbacks or reputation scores assigned by individual service consumers mediated via Grid resource brokers. The proposed algorithm dynamically adapts to changing resource conditions and offers significant performance gains as compared to traditional approaches in the event of unsuccessful job execution or resource failure. The results evaluated through an extensive trace driven simulation show that our scheduling technique can reduce the makespan up to 50% and successfully isolate the failure-prone resources from the system.     

Grid Deployment Experiences: Grid Interoperation

Over recent years a number of Grid projects have emerged which have built Grid infrastructures that are now the computing backbones for various user communities. A significant number of these communities are limited to one Grid Infrastructure due to the different middleware and operations procedures used. Grid Interoperation is trying to bridge these differences and enable Virtual Organizations to access resources independent of the Grid project affiliation. Building upon the experiences the authors have gained while working on interoperation between EGEE and various other Grid infrastructures as well as through co-chairing the Grid Interoperation Now (GIN) efforts of the Open Grid Forum (OGF), this paper gives an overview of Grid Interoperation and describes various methods that can be used to connect Grid Infrastructures. The case is made for standardization in key areas and why the Grid community should move more aggressively towards standards.     

Online algorithms for advance resource reservations

We consider the problem of providing QoS guarantees to Grid users through advance reservation of resources. Advance reservation mechanisms provide the ability to allocate resources to users based on agreed-upon QoS requirements and increase the predictability of a Grid system, yet incorporating such mechanisms into current Grid environments has proven to be a challenging task due to the resulting resource fragmentation. We use concepts from computational geometry to present a framework for tackling the resource fragmentation, and for formulating a suite of scheduling strategies. We also develop efficient implementations of the scheduling algorithms that scale to large Grids. We conduct a comprehensive performance evaluation study using simulation, and we present numerical results to demonstrate that our strategies perform well across several metrics that reflect both user- and system-specific goals. Our main contribution is a timely, practical, and efficient solution to the problem of scheduling resources in emerging on-demand computing environments.     

A software tool for semi-automatic gridification of resource-intensive Java bytecodes and its application to ray tracing and sequence alignment

Computational Grids deliver the necessary computational infrastructure to perform resource-intensive computations such as the ones that solve the problems scientists are facing today. Exploiting Computational Grids comes at the expense of explicitly adapting the ordinary software implementing scientific problems to take advantage of Grid resources, which unavoidably requires knowledge on Grid programming. The recent notion of “gridifying” ordinary applications, which is based on semi-automatically deriving a Grid-aware version from the compiled code of a sequential application, promises users to be relieved from the requirement of manual usage of Grid APIs within their source codes. In this paper, we describe a novel gridification tool that allows users to easily parallelize Java applications on Grids. Extensive experiments with two real-world applications - ray tracing and sequence alignment - suggest that our approach provides a convenient balance between ease of gridification and Grid resource exploitation compared to manually using Grid APIs for gridifying ordinary applications.