Scalable Grid‐wide capacity allocation with the SweGrid Accounting System (SGAS)

The SweGrid Accounting System (SGAS) allocates capacity in collaborative Grid environments by coordinating enforcement of Grid‐wide usage limits as a means to offer usage guarantees and prevent overuse. SGAS employs a credit‐based allocation model where Grid capacity is granted to projects via Grid‐wide quota allowances that can be spent across the Grid resources. The resources collectively enforce these allowances in a soft, real‐time manner. SGAS is built on service‐oriented principles with a strong focus on interoperability and Web services standards. This article covers the SGAS design and implementation, which, besides addressing inherent Grid challenges (scale, security, heterogeneity, decentralization), emphasizes generality and flexibility to produce a customizable system with lightweight integration into different middleware and scheduling system combinations. We focus the discussion around the system design, a flexible allocation model, middleware integration experiences and scalability improvements via a distributed virtual banking system, and finally, an extensive set of testbed experiments. The experiments evaluate the performance of SGAS in terms of response times, request throughput, overall system scalability, and its performance impact on the Globus Toolkit 4 job submission software. We conclude that, for all practical purposes, the quota enforcement overhead incurred by SGAS on job submissions is not a limiting factor for the job‐handling capacity of the job submission software. Copyright © 2008 John Wiley & Sons, Ltd.     

Grid resource brokering algorithms enabling advance reservations and resource selection based on performance predictions

We present algorithms, methods, and software for a Grid resource manager, that performs resource brokering and job scheduling in production Grids. This decentralized broker selects computational resources based on actual job requirements, job characteristics, and information provided by the resources, with the aim to minimize the total time to delivery for the individual application. The total time to delivery includes the time for program execution, batch queue waiting, and transfer of executable and input/output data to and from the resource. The main features of the resource broker include two alternative approaches to advance reservations, resource selection algorithms based on computer benchmark results and network performance predictions, and a basic adaptation facility. The broker is implemented as a built-in component of a job submission client for the NorduGrid/ARC middleware.     

Ontology‐based Grid resource management

Grid resources are typically diverse in nature with respect to their software and hardware configurations, resource usage policies and the kind of application they support. Aggregating and monitoring these resources, and discovering suitable resources for the applications become a challenging issue. This is partially due to the representation of Grid metadata supported by the existing Grid middleware which offers limited scope for matching the job requirements that directly affect scheduling decisions. This paper proposes a semantic component in conventional Grid architecture to support ontology‐based representation of Grid metadata and facilitate context‐based information retrieval that complements Grid schedulers for effective resource management. Web Ontology language is used for creating Grid resource ontology and Algernon inference engine has been used for resource discovery. This semantic component has been integrated with conventional Grid schedulers. Several experiments have also been carried out to investigate the performance overhead that arises while integrating this component with Grid schedulers. Copyright © 2009 John Wiley & Sons, Ltd.     

Flexible Grid service management through resource partitioning

In this paper, a distributed and scalable Grid service management architecture is presented. The proposed architecture is capable of monitoring task submission behaviour and deriving Grid service class characteristics, for use in performing automated computational, storage and network resource-to-service partitioning. This partitioning of Grid resources amongst service classes (each service class is assigned exclusive usage of a distinct subset of the available Grid resources), along with the dynamic deployment of Grid management components dedicated and tuned to the requirements of a particular service class introduces the concept of Virtual Private Grids. We present two distinct algorithmic approaches for the resource partitioning problem, the first based on Divisible Load Theory (DLT) and the second built on Genetic Algorithms (GA). The advantages and drawbacks of each approach are discussed and their performance is evaluated on a sample Grid topology using NSGrid, an ns-2 based Grid simulator. Results show that the use of this Service Management Architecture in combination with the proposed algorithms improves computational and network resource efficiency, simplifies schedule making decisions, reduces the overall complexity of managing the Grid system, and at the same time improves Grid QoS support (with regard to job response times) by automatically assigning Grid resources to the different service classes prior to scheduling.     

Standards-Based Job Management in Grid Systems

The Grid paradigm for accessing heterogeneous distributed resources proved to be extremely effective, as many organizations are relying on Grid middlewares for their computational needs. Many different middlewares exist, the result being a proliferation of self-contained, non interoperable “Grid islands”. This means that different Grids, based on different middlewares, cannot share resources, e.g. jobs submitted on one Grid cannot be forwarded for execution on another one. To address this problem, standard interfaces are being proposed for some of the important functionalities provided by most Grids, namely job submission and management, authorization and authentication, resource modeling, and others. In this paper we review some recent standards which address interoperability for three types of services: the BES/JSDL specifications for job submission and management, the SAML notation for authorization and authentication, and the GLUE specification for resource modeling. We describe how standards-enhanced Grid components can be used to create interoperable building blocks for a Grid architecture. Furthermore, we describe how existing components from the gLite middleware have been re-engineered to support BES/JSDL, GLUE and SAML. From this experience we draw some conclusions on the strengths and weaknesses of these specifications, and how they can be improved.     

A Grid resource brokering strategy based on resource and network performance in Grid

To achieve high performance distributed data access and computing in Grid environment, monitoring of resource and network performance is vital. Our proposed Grid network monitoring architecture is modeled by the Grid scheduler. The proposed Grid network monitoring retrieves network metrics using sensors as network monitoring tools. The mobile agents are migrated to start the sensors to measure the network metrics in all Grid Resources from the Resource Broker. The raw data provided by the monitoring tools is used to produce a high level view of the Grid through the set of internal cost functions. The network cost function is formed by combining various network metrics such as bandwidth, Round Trip Time, jitter and packet loss to measure the network performance. This paper presents the Grid Resource Brokering strategy which analyzes the network metrics along with the resource metrics for the selection of the Grid resource to submit the job and the proposed approach is integrated with CARE Resource Broker (CRB) for job submission. The experimental results are evident for the minimization of job completion time for the submitted job. The simulation results also prove that the more number of jobs are completed with the proposed strategy which influences the better utilization of the Grid resources.     

A standards-based interoperable single sign-on framework in ARC Grid middleware

Security infrastructure is one of the most challenging tasks in the development, integration and deployment of Grid middlewares. Even though the Grid community addresses the security issue through public key infrastructures (PKI) to support mutual authentication using X.509 certificates, maintaining X.509 credentials is not that easy for non-IT-experts, and has proved to be an obstacle for a more wide deployment of Grid technologies. The identity federation is an increasingly popular technology that can facilitate cross-domain single sign-on without requiring the users to maintain any credentials additional to their own institutional accounts. We believe that utilizing identity federation for Grid middlewares is a promising path for the Grid technology to get more widely used. This paper describes a single sign-on infrastructure developed as a part of the NorduGrid ARC (Advanced Resource Connector) Grid middleware. It adopts the identity federation standard (SAML), as well as other Web Service standards. It focuses on a single sign-on solution at the middleware level for users to access Grids by only using their frequently used accounts, without being bothered to maintain X.509 credentials. Users can use their username/password only to access Grids developed in ARC middleware, as well as access Grids developed in other middlewares that requires users to provide X.509 certificates. Moreover, the single sign-on for workflow-like Grid applications (in which intermediate entities act on behalf of users) is also supported. As an important aspect of single sign-on, authorization is also considered by implementing an attribute-based authorization using SAML standard. In addition, the performance of single sign-on solution is measured. We identify performance limitations of security-related services inside this solution, and analyse the ways to avoid the limitations. To our knowledge, the work presented in this paper is the first evaluated implementation that utilizes identity federation for Grid usage on the middleware level.     

HITGRID:基于动态规划技术的网格任务调度中间件

随着计算规模不断扩大,人们对并行计算调度的性能要求也不断提高。网格技术的出现,解决了计算资源的共享性、异构性、规模可扩展性、鲁棒性以及安全性等方面的问题,但同时也在资源调度方面带来了新的挑战。HITGRID是一个基于Globus网格工具平台上的网格调度中间件,它接收用户提交的计算任务,针对用户对计算资源提出的要求,在网格环境中查找符合条件的资源进行调度。文中讨论了将NWS预测技术封装为标准网格资源信息的方法,以及资源调度的性能模型,给出了调度中资源选择的策略。最后通过一个计算大规模N体问题的实例表明笔者的工作是有成效的。     

Market‐based grid resource co‐allocation and reservation for applications with hard deadlines

Grid computing technology enables the creation of large‐scale IT infrastructures that are shared across organizational boundaries. In such shared infrastructures, conflicts between user requirements are common and originate from the selfish actions that users perform when formulating their service requests. The introduction of economic principles in grid resource management offers a promising way of dealing with these conflicts. We develop and analyze both a centralized and a decentralized algorithm for economic grid resource management in the context of compute bound applications with deadline‐based quality of service requirements and non‐migratable workloads. Through the use of reservations, we co‐allocate resources across multiple providers in order to ensure that applications finish within their deadline. An evaluation of both algorithms is presented and their performance in terms of realized user value is compared with an existing market‐based resource management algorithm. We establish that our algorithms, which operate under a more realistic workload model, can closely approximate the performance of this algorithm. We also quantify the effect of allowing local workload preemption and different scheduling heuristics on the realized user value. Copyright © 2009 John Wiley & Sons, Ltd.     

An optimization approach for decentralized QoS‐based scheduling based on utility and pricing in Grid computing

This paper presents an optimization approach for decentralized Quality of Service (QoS)‐based scheduling based on utility and pricing in Grid computing. The paper assumes that the quality dimensions can be easily formulated as utility functions to express quality preferences for each task agent. The utility values are calculated by the user‐supplied utility function that can be formulated with the task parameters. The QoS constraint Grid resource scheduling problem is formulated into a utility optimization problem. The QoS‐based Grid resource scheduling optimization is decomposed into two subproblems by applying the Lagrangian method. In the Grid, a Grid task agent acts as a consumer paying for the Grid resource and the resource providers receive profits from task agents. A pricing‐based QoS scheduling algorithm is used to perform optimally decentralized QoS‐based resource scheduling. The experiments investigate the effect of the QoS metrics on the global utility and compare the performance of the proposed algorithm with other economical Grid resource scheduling algorithms. Copyright © 2006 John Wiley & Sons, Ltd.     

数据网格信息服务体系的研究与设计

网格信息服务是网格中间件的核心部分之一,提供各种分布式网格资源的静态与动态信息。现有的网格中间件几乎都是针对计算网格设计的,它们所提供的信息服务不能胜任数据网格的要求。论文主要对数据网格中的信息服务体系进行研究和设计。     

Distributed computing with Triana on the Grid

In this paper, we describe Triana, a distributed problem‐solving environment that makes use of the Grid to enable a user to compose applications from a set of components, select resources on which the composed application can be distributed and then execute the application on those resources. We describe Triana's current pluggable architecture that can support many different modes of operation by the use of flexible writers for many popular Web service choreography languages. We further show, that the Triana architecture is middleware‐independent through the use of the Grid Application Toolkit (GAT) API and demonstrate this through the use of a GAT binding to JXTA. We describe how other bindings being developed to Grid infrastructures, such as OGSA, can seamlessly be integrated within the current prototype by using the switching capability of the GAT. Finally, we outline an experiment we conducted using this prototype and discuss its current status. Copyright © 2005 John Wiley & Sons, Ltd.     

Optimization of Jobs Submission on the EGEE Production Grid: Modeling Faults Using Workload

It is commonly observed that production Grids are inherently unreliable. The aim of this work is to improve Grid application performances by tuning the job submission system. A stochastic model, capturing the behavior of a complex Grid workload management system is proposed. To instantiate the model, detailed statistics are extracted from dense Grid activity traces. The model is exploited for optimizing a simple job resubmission strategy. It provides quantitative inputs to improve job submission performance and it enables the impact of faults and outliers on Grid operations to be quantified.     

EMPEROR: An OGSA Grid Meta-Scheduler Based on Dynamic Resource Predictions

Scheduling constitutes an integral feature of Grid computing infrastructures, being also a key to realizing several of the Grid promises. In particular, scheduling can maximize the resources available to end users, accelerate the execution of jobs, while also supporting scalable and autonomic management of the resources comprising a Grid. Grid scheduling functionality hinges on middleware components called meta-schedulers, which undertake to automatically distribute jobs across the dispersed heterogeneous resources of a Grid. In this paper we present the design and implementation of a Grid meta-scheduler, which we call EMPEROR. EMPEROR provides a framework for implementing scheduling algorithms based on performance criteria. In implementing a particular instantiation of this framework, we have devised models for predicting host load and memory resources, and accordingly for estimating the running time of a task. These models hinge on time series analysis techniques and take into account results of the cluster computing literature. Apart from incorporating these models, EMPEROR provides fully fledged Grid scheduling functionality, which complies with OGSA standards as the later are reflected in the Globus toolkit. Specifically, EMPEROR interfaces to Globus middleware services (i.e., GSI, MDS, GRAM) towards discovering resources, implementing the scheduling algorithm and ultimately submitting jobs to local scheduling systems. By and large, EMPEROR is one of the few standards based meta-schedulers making use of dynamic scheduling information.     

The Virtual Service Grid: an architecture for delivering high‐end network services

This paper presents the design of a new system architecture, Virtual Service Grid (VSG), for delivering high‐performance network services. The VSG is based on the concept of the virtual service which provides location, replication, and fault transparency to clients accessing remotely deployed high‐end services. One of the novel features of the virtual service is the ability to self‐scale in response to client demand. The VSG exploits network and service information to make adaptive dynamic replica selection, creation, and deletion decisions. We describe the VSG architecture, middleware, and replica management policies. We have deployed the VSG on a wide‐area Internet testbed to evaluate its performance. The results indicate that the VSG can deliver efficient performance for a wide range of client workloads, both in terms of reduced response time and in the utilization of system resources. Copyright © 2002 John Wiley & Sons, Ltd.     

GMarte: Grid middleware to abstract remote task execution

Grid computing technologies are now being largely deployed with the widespread adoption of the Globus Toolkit as the industrial standard Grid middleware. However, its inherent steep learning curve discourages the use of these technologies for non‐experts. Therefore, to increase the use of Grid computing, it is important to have high‐level tools that simplify the process of remote task execution. In this paper we introduce a middleware, developed on top of the Java Commodity Grid, which offers an object‐oriented, user‐friendly application programming interface, from the Java language, which eases remote task execution for computationally intensive applications. Copyright © 2006 John Wiley & Sons, Ltd.     

Quality of Service Negotiation for Commercial Medical Grid Services

The GEMSS project has developed a service-oriented Grid that supports the provision of medical simulation services by service providers to clients such as hospitals. We outline the GEMSS architecture, legal framework and the security features that characterise the GEMSS infrastructure. High levels of quality of service are required and we describe a reservation-based approach to quality of service, employing a quality of service management system that iteratively finds suitable reservations and uses application specific performance models. The GEMSS Grid is a commercial environment so we support flexible pricing models and a FIPA reverse English auction protocol. Signed Web Service Level Agreement contracts are exchanged to commit parties to a quality of service agreement before job execution occurs. We run four experiments across European countries using high performance computing resources running advanced resource reservation schedulers. These experiments provide evidence for our Grid’s rational behaviour, both at the level of service provider quality of service management and at the higher level of the client choosing between competing service providers. The results lend support to our economic model and the technology we use for our medical application domain.     

GridBLAST: a Globus‐based high‐throughput implementation of BLAST in a Grid computing framework

Improvements in the performance of processors and networks have made it feasible to treat collections of workstations, servers, clusters and supercomputers as integrated computing resources or Grids. However, the very heterogeneity that is the strength of computational and data Grids can also make application development for such an environment extremely difficult. Application development in a Grid computing environment faces significant challenges in the form of problem granularity, latency and bandwidth issues as well as job scheduling. Currently existing Grid technologies limit the development of Grid applications to certain classes, namely, embarrassingly parallel, hierarchical parallelism, work flow and database applications. Of all these classes, embarrassingly parallel applications are the easiest to develop in a Grid computing framework. The work presented here deals with creating a Grid‐enabled, high‐throughput, standalone version of a bioinformatics application, BLAST, using Globus as the Grid middleware. BLAST is a sequence alignment and search technique that is embarrassingly parallel in nature and thus amenable to adaptation to a Grid environment. A detailed methodology for creating the Grid‐enabled application is presented, which can be used as a template for the development of similar applications. The application has been tested on a ‘mini‐Grid’ testbed and the results presented here show that for large problem sizes, a distributed, Grid‐enabled version can help in significantly reducing execution times. Copyright © 2005 John Wiley & Sons, Ltd.     

Service level agreement based adaptive Grid superscheduling

Grid computing brings heterogeneity and decentralization to the world of science and technology. It leverages every bit of idle computing resources and provides a straightforward middleware for integrating cross-domain scientific devices and legacy systems. In a super big Grid, job scheduling is challenging specifically when it needs to have access to vast amount of resources. The process of mapping jobs onto Grid resources requires significant consideration in terms of Grid architecture design, consumer demands and provider revenues. In this paper, we simultaneously utilize the legacy architecture of superscheduling, forwarding strategy, service level, success rate, and service pricing strategies and finally propose a service level agreement based on adaptive superscheduling (SAS) algorithm. SAS algorithm presents unified connectivity via efficient diffusion of jobs through the Grid infrastructure that is fueled from the previous scheduling events across the Grid. Moreover, by enforcing the service level agreement terms from a rich set of ask and bid prices, system performance, and load statistics, SAS successfully boosts revenue and utilization statistics. We perform an extensive experimental analysis for different Grid scales. Based on our experimental result, the SAS algorithm maximizes revenue while guarantees quality of service. More specifically, the quality of service is achieved through a high ratio of completed jobs and remarkable utilization of resources.