Network Bandwidth-aware job scheduling with dynamic information model for Grid resource brokers

A resource broker with a user-friendly interface for job submission developed on a platform constructed using the Globus toolkit is proposed. The broker employs a domain-based network information model and dynamic version to measure network statuses, and also monitors and collects resource statuses and network-related information as the basis of its brokerage. A network bandwidth-aware job scheduling algorithm for brokering suitable Grid resources to communication-intensive jobs based on improving and preserving the advantages of our previously developed network information model is also proposed. Using timely information, the resource broker effectively matches Grid resources and user requests, thus improving job execution efficiency.     

A core stateless bandwidth broker architecture for scalable support of guaranteed services

We present a novel bandwidth broker architecture for scalable support of guaranteed services that decouples the QoS control plane from the packet forwarding plane. More specifically, under this architecture, core routers do not maintain any QoS reservation states, whether per-flow or aggregate. Instead, the QoS reservation states are stored at and managed by a bandwidth broker. There are several advantages of such a bandwidth broker architecture. Among others, it avoids the problem of inconsistent QoS states faced by the conventional hop-by-hop, distributed admission control approach. Furthermore, it allows us to design efficient admission control algorithms without incurring any overhead at core routers. The proposed bandwidth broker architecture is designed based on a core stateless virtual time reference system developed recently. This virtual time reference system provides a unifying framework to characterize, in terms of their abilities to support delay guarantees, both the per-hop behaviors of core routers and the end-to-end properties of their concatenation. We focus on the design of efficient admission control algorithms under the proposed bandwidth broker architecture. We consider both per-flow end-to-end guaranteed delay services and class-based guaranteed delay services with flow aggregation. Using our bandwidth broker architecture, we demonstrate how admission control can be done on a per domain basis instead of on a "hop-by-hop" basis. Such an approach may significantly reduce the complexity of the admission control algorithms. In designing class-based admission control algorithms, we investigate the problem of dynamic flow aggregation in providing guaranteed delay services and devise a new apparatus to effectively circumvent this problem. We conduct detailed analyses to provide theoretical underpinning for our schemes as well as to establish their correctness. Simulations are also performed to demonstrate the efficacy of our schemes.     

A reinforcement learning approach to autonomous decision-making in smart electricity markets

The vision of a Smart Electric Grid relies critically on substantial advances in intelligent decentralized control mechanisms. We propose a novel class of autonomous broker agents for retail electricity trading that can operate in a wide range of Smart Electricity Markets, and that are capable of deriving long-term, profit-maximizing policies. Our brokers use Reinforcement Learning with function approximation, they can accommodate arbitrary economic signals from their environments, and they learn efficiently over the large state spaces resulting from these signals. We show how feature selection and regularization can be leveraged to automatically optimize brokers for particular market conditions, and demonstrate the performance of our design in extensive experiments using real-world energy market data.     

The Virtual Laboratory: a toolset to enable distributed molecular modelling for drug design on the World‐Wide Grid

Computational Grids are emerging as a new paradigm for sharing and aggregation of geographically distributed resources for solving large‐scale compute and data intensive problems in science, engineering and commerce. However, application development, resource management and scheduling in these environments is a complex undertaking. In this paper, we illustrate the development of a Virtual Laboratory environment by leveraging existing Grid technologies to enable molecular modelling for drug design on geographically distributed resources. It involves screening millions of compounds in the chemical database (CDB) against a protein target to identify those with potential use for drug design. We have used the Nimrod‐G parameter specification language to transform the existing molecular docking application into a parameter sweep application for executing on distributed systems. We have developed new tools for enabling access to ligand records/molecules in the CDB from remote resources. The Nimrod‐G resource broker along with molecule CDB data broker is used for scheduling and on‐demand processing of docking jobs on the World‐Wide Grid (WWG) resources. The results demonstrate the ease of use and power of the Nimrod‐G and virtual laboratory tools for grid computing. Copyright © 2003 John Wiley & Sons, Ltd.     

A computational economy for grid computing and its implementation in the Nimrod-G resource broker

Computational grids that couple geographically distributed resources such as PCs, workstations, clusters, and scientific instruments, have emerged as a next generation computing platform for solving large-scale problems in science, engineering, and commerce. However, application development, resource management, and scheduling in these environments continue to be a complex undertaking. In this article, we discuss our efforts in developing a resource management system for scheduling computations on resources distributed across the world with varying quality of service (QoS). Our service-oriented grid computing system called Nimrod-G manages all operations associated with remote execution including resource discovery, trading, scheduling based on economic principles and a user-defined QoS requirement. The Nimrod-G resource broker is implemented by leveraging existing technologies such as Globus, and provides new services that are essential for constructing industrial-strength grids. We present the results of experiments using the Nimrod-G resource broker for scheduling parametric computations on the World Wide Grid (WWG) resources that span five continents.     

Developing adaptable user interfaces for component-based systems

Software components are becoming increasingly popular design and implementation technologies that can be plugged and played to provide user-enhanceable software. However, developing software components with user interfaces that can be adapted to diverse reuse situations is challenging. Examples of such adaptations include extending, composing and reconfiguring multiple component user interfaces, and adapting component user interfaces to particular user preferences, roles and subtasks. We describe our recent work in facilitating such adaptation via the concept of user interface aspects, which support effective component user interface design and realisation using an extended, component-based software architecture.     

A multicriteria approach to two-level hierarchy scheduling in grids

In this paper we address a multicriteria scheduling problem for computational Grid systems. We focus on the two-level hierarchical Grid scheduling problem, in which at the first level (the Grid level) a Grid broker makes scheduling decisions and allocates jobs to Grid nodes. Jobs are then sent to the Grid nodes, where local schedulers generate local schedules for each node accordingly. A general approach is presented taking into account preferences of all the stakeholders of Grid scheduling (end-users, Grid administrators, and local resource providers) and assuming a lack of knowledge about job time characteristics. A single-stakeholder, single-criterion version of the approach has been compared experimentally with the existing approaches.     

Quality of Service Networking for High Performance Grid Applications

This paper reports on different efforts to provide quality of service (QoS) Networking to Grid applications done in the context of the MB-NG, GRS and DataTAG EU projects. These are leading edge network research projects involving more that 50 researchers in the UK, Europe and North America, concerned with the development and testing of protocols and standards for the next generation of high speed networks. We have implemented and tested the Differentiated Services Architecture (DiffServ) in a multi-domain, 2.5 Gbits/s network (the first such deployment) defining appropriate Service Level Agreements (SLAs) to be used between administrative domains to guarantee end-to-end Quality of Service. We characterised several hardware implementations of DiffServ and concluded on their appropriateness for several network scenarios. Since current and future Grid applications will have to use modified mechanisms of congestion control we have evaluated old and new TCP implementations over a Differentiated Services Networks. These quality of service tests have also included innovative MPLS (Multi-Protocol Label Switching) experiments to establish guaranteed bandwidth connections to Grid applications in a fast and efficient way. We have also developed a software based bandwidth broker architecture for Grids based on IETF standards which allows applications to transparently request dynamic and advanced reservations and implemented it in a real experimental network. We finally report on experiences delivering Quality of Service networking to high performance applications like Particle Physics data transfer and High Performance Computation. This includes quantitative results on the performance improvements that QoS brought to real data transfers in the context of High Performance Computing.     

An infrastructure to support cooperation of knowledge-level agents on the semantic Grid

Can a Knowledge-Level layer be located in the Semantic Grid infrastructure? Is it possible to design an Agent Communication Language (ACL) which enables Knowledge-Level agents to cooperate in a geographically distributed Semantic Grid despite nodes’ failures or malfunctions? This paper tries to address the above Semantic Grid challenges presenting an agent-based Open Service Architecture which integrates geographically distributed agents in a Semantic Grid. The architecture is well integrated with standard Internet components and technologies and supports communication among Knowledge-Level agents. The role of agents is to retrieve, execute and compose available services providing more sophisticated instances of them. Inter-agent communication is realized by exploiting an advanced Agent Communication Language which supports a fault-tolerant anonymous interaction protocol and satisfies a set of well defined Knowledge-Level programming requirements. Here, we present the design of the architecture and of the Agent Communication Language as well as their implementation. The architecture is evaluated by means of several case studies which highlight the main feature of our proposal. The main advantage of our approach is to demonstrate that different issues, such as high level inter-agent communication and fault tolerance, can be successfully integrated in Grid infrastructures which provide Web Services maintaining a clean design of the architecture and a Knowledge-Level characterization.     

An External Broker Interface Based on Existing Trading Protocols

This paper presents an approach to the specification of external interfaces of an electronic broker used in the Architecture for Electronic Brokerage Systems (AEBS). The external interface specified by the AEBS architecture provides both rich functionality and backward compatibility with trading methods that are currently in use in the electronic marketplace. This is achieved by defining the interface at two levels: the external protocol interface and the external object interface. The external protocol interface has been defined in terms of traditional protocols in order to provide access to basic brokerage services for customers and suppliers that already operate in the electronic market. This facilitates the easy introduction of a broker to the market since no modifications are required on the customer or supplier side to use the new brokerage services. The advanced external object interface has been designed and defined using the CORBA model. This provides other actors that need advanced brokerage services with access to the full functionality of a broker. This paper focuses on the design of the external protocol interface of a brokerage system.     

Design and implementation of a workflow-based resource broker with information system on computational grids

The grid is a promising infrastructure that can allow scientists and engineers to access resources among geographically distributed environments. Grid computing is a new technology which focuses on aggregating resources (e.g., processor cycles, disk storage, and contents) from a large-scale computing platform. Making grid computing a reality requires a resource broker to manage and monitor available resources. This paper presents a workflow-based resource broker whose main functions are matching available resources with user requests and considering network information statuses during matchmaking in computational grids. The resource broker provides a graphic user interface for accessing available and the appropriate resources via user credentials. This broker uses the Ganglia and NWS tools to monitor resource status and network-related information, respectively. Then we propose a history-based execution time estimation model to predict the execution time of parallel applications, according to previous execution results. The experimental results show that our model can accurately predict the execution time of embarrassingly parallel applications. We also report on using the Globus Toolkit to construct a grid platform called the TIGER project that integrates resources distributed across five universities in Taichung city, Taiwan, where the resource broker was developed.

基于经济模型的资源代理

网格计算是为解决大规模资源密集型问题而提出的新一代计算平台,是当前并行和分布处理技术的一个发展方向,而资源管理是计算网格的关键技术之一。对各种各样可利用资源的整合和管理是网格应用的基础,而资源的分布性、动态性、异构性、自治性和需要协调一致性使得网格资源的管理调度成为一个棘手的问题。目前基于市场的经济资源管理和调度算法非常适合计算网格中的资源管理问题,但有调度价格不能更改、负载平衡等问题。文中提出了“网格环境下基于经济模型的资源代理”,依靠多维QoS指导的调度策略和经济模型的启发式调节资源价格,改进和优化计算网格资源的分配。     

A brokerage system for solar data archives

Scientific research and practical applications of solar physics require data and computational services to be integrated seamlessly and efficiently. The European Grid for Solar Observations (EGSO) leverages Grid‐oriented concepts and technology to provide a high‐performance infrastructure for solar applications. In this paper, an architecture for a data brokerage service is proposed. Brokers interact with providers and consumers in order to build a profile of both parties. In particular, a broker interacts with providers in order to gather information on the data potentially available to consumers, and with the consumers in order to identify the set of providers that are most likely to satisfy specific data needs. The brokerage technique is based on a multi‐tier management of metadata. Copyright © 2006 John Wiley & Sons, Ltd.     

Active Grid Information Server for grid computing

We describe the Active Grid Information Server for performing expressive resource discovery searches and resource management in a grid environment. We represent server meta-data, such as their CPU power, storage capacity and bandwidth as points in a multi-dimensional space and then express queries as predicates over these points. The design of the system is driven by its application as a part of the information infrastructure for computational grids. Such grids provide an infrastructure for sharing computing resources; an information infrastructure is their inherent part which collects resource data and provides search functionality. Our approach complements current solutions such as MDS by adding Event Condition Action rules and an ability to efficiently handle dynamic attributes. An Event Condition Action rule-based system can support ad hoc, adaptive, flexible, and dynamic schedulers that are modifiable at runtime. We evaluate our Active Grid Information Server by evaluating Event Condition Action rules for termination, confluence, and conflict.     

Resource brokering using a multi‐site resource allocation strategy for computational grids

Grid computing employs heterogeneous resources which may be installed on different platforms, hardware/software, computer architectures, and perhaps using different computer languages to solve large‐scale computational problems. As many more Grids are being developed worldwide, the number of multi‐institutional collaborations is growing rapidly. However, to realize Grid computing's full potential, it is expected that Grid participants must be able to share one another's resources. This paper presents a resource broker that employs the multi‐site resource allocation (MSRA) strategy and the dynamic domain‐based network information model that we propose to allocate Grid resources to submitted jobs, where the Grid resources may be dispersed at different sites, and owned and governed by different organizations or institutes. The jobs and resources may also belong to different clusters/sites. Resource statuses collected by the Ganglia, and network bandwidths gathered by the Network Weather Service, are both considered in the proposed scheduling approach. A dynamic domain‐based model for network information measurement is also proposed to choose the most appropriate resources that meet the jobs' execution requirements. Experimental results show that MSRA outperformed the other tested strategies. Copyright © 2010 John Wiley & Sons, Ltd.     

Grid resource brokering algorithms enabling advance reservations and resource selection based on performance predictions

We present algorithms, methods, and software for a Grid resource manager, that performs resource brokering and job scheduling in production Grids. This decentralized broker selects computational resources based on actual job requirements, job characteristics, and information provided by the resources, with the aim to minimize the total time to delivery for the individual application. The total time to delivery includes the time for program execution, batch queue waiting, and transfer of executable and input/output data to and from the resource. The main features of the resource broker include two alternative approaches to advance reservations, resource selection algorithms based on computer benchmark results and network performance predictions, and a basic adaptation facility. The broker is implemented as a built-in component of a job submission client for the NorduGrid/ARC middleware.     

Transparent access to Grid resources for user software

Grid computing promises access to large amounts of computing power, but so far adoption of Grid computing has been limited to highly specialized experts for three reasons. First, users are used to batch systems, and interfaces to Grid software are often complex and different to those in batch systems. Second, users are used to having transparent file access, which Grid software does not conveniently provide. Third, efforts to achieve wide‐spread coordination of computers while solving the first two problems is hampered when clusters are on private networks. Here we bring together a variety of software that allows users to almost transparently use Grid resources as if they were local resources while providing transparent access to files, even when private networks intervene. As a motivating example, the BaBar Monte Carlo production system is deployed on a truly distributed environment, the European DataGrid, without any modification to the application itself. Copyright © 2005 John Wiley & Sons, Ltd.