Network Bandwidth-aware job scheduling with dynamic information model for Grid resource brokers

A resource broker with a user-friendly interface for job submission developed on a platform constructed using the Globus toolkit is proposed. The broker employs a domain-based network information model and dynamic version to measure network statuses, and also monitors and collects resource statuses and network-related information as the basis of its brokerage. A network bandwidth-aware job scheduling algorithm for brokering suitable Grid resources to communication-intensive jobs based on improving and preserving the advantages of our previously developed network information model is also proposed. Using timely information, the resource broker effectively matches Grid resources and user requests, thus improving job execution efficiency.     

一种基于预测的资源匹配算法

网格任务调度过程中的资源匹配是根据任务要求从网格资源信息服务（GRIS）中查找出合适资源的过程。GRIS中记录的往往是资源的静态信息，由于本地负载的动态变化使得基于资源静态信息来确定的候选资源集中一些资源并不能满足任务的QoS需求。基于相关资源动态信息预测资源未来状态，给出了网格任务平均完成时间及完成时间的分布函数，并根据任务QoS需求，兼顾考虑资源当前及未来状态，提出了一种资源匹配模型与匹配算法。通过实验表明，该算法能有效减少候选资源数目，从而降低调度时间复杂度。     

Neural network‐based multi‐agent approach for scheduling in distributed systems

A distributed system consists of a collection of autonomous heterogeneous resources that provide resource sharing and a common platform for running parallel compute‐intensive applications. The different application characteristics combined with the heterogeneity and performance variations of the distributed system make it difficult to find the optimal set of needed resources. When deployed, user applications are usually handled by application domain experts or system administrators who depending on the infrastructure provide a scheduling strategy for selecting the best candidate resource over a set of available resources. However, the provided strategy is usually generic, aimed at handling a wide array of applications and does not take into consideration specific application resource requirements. As such, an intelligent method for selecting the best resources based on expert knowledge is needed. In this paper, we propose a neural network‐based multi‐agent resource selection technique capable of mimicking the services of an expert user. In addition, to cope with the geographical distribution of the underlying system, we employ a multi‐agent coordination mechanism. The proposed neural network‐based scheduling framework combined with the multi‐agent intelligence is a unique approach to efficiently deal with the resource selection problem. Results run on a simulated environment show the efficiency of our proposed method. Several scheduling simulations were conducted to compare the performance of some conventional resource selection methods against the proposed agent‐based neural network technique. The results obtained indicate that the agent‐based approach outperformed the classical algorithms by reducing the amount of time required to search for suitable resources irrespective of the resource size. Copyright © 2016 John Wiley & Sons, Ltd.     

A replicated information system to enable dynamic collaborations in the Grid

The main advantage of Grid computing over other distributed computing paradigms is its capability to coordinate the access to data and resources in a virtual multi‐institutional environment. To this end, the information system plays a decisive role in selecting the services that meet the applications' needs. This paper presents an information system for the Grid that provides transparent and scalable group communication services to standard Grid applications, with the objective of supporting dynamic collaborations that could help address problems that involve only some participants of a virtual organization. In particular, it enables more flexible delivery mechanisms, which allows applications to select the appropriate services before sending their data to the information system. This significantly enhances the protection of data from unauthorized access, and avoids the transmission of unnecessary messages over the network. The proposed information system is based on the use of XML technologies and replication. It introduces several new advanced features that are not currently supported as a whole by any Grid middleware, such as: several entry points to the information, persistent capabilities, support for advanced queries based on XQuery, and support for the industrial standard WS‐Policy. The information system has been stress tested under realistic workloads in a Grid infrastructure with 50 sites. Scalability has been evaluated in up to 1000 messages that can be up to 10KB in size each, updated with a frequency of 5min. Copyright © 2012 John Wiley & Sons, Ltd.     

A new step toward load balancing based on competency rank and transitional phases in Grid networks

Grid computing is a network of software-hardware capabilities. It serves as a comprehensive and complete system for organizations by which the maximum utilization from resources is achieved. Resource distribution in a heterogeneous and unstable environment and also effective load distribution among these resources are the important and difficult problems in Grid networks. Using dynamic and static algorithms or searching tree and Branch and Bound algorithm are considered to be among the available methods to reach the load balancing in Grid networks. This paper presents a new method for dynamic load balancing. In this method, we use the subtraction of forward and backward ants as a competency rank to take the priority of the sites, and we use a control word to search the suitable resource as well. Our main purpose is to devote jobs to the existing resources based on their processing power. Simulation results show that the proposed method can reduce the total completion time and also total tardiness to get the load balancing. The cost of using resources as an effective factor in load balancing is also observed.     

Resource Space Grid: model,method and platform

A Resource Space Grid is a virtual Grid that aims at effectively sharing, using and managing versatile resources across the Internet. The kernel of the Resource Space Grid includes a Resource Space Model (RSM) and a uniform Resource Using Mechanism (RUM). This paper presents the Resource Space Grid's core scientific issues and methodology, architecture, model and theory, design criteria and method, and practice. A normal form theory is proposed to normalize the resource space—a coordinate system for uniformly specifying and organizing resources. The RUM provides not only the end‐users with an operable resource browser to operate resources using the built‐in Resource Operation Language (ROL), but also the application developers with the ROL‐based programming environment. The prototype platform based on the proposed model and method has been implemented and used for sharing and managing resources in distributed research teams. Operations on Resource Spaces can constitute the virtual communities of Resource Space Grids—a platform independent resource sharing environment. Copyright © 2004 John Wiley & Sons, Ltd.     

可靠的网格作业调度机制

针对网格环境的动态性特征,提出了一种可靠的网格作业调度机制(DGJS)。按照作业完成时间期限,DGJS将作业分为：高QoS级、低QoS级和无QoS级,不同QoS级作业有不同的调度优先权;基于资源可用性预测,DGJS采用基于可靠性代价的作业调度策略,将作业尽可能调度到可靠性高的资源节点;另外,DGJS对不同QoS级作业采用不同的容错策略,在保证故障容错的同时,节省网格资源。实验表明：在动态的网格环境下,较之传统的网格作业调度算法,DGJS提高了作业成功率,减少了作业完成时间。     

一种全局统一的层次化网格资源模型

网格计算通过新的组织方式将广域网上的各种计算资源、信息资源、设备资源等集成起来，以统一的方式向用户提供服务，是当前网络计算领域的研究热点，引入逻辑资源树的概念，通过抽象资源参数，提出了一种全局统一的层次化网格资源模型，支持资源的动态加入与删除，与资源池及全局一本地两层资源模型相比，提出的资源模型有效地屏蔽了广域网上资源的异构性，提高了资源的可扩展性；同时根据网络通信性能对资源进行层次化组织，避免了盲目的资源选择，该模型进行资源查找的时间复杂度为Olog(N)，有较高的查找效率。     

VEGA Infrastructure for Resource Discovery in Grids

Grids enable users to share and access large collections and various types of re-sources in wide areas, and how to locate resources in such dynamic, heterogeneous and autonomousdistributed environments is a key and challenging issue. In this paper, a three-level decentralizedand dynamic VEGA Infrastructure for Resource Discovery (VIRD) is proposed. In this architec-ture, every Border Grid Resource Name Server (BGRNS) or Grid Resource Name Server (GRNS)has its own local policies, governing information organization, management and searching. Changesin resource information are propagated dynamically among GRNS servers according to a link-state-like algorithm. A client can query its designated GRNS either recursively or iteratively. Optimizing techniques, such as shortcut, are adopted to make the dynamic framework more flexible and effi-cient. A simulator called SimVIRD is developed to verify the proposed architecture and algorithm.q.Experiment results indicate that this architecture could deliver good scalability and performance for grid resource discovery.     

Ontology‐based Grid resource management

Grid resources are typically diverse in nature with respect to their software and hardware configurations, resource usage policies and the kind of application they support. Aggregating and monitoring these resources, and discovering suitable resources for the applications become a challenging issue. This is partially due to the representation of Grid metadata supported by the existing Grid middleware which offers limited scope for matching the job requirements that directly affect scheduling decisions. This paper proposes a semantic component in conventional Grid architecture to support ontology‐based representation of Grid metadata and facilitate context‐based information retrieval that complements Grid schedulers for effective resource management. Web Ontology language is used for creating Grid resource ontology and Algernon inference engine has been used for resource discovery. This semantic component has been integrated with conventional Grid schedulers. Several experiments have also been carried out to investigate the performance overhead that arises while integrating this component with Grid schedulers. Copyright © 2009 John Wiley & Sons, Ltd.     

Decentralized Grid Resource Discovery Based on Resource Information Community

A decentralized Grid resource discovery solution is presented in this paper under predefined resource taxonomy, in which information nodes with the same type of registered resources are organized together to form resource information communities (RIC), and efficient navigation between different communities is achieved by a DHT P2P based bootstrap network. Periodical topology maintenance communications are used to piggyback and disseminate popular data in bootstrap network to achieve better load balance. The performance of RIC-based Grid resource discovery is evaluated by simulation under different cases, and overhead is also studied.     

A Grid resource brokering strategy based on resource and network performance in Grid

To achieve high performance distributed data access and computing in Grid environment, monitoring of resource and network performance is vital. Our proposed Grid network monitoring architecture is modeled by the Grid scheduler. The proposed Grid network monitoring retrieves network metrics using sensors as network monitoring tools. The mobile agents are migrated to start the sensors to measure the network metrics in all Grid Resources from the Resource Broker. The raw data provided by the monitoring tools is used to produce a high level view of the Grid through the set of internal cost functions. The network cost function is formed by combining various network metrics such as bandwidth, Round Trip Time, jitter and packet loss to measure the network performance. This paper presents the Grid Resource Brokering strategy which analyzes the network metrics along with the resource metrics for the selection of the Grid resource to submit the job and the proposed approach is integrated with CARE Resource Broker (CRB) for job submission. The experimental results are evident for the minimization of job completion time for the submitted job. The simulation results also prove that the more number of jobs are completed with the proposed strategy which influences the better utilization of the Grid resources.     

Grids and Grid technologies for wide‐area distributed computing

The last decade has seen a substantial increase in commodity computer and network performance, mainly as a result of faster hardware and more sophisticated software. Nevertheless, there are still problems, in the fields of science, engineering, and business, which cannot be effectively dealt with using the current generation of supercomputers. In fact, due to their size and complexity, these problems are often very numerically and/or data intensive and consequently require a variety ofheterogeneous resources that are not available on a single machine. A number of teams have conducted experimental studies on the cooperative use of geographically distributed resources unified to act as a single powerful computer. This new approach is known by several names, such as metacomputing, scalable computing, global computing, Internet computing, and more recently peer‐to‐peer or Grid computing. The early efforts in Grid computing started as a project to link supercomputing sites, but have now grown far beyond their original intent. In fact, many applications can benefit from the Grid infrastructure, including collaborative engineering, data exploration, high‐throughput computing, and of course distributed supercomputing. Moreover, due to the rapid growth of the Internet and Web, there has been a rising interest in Web‐based distributed computing, and many projects have been started and aim to exploit the Web as an infrastructure for running coarse‐grained distributed and parallel applications. In this context, the Web has the capability to be a platform for parallel and collaborative work as well as a key technology to create a pervasive and ubiquitous Grid‐based infrastructure. This paper aims to present the state‐of‐the‐art of Grid computing and attempts to survey the major international efforts in developing this emerging technology. Copyright © 2002 John Wiley & Sons, Ltd.     

Scalable dimensioning of resilient Lambda Grids

Grids consist of the aggregation of numerous dispersed computational, storage and network resources, able to satisfy even the most demanding computing jobs. Due to the data-intensive nature of Grid jobs, there is an increasing interest in Grids using optical transport networks as this technology allows for the timely delivery of large amounts of data. Such Grids are commonly referred to as Lambda Grids.

An important aspect of Grid deployment is the allocation and activation of installed network capacity, needed to transfer data and jobs to and from remote resources. However, the exact nature of a Grid’s network traffic depends on the way arriving workload is scheduled over the various Grid sites. As Grids possibly feature high numbers of resources, jobs and users, solving the combined Grid network dimensioning and workload scheduling problem requires the use of scalable mathematical methods such as Divisible Load Theory (DLT). Lambda Grids feature additional complexity such as wavelength granularity and continuity or conversion constraints must be enforced. Additionally, Grid resources cannot be expected to be available at all times. Therefore, the extra complexity of resilience against possible resource failures must be taken into account when modelling the combined Grid network dimensioning and workload scheduling problem, enforcing the need for scalable solution methods. In this work, we tackle the Lambda Grid combined dimensioning and workload scheduling problem and incorporate single-resource failure or unavailability scenarios. We use Divisible Load Theory to tackle the scalability problem and compare non-resilient lambda Grid dimensioning to the dimensions needed to survive single-resource failures. We distinguish three failure scenarios relevant to lambda Grid deployment: computational element, network link and optical cross-connect failure. Using regular network topologies, we derive analytical bounds on the dimensioning cost. To validate these bounds, we present comparisons for the resulting Grid dimensions assuming a 2-tier Grid operation as a function of varying wavelength granularity, fiber/wavelength cost models, traffic demand asymmetry and Grid scheduling strategy for a specific set of optical transport networks.     

网格资源的超拓扑空间模型

为了更好地共享和使用网格资源,资源模型已经成为网格计算的重要研究问题之一。论文从超拓扑空间的角度重新审视了网格资源模型的内在机理与结构,对资源的描述、抽象以及单一系统映像方法进行了深入的研究,提出了一个形式化资源模型,具有三层结构,分别是:特征信息层、特征处理层和功能层。各层有着不同的组织形式与拓扑结构,并且进一步可细分为若干亚层;利用所提出的聚类方法,上层结构可由下层自动形成。应用分析表明,该模型能够有效地提高网格资源管理系统的可用性、模块性、透明性和自动化。     

Validating and Scaling the MicroGrid: A Scientific Instrument for Grid Dynamics

Large-scale Grids that aggregate and share resources over wide-area networks present major challenges in understanding dynamic application and resource behavior for performance, stability, and reliability. Accurate study of the dynamic behavior of applications, middleware, resources, and networks depends on coordinated and accurate modeling of all four of these elements simultaneously. We have designed and implemented a tool called the MicroGrid which enables accurate and comprehensive study of the dynamic interaction of applications, middleware, resource, and networks. The MicroGrid creates a virtual Grid environment – accurately modeling networks, resources, the information services (resource and network metadata) transparently. Thus, the MicroGrid enables users, Grid researchers, or Grid operators to study arbitrary collections of resources and networks. The MicroGrid includes the MaSSF online network simulator which provides packet-level accurate, but scalable network modeling. We present experimental results with applications which validate the implementation of the MicroGrid, showing that it not only runs real Grid applications and middleware, but that it accurately models both their and underlying resource and network behavior. We also study a range of techniques for scaling a critical part of the online network simulator to the simulation of large networks. These techniques employ a sophisticated graph partitioner, and a range of edge and node weighting schemes exploiting a range of static network and dynamic application information. The best of these, profile-driven placement, scales well to online simulation of large networks of 6,000 nodes using 24 simulation engine nodes.     

一种基于语义的P2P网格资源发现模型

网格环境中资源的有效管理和发现在网格研究中占有十分重要的地位。文章提出了一种基于语义和P2P技术的新的网格资源发现模型——Semantic-basedPeertoPeerGrid(简称SPTOPGrid)。该模型采用P2P技术进行资源检索,每个peer节点都能提供资源描述和某个知识领域的本体库(ontology);采用publish/subscribe模式进行资源分发;利用语义知识库进行资源的匹配,进行智能化的资源搜索,能够适应网格资源的复杂性、异构性和动态性,具有良好的扩展性。     

Resource Allocation in the Grid with Learning Agents

One of the main challenges in Grid computing is efficient allocation of resources (CPU – hours, network bandwidth, etc.) to the tasks submitted by users. Due to the lack of centralized control and the dynamic/stochastic nature of resource availability, any successful allocation mechanism should be highly distributed and robust to the changes in the Grid environment. Moreover, it is desirable to have an allocation mechanism that does not rely on the availability of coherent global information. In this paper we examine a simple algorithm for distributed resource allocation in a simplified Grid-like environment that meets the above requirements. Our system consists of a large number of heterogenous reinforcement learning agents that share common resources for their computational needs. There is no explicit communication or interaction between the agents: the only information that agents receive is the expected response time of a job it submitted to a particular resource, which serves as a reinforcement signal for the agent. The results of our experiments suggest that even simple reinforcement learning can indeed be used to achieve load balanced resource allocation in large scale heterogenous system.     

改进的网格资源信息更新策略

网格是构筑在互联网上的一种新型技术,正成为新一代的网络基础设施。网格的目的是把网络的资源有效的整合,实现资源的跨组织、跨地域的动态共享,因此网格资源的有效管理成为网格研究中的关键技术之一。数据变化更新策略是在资源状态发生变化时通知资源信息中心更新数据。在数据变化更新策略的基础上,提出了一种改进的资源更新策略:如果资源状态信息的变化超过了一定的幅度,资源才发出信息更新通知。     

Randomized gossip algorithms for maintaining a distributed bulletin board with guaranteed age properties

Scalable computer systems, including clusters and multi‐cluster grids, require routine exchange of information about the state of system‐wide resources among their nodes. Gossip‐based algorithms are popular for providing such information services due to their simplicity, fault tolerance and low communication overhead. This paper presents a randomized gossip algorithm for maintaining a distributed bulletin board among the nodes of a scalable computer system. In this algorithm each node routinely disseminates its most recently acquired information while maintaining a snapshot of the other nodes' states. The paper provides analytical approximations for the expected average age, the age distribution and the expected maximal age for the acquired information at each node. We confirm our results by measurements of the performance of the algorithm on a multi‐cluster campus grid with 256 nodes and by simulations of configurations with up to 2048 nodes. The paper then presents practical enhancements of the algorithm, which makes it more suitable for a real system. Such enhancements include using fixed‐size messages, reducing the number of messages sent to inactive nodes and supporting urgent information. The enhanced algorithm guarantees the age properties of the information at each node in the configurations with an arbitrary number of inactive nodes. It is being used in our campus grid for resource discovery, for dynamic assignment of processes to the best available nodes, for load‐balancing and for on‐line monitoring. Copyright © 2009 John Wiley & Sons, Ltd.