An agent-based approach for dynamic adjustment of scheduled jobs in computational grids

Grid computing is a newly developed technology for complex systems with large-scale resource sharing, wide-area communication, and multi-institutional collaboration. Grid scheduling is an important infrastructure in the grid computing environment. Most of the existing grids scheduling methods focus on maximizing processor utilization without taking grid load into consideration. This may lead to significant inefficiencies in performance such as large job queues and processing delays. In this paper, we propose a multiagent-based scheduling system for computational grids with a new approach. Agent technology is suitable for a computational grid because of the dynamic, heterogeneous, and autonomous nature of the grid. The main idea of the proposed system is a combination of a static scheduling using a fixed scheduling algorithm and a dynamic adjustment through the autonomous behavior of agents. The superiority of the proposed system, in reducing the load of the grid and minimizing the response time for executing user applications, is demonstrated by simulation experiments.     

Evolutionary based hybrid GA for solving multi-objective grid scheduling problem

The grid computing aims at bringing computing capacities together in a manner that can be used to find solutions for complicated problems of science. Conventional algorithms like first come first serve (FCFS), shortest job first (SJF) has been used for solving grid scheduling problem (GSP), but the increased complexity and job size led to the poor performance of these algorithms especially in the grid environment due to its dynamic nature. Previously, researchers have used a genetic algorithm (GA) to schedule jobs in the grid environment. In this paper, a multi-objective GSP is solved and optimized using the proposed algorithm. The proposed algorithm enhances the way the genetic algorithm performs and incorporate significant changes in the initialization step of the algorithm. The proposed algorithm uses SJF during its initialization step for producing the initial population solution. The proposed GA has three key features which are discussed in this paper: It executes jobs with minimum job completion time. It performs load balancing and improves resource utilization. Lastly, it supports scalability. The proposed algorithm is tested using a standard workload (given by Czech National Grid Infrastructure named Metacentrum) which can be a benchmark for further research. A performance comparison shows that the proposed algorithm has got better scheduling results than other scheduling algorithms.

     

计算密集型与数据密集型混合网格作业调度算法

针对计算密集型作业与数据密集型作业混合情况,在一个作业有时间限制的动态环境中,对传统的网格作业调度方法进行扩展,提出了三种网格作业调度启发式算法:Emin min、Ebest、Esufferage。并在一个由多个Cluster组成的、通过高速网络连接的网格模型上,对三种算法进行验证。与Min min算法的比较结果显示：三种算法均优于Min min算法。与ASJS算法比较结果显示：Emin min减少了等待时间与作业的makespan; Esufferage算法以减少作业完成量为代价,减少了作业的等待时间及makespan; Ebest在完成作业数量上与ASJS基本保持一致,但却增加了作业的等待时间与makespan。总体上,Emin min具有比较大的优势。     

基于信任机制与遗传算法的渲染网格作业调度研究

信任关系是网格作业调度中一个很重要的因素,也是影响网格计算有效性和性能的关键技术之一。将信任机制引入到渲染网格作业调度中,建立渲染网格环境中基于信任机制的作业调度模型,在调度策略上对基本遗传算法进行了改进,提出了基于信任机制的遗传算法。实验结果表明,该算法可以提高任务完成率和平均信任效益,是适用于渲染网格的一种有效作业调度方法。     

基于ECT的优先权约束作业调度策略研究

为了协调网格计算中异构资源在多用户之间的合理共享,满足不同用户需求,该文提出一种基于ECT的优先权约束作业调度策略。该策略充分考虑不同作业的期望完成时间,并通过为不同级别用户设置优先级,使得高优先权用户的作业优先执行,保证绝大多数作业在期望完成时间之内完成,同时平衡了各种资源的利用率。该策略解决了网格环境下不同类别用户无冲突共享资源问题,提高了用户满意程度,实现了作业与异构资源之间的合理匹配。     

A multi-dimensional scheduling scheme in a Grid computing environment

In this paper, we propose a novel distributed resource-scheduling algorithm capable of handling multiple resource requirements for jobs that arrive in a Grid computing environment. In our proposed algorithm, referred to as multiple resource scheduling (MRS) algorithm, we take into account both the site capabilities and the resource requirements of jobs. The main objective of the algorithm is to obtain a minimal execution schedule through efficient management of available Grid resources. We first propose a model in which the job and site resource characteristics can be captured together and used in the scheduling algorithm. To do so, we introduce the concept of a n-dimensional virtual map and resource potential. Based on the proposed model, we conduct rigorous simulation experiments with real-life workload traces reported in the literature to quantify the performance. We compare our strategy with most of the commonly used algorithms in place on performance metrics such as job wait times, queue completion times, and average resource utilization. Our combined consideration of job and resource characteristics is shown to render high-performance with respect to above-mentioned metrics in the environment. Our study also reveals the fact that MRS scheme has a capability to adapt to both serial and parallel job requirements, especially when job fragmentation occurs. Our experimental results clearly show that MRS outperforms other strategies and we highlight the impact and importance of our strategy.     

计算网格中基于时间均衡的并行粗粒度任务调度算法

考虑网格资源异构、自治、动态等特性,讨论本地用户具有强占优先权情况下的任务调度问题,提出了TBBS(Time-Balancing Based Scheduling Algorithm)算法.建立调度优化模型,以期望完成时间最小为目标选择执行任务的最佳资源组合.以时间均衡策略将任务分解并调度到资源上执行,减少了子任务同步时因等待而产生的延时,获得较好的并行计算性能.采用重复调度策略,适应计算网格中资源的特性.     

基于经济模型的资源代理

网格计算是为解决大规模资源密集型问题而提出的新一代计算平台,是当前并行和分布处理技术的一个发展方向,而资源管理是计算网格的关键技术之一。对各种各样可利用资源的整合和管理是网格应用的基础,而资源的分布性、动态性、异构性、自治性和需要协调一致性使得网格资源的管理调度成为一个棘手的问题。目前基于市场的经济资源管理和调度算法非常适合计算网格中的资源管理问题,但有调度价格不能更改、负载平衡等问题。文中提出了“网格环境下基于经济模型的资源代理”,依靠多维QoS指导的调度策略和经济模型的启发式调节资源价格,改进和优化计算网格资源的分配。     

Service level agreement based adaptive Grid superscheduling

Grid computing brings heterogeneity and decentralization to the world of science and technology. It leverages every bit of idle computing resources and provides a straightforward middleware for integrating cross-domain scientific devices and legacy systems. In a super big Grid, job scheduling is challenging specifically when it needs to have access to vast amount of resources. The process of mapping jobs onto Grid resources requires significant consideration in terms of Grid architecture design, consumer demands and provider revenues. In this paper, we simultaneously utilize the legacy architecture of superscheduling, forwarding strategy, service level, success rate, and service pricing strategies and finally propose a service level agreement based on adaptive superscheduling (SAS) algorithm. SAS algorithm presents unified connectivity via efficient diffusion of jobs through the Grid infrastructure that is fueled from the previous scheduling events across the Grid. Moreover, by enforcing the service level agreement terms from a rich set of ask and bid prices, system performance, and load statistics, SAS successfully boosts revenue and utilization statistics. We perform an extensive experimental analysis for different Grid scales. Based on our experimental result, the SAS algorithm maximizes revenue while guarantees quality of service. More specifically, the quality of service is achieved through a high ratio of completed jobs and remarkable utilization of resources.     

一种改进的启发式网格任务调度算法

网格任务调度算法是影响网格成功与否的关键技术之一。网格计算中,一个好的任务调度算法不但要考虑所有任务的makespan,使其值尽量小,同样要考虑到整个系统机器间的负载平衡问题。文章对异构计算环境下的元任务调度算法进行了分析,针对Min-min算法可能引发的负载不平衡问题,结合网格计算环境的特点,提出了一种适用于网格计算环境中的任务调度算法。     

基于任务复制的网格任务调度算法

网格中资源之间存在着通信延迟，通过任务复制的冗余，可以减少任务之间的通信开销，缩短整个计算程序的计算时间。目前网格中的任务调度算法基本上是没有考虑任务复制的；而基于任务复制调度算法往往会产生过多的复制任务，增大系统开销，甚至有可能延迟计算时间。由于基于任务复制的任务调度是一个NP问题，因此本文提出了一种基于任务复制的网格资源调度算法，以减少调度长度为主要目标、减少任务复制量和资源占用量为次要目标。该算法在调度长度和任务复制数量以及占用资源数量方面都等于或优于其它算法。     

EMPEROR: An OGSA Grid Meta-Scheduler Based on Dynamic Resource Predictions

Scheduling constitutes an integral feature of Grid computing infrastructures, being also a key to realizing several of the Grid promises. In particular, scheduling can maximize the resources available to end users, accelerate the execution of jobs, while also supporting scalable and autonomic management of the resources comprising a Grid. Grid scheduling functionality hinges on middleware components called meta-schedulers, which undertake to automatically distribute jobs across the dispersed heterogeneous resources of a Grid. In this paper we present the design and implementation of a Grid meta-scheduler, which we call EMPEROR. EMPEROR provides a framework for implementing scheduling algorithms based on performance criteria. In implementing a particular instantiation of this framework, we have devised models for predicting host load and memory resources, and accordingly for estimating the running time of a task. These models hinge on time series analysis techniques and take into account results of the cluster computing literature. Apart from incorporating these models, EMPEROR provides fully fledged Grid scheduling functionality, which complies with OGSA standards as the later are reflected in the Globus toolkit. Specifically, EMPEROR interfaces to Globus middleware services (i.e., GSI, MDS, GRAM) towards discovering resources, implementing the scheduling algorithm and ultimately submitting jobs to local scheduling systems. By and large, EMPEROR is one of the few standards based meta-schedulers making use of dynamic scheduling information.     

Semantic-enabled CARE Resource Broker (SeCRB) for managing grid and cloud environment

Grid computing is mainly helpful for executing high-performance computing applications. However, conventional grid resources sometimes fail to offer a dynamic application execution environment and this increases the rate at which the job requests of users are rejected. Integrating emerging virtualization technologies in grid and cloud computing facilitates the provision of dynamic virtual resources in the required execution environment. Resource brokers play a significant role in managing grid and cloud resources as well as identifying potential resources that satisfy users’ application requests. This research paper proposes a semantic-enabled CARE Resource Broker (SeCRB) that provides a common framework to describe grid and cloud resources, and to discover them in an intelligent manner by considering software, hardware and quality of service (QoS) requirements. The proposed semantic resource discovery mechanism classifies the resources into three categories viz., exact, high-similarity subsume and high-similarity plug-in regions. To achieve the necessary user QoS requirements, we have included a service level agreement (SLA) negotiation mechanism that pairs users’ QoS requirements with matching resources to guarantee the execution of applications, and to achieve the desired QoS of users. Finally, we have implemented the QoS-based resource scheduling mechanism that selects the resources from the SLA negotiation accepted list in an optimal manner. The proposed work is simulated and evaluated by submitting real-world bio-informatics and image processing application for various test cases. The result of the experiment shows that for jobs submitted to the resource broker, job rejection rate is reduced while job success and scheduling rates are increased, thus making the resource management system more efficient.     

Predicting Job Failures in AuverGrid Based on Workload Log Analysis

Grid systems are popular today due to their ability to solve large problems in business and science. Job failures which are inherent in any computational environment are more common in grids due to their dynamic and complex nature. Furthermore, traditional methods for job failure recovery have proven costly and thus a need to shift toward proactive and predictive management strategies is necessary in such systems. In this paper, an innovative effort has been made to predict the futurity of jobs in a production grid environment. First of all, we investigated the relationship between workload characteristics and job failures by analyzing workload traces of AuverGrid which is a part of EGEE (Enabling Grids for E-science) project. After the recognition of failure patterns, the success or failure status of jobs during 6 months of AuverGrid activity was predicted with approximately 96% accuracy. The quality of services on the grid can be improved by integrating the result of this work into management services like scheduling and monitoring.     

Analysis of Scheduling and Replica Optimisation Strategies for Data Grids Using OptorSim

Many current international scientific projects are based on large scale applications that are both computationally complex and require the management of large amounts of distributed data. Grid computing is fast emerging as the solution to the problems posed by these applications. To evaluate the impact of resource optimisation algorithms, simulation of the Grid environment can be used to achieve important performance results before any algorithms are deployed on the Grid. In this paper, we study the effects of various job scheduling and data replication strategies and compare them in a variety of Grid scenarios using several performance metrics. We use the Grid simulator , and base our simulations on a world-wide Grid testbed for data intensive high energy physics experiments. Our results show that scheduling algorithms which take into account both the file access cost of jobs and the workload of computing resources are the most effective at optimising computing and storage resources as well as improving the job throughput. The results also show that, in most cases, the economy-based replication strategies which we have developed improve the Grid performance under changing network loads.     

Data Intensive and Network Aware (DIANA) Grid Scheduling

In Grids scheduling decisions are often made on the basis of jobs being either data or computation intensive: in data intensive situations jobs may be pushed to the data and in computation intensive situations data may be pulled to the jobs. This kind of scheduling, in which there is no consideration of network characteristics, can lead to performance degradation in a Grid environment and may result in large processing queues and job execution delays due to site overloads. In this paper we describe a Data Intensive and Network Aware (DIANA) meta-scheduling approach, which takes into account data, processing power and network characteristics when making scheduling decisions across multiple sites. Through a practical implementation on a Grid testbed, we demonstrate that queue and execution times of data-intensive jobs can be significantly improved when we introduce our proposed DIANA scheduler. The basic scheduling decisions are dictated by a weighting factor for each potential target location which is a calculated function of network characteristics, processing cycles and data location and size. The job scheduler provides a global ranking of the computing resources and then selects an optimal one on the basis of this overall access and execution cost. The DIANA approach considers the Grid as a combination of active network elements and takes network characteristics as a first class criterion in the scheduling decision matrix along with computations and data. The scheduler can then make informed decisions by taking into account the changing state of the network, locality and size of the data and the pool of available processing cycles.     

分布式环境中的多作业执行调度策略与优化

分布式大数据计算引擎是科研机构、互联网企业和政府部门处理大规模数据必不可少的工具,它们的使用和推广促进了各个领域的快速发展,为社会进步做出了巨大贡献。但是,在多作业处理的情况下,目前主流的大数据计算引擎在资源分配和作业调度方面仍有许多不足之处,它们通常对多作业平均划分内存资源并以先进先出FIFO的方式调度作业,这样简单的资源划分方式和作业调度机制并不能充分利用系统性能。针对此问题,从计算引擎的作业层面做出了改进：在资源划分方面,通过提取作业特征对作业的任务量进行预估,判断作业任务量和作业预分配资源间的差异,合并对集群资源浪费较高的作业,充分利用计算资源;在作业调度方面,对作业池中的作业进行特征提取,使用多路K-means算法对作业进行聚类分析,然后基于分析的结果,使用自平衡轮询调度算法对作业进行调度,达到负载均衡的目的。为了验证所提算法的有效性,使用大规模文本数据集在分布式集群环境中进行对比实验,实验结果表明,提出的作业合并算法和多作业调度算法可以减少5%~23%的作业运行时间,提高了7.5%~29%的系统吞吐量,在最好情况下可减少40%的线程启动数。     

结合价值密度和截止期的网格实时作业调度

在商业网格和云计算环境中,作业有到达时间、计算量、预算、截止期等参数,其中,预算是时间的函数。准确区分作业的重要性和紧迫性是作业调度系统的一个关键问题。综合利用这四个参数来定义作业的优先级,并提出基于价值密度和相对截止期的网格作业调度算法。分别对弱实时和强实时网格作业的调度进行仿真。仿真结果显示,所提出的调度算法的性能在两种情况下都优于所有对比算法的性能,且在强实时作业情况下优势更明显。     

Enhanced Dynamic Hierarchical Replication and Weighted Scheduling Strategy in Data Grid

The Data Grid provides massive aggregated computing resources and distributed storage space to deal with data-intensive applications. Due to the limitation of available resources in the grid as well as production of large volumes of data, efficient use of the Grid resources becomes an important challenge. Data replication is a key optimization technique for reducing access latency and managing large data by storing data in a wise manner. Effective scheduling in the Grid can reduce the amount of data transferred among nodes by submitting a job to a node where most of the requested data files are available. In this paper two strategies are proposed, first a novel job scheduling strategy called Weighted Scheduling Strategy (WSS) that uses hierarchical scheduling to reduce the search time for an appropriate computing node. It considers the number of jobs waiting in a queue, the location of the required data for the job and the computing capacity of the sites Second, a dynamic data replication strategy, called Enhanced Dynamic Hierarchical Replication (EDHR) that improves file access time. This strategy is an enhanced version of the Dynamic Hierarchical Replication strategy. It uses an economic model for file deletion when there is not enough space for the replica. The economic model is based on the future value of a data file. Best replica placement plays an important role for obtaining maximum benefit from replication as well as reducing storage cost and mean job execution time. So, it is considered in this paper. The proposed strategies are implemented by OptorSim, the European Data Grid simulator. Experiment results show that the proposed strategies achieve better performance by minimizing the data access time and avoiding unnecessary replication.     

多QoS约束网格作业调度问题的多目标演化算法

针对网格计算中的多QoS约束网格作业调度问题，以独立作业为研究对象，将其规约为多目标组合最优化问题．通过深入剖析多目标最优化理论及其演化算法，结合网格作业调度自然特征，提出了一种解决多QoS约束网格作业调度问题的多目标演化算法．该算法求解多个QoS维度效用函数指标的非劣解集，尝试解决多管理域间网格用户、资源管理者等网格实体的多目标协同问题．仿真结果表明，在时间维度、可靠性维度、安全性维度QoS效用值等用户级QoS指标，以及丢弃作业数等系统级指标方面该算法与QoS-Min-min和QoS-Sufferage等同类算法相比具有较好的综合性能．