Data Intensive and Network Aware (DIANA) Grid Scheduling

In Grids scheduling decisions are often made on the basis of jobs being either data or computation intensive: in data intensive situations jobs may be pushed to the data and in computation intensive situations data may be pulled to the jobs. This kind of scheduling, in which there is no consideration of network characteristics, can lead to performance degradation in a Grid environment and may result in large processing queues and job execution delays due to site overloads. In this paper we describe a Data Intensive and Network Aware (DIANA) meta-scheduling approach, which takes into account data, processing power and network characteristics when making scheduling decisions across multiple sites. Through a practical implementation on a Grid testbed, we demonstrate that queue and execution times of data-intensive jobs can be significantly improved when we introduce our proposed DIANA scheduler. The basic scheduling decisions are dictated by a weighting factor for each potential target location which is a calculated function of network characteristics, processing cycles and data location and size. The job scheduler provides a global ranking of the computing resources and then selects an optimal one on the basis of this overall access and execution cost. The DIANA approach considers the Grid as a combination of active network elements and takes network characteristics as a first class criterion in the scheduling decision matrix along with computations and data. The scheduler can then make informed decisions by taking into account the changing state of the network, locality and size of the data and the pool of available processing cycles.     

Multi-objective real-time dispatching for integrated delivery in a Fab using GA based simulation optimization

In a wafer fabrication Fab, the “integrated delivery”, which integrates the automated material handling system (AMHS) with processing tools to automate the material flow, is difficult to implement due to the system complexity and uncertainty. The previous dispatching studies in semiconductor manufacturing have mainly focused on the tool dispatching. Few studies have been done for analyzing combinatorial dispatching rules including lot dispatching, batch dispatching and automated guided vehicle (AGV) dispatching. To handle this problem, a GA (genetic algorithm) based simulation optimization methodology, which consists of the on-line scheduler and the off-line scheduler, is presented in this paper. The on-line scheduler is used to monitor and implement optimal combinatorial dispatching rules to the semiconductor wafer fabrication system. The off-line scheduler is employed to search for optimal combinatorial dispatching rules. In this study, the response surface methodology is adopted to optimize the GA parameters. Finally, an experimental bay of wafer fabrication Fab is constructed and numerical experiments show that the proposed approach can significantly improve the performance of the “integrated delivery system” compared with the traditional single dispatching rule approach.     

Performance and energy aware cluster-level scheduling of compute-intensive jobs with unknown service times

In this paper we examine three local resource allocation policies, which are based on shortest queue, in a cluster with heterogeneous servers. Two of them are optimized for performance and the third one is optimized for energy conservation. We assume that there are two types of processors in the cluster, with different performance and energy characteristics. We consider that service times of jobs are unknown to the scheduler. A simulation model is used to evaluate the performance and energy behavior of the policies. Simulation results indicate that the differences among the policies depend on system load and there is a trade-off between performance and energy consumption.     

Using a planning scheduler to improve the flexibility of real-time fieldbus networks

A typical approach to real-time fieldbus arbitration is to use an off-line scheduler that generates a cyclic static table containing the allocation of bus-time-slots to the transaction of process-control variables. This approach, e.g. as used in the Factory Instrumentation Protocol fieldbus, is rather inflexible in the sense that any system changes, such as the addition of a sensor, requires an interruption of the fieldbus operation. In this paper the use of a planning scheduler is proposed to overcome such inflexibility. This scheduler compromises between the advantages and disadvantages of typical dynamic and static scheduling. A sufficient schedulability condition is also derived, in order to overcome the typical inability of dynamic (or even planning) schedulers to guarantee schedulability for long-term system operation. The evaluation of this condition incurs very small run time overhead, and can therefore be used with advantage in a fieldbus system that relies on the planning scheduler. An experimental test is described, to illustrate how the planning scheduler works.     

A metric of fairness for parallel job schedulers

Fairness is an important aspect in queuing systems. Several fairness measures have been proposed in queuing systems in general and parallel job scheduling in particular. Generally, a scheduler is considered unfair if some jobs are discriminated whereas others are favored. Some of the metrics used to measure fairness for parallel job schedulers can imply unfairness where there is no discrimination (and vice versa). This makes them inappropriate. In this paper, we show how the existing approach misrepresents fairness in practice. We then propose a new approach for measuring fairness for parallel job schedulers. Our approach is based on two principles: (i) as jobs have different resource requirements and find different queue/system states, they need not have the same performance for the scheduler to be fair and (ii) to compare two schedulers for fairness, we make comparisons of how the schedulers favor/discriminate individual jobs. We use performance and discrimination trends to validate our approach. We observe that our approach can deduce discrimination more accurately. This is true even in cases where the most discriminated jobs are not the worst performing jobs. Copyright © 2008 John Wiley & Sons, Ltd.     

A neural network job-shop scheduler

This paper focuses on the development of a neural network (NN) scheduler for scheduling job-shops. In this hybrid intelligent system, genetic algorithms (GA) are used to generate optimal schedules to a known benchmark problem. In each optimal solution, every individually scheduled operation of a job is treated as a decision which contains knowledge. Each decision is modeled as a function of a set of job characteristics (e.g., processing time), which are divided into classes using domain knowledge from common dispatching rules (e.g., shortest processing time). A NN is used to capture the predictive knowledge regarding the assignment of operation’s position in a sequence. The trained NN could successfully replicate the performance of the GA on the benchmark problem. The developed NN scheduler was then tested against the GA, Attribute-Oriented Induction data mining methodology and common dispatching rules on a test set of randomly generated problems. The better performance of the NN scheduler on the test problem set compared to other methods proves the feasibility of NN-based scheduling. The scalability of the NN scheduler on larger problem sizes was also found to be satisfactory in replicating the performance of the GA.     

Training a neural network to select dispatching rules in real time

Dispatching rules are often suggested to schedule manufacturing systems in real-time. Numerous dispatching rules exist. Unfortunately no dispatching rule (DR) is known to be globally better than any other. Their efficiency depends on the characteristics of the system, operating condition parameters and the production objectives. Several authors have demonstrated the benefits of changing dynamically these rules, so as to take into account the changes that can occur in the system state. A new approach based on neural networks (NN) is proposed here to select in real time, each time a resource becomes available, the most suited DR. The selection is made in accordance with the current system state and the workshop operating condition parameters. Contrarily to the few learning approaches presented in the literature to select scheduling heuristics, no training set is needed. The NN parameters are determined through simulation optimization. The benefits of the proposed approach are illustrated through the example of a simplified flow-shop already published. It is shown that the NN can automatically select efficient DRs dynamically: the knowledge is only generated from simulation experiments, which are driven by the optimization method. Once trained offline, the resulting NN can be used online, in connection with the monitoring system of a flexible manufacturing system.     

最小化多MapReduce任务总完工时间的分析模型及其应用

随着大规模的MapReduce集群广泛地用于大数据处理,特别是当有多个任务需要使用同一个Hadoop集群时,一个关键问题是如何最大限度地减少集群的工作时间,提高MapReduce作业的服务效率。可将多个MapReduce作业当做一个调度任务建模,观察发现多个任务的总完工时间和任务的执行顺序有密切关系。 研究目标是设计作业调度系统分析模型,最小化一批MapReduce作业的总完工时间。提出一个更好的调度策略和实现方法, 使整个调度系统符合经典Johnson算法的条件, 从而可使用经典Johnson算法在线性时间内获取总完工时间的最优解。同时,针对需要使用两个或多个资源池进行平衡的问题, 提出了一种线性时间解决方案, 优于已知的近似模拟方案。该理论模型可应用于提高系统响应速度、节能和负载均衡等方面, 对应的应用实例提供了证实。     

Job scheduling in mesh multicomputers

A new approach for dynamic job scheduling in mesh-connected multiprocessor systems, which supports a multiuser environment, is proposed in this paper. Our approach combines a submesh reservation policy with a priority-based scheduling policy to obtain high performance in terms of high throughput, high utilization, and low turn-around times for jobs. This high performance is achieved at the expense of scheduling jobs in a strictly fair, FCFS fashion; in fact, the algorithm is parameterized to allow trade-offs between performance and (short-term) POPS fairness. The proposed scheduler can be used with any submesh allocation policy. A fast and efficient implementation of the proposed scheduler has also been presented. The performance of the proposed scheme has been compared with the FCFS policy, the only existing scheduling strategy for meshes, to demonstrate the effectiveness of the proposed approach. Simulation results indicate that our scheduling strategy outperforms the FCFS policy significantly. Specifically, our strategy significantly reduces the average waiting delay of jobs over the FCFS policy. The fast implementation of the proposed scheduler results in low allocation and deallocation time overhead, as well as low space overhead     

Dynamic scheduling of a batch of parallel task jobs on heterogeneous clusters

This paper addresses the problem of minimizing the scheduling length (make-span) of a batch of jobs with different arrival times. A job is described by a direct acyclic graph (DAG) of parallel tasks. The paper proposes a dynamic scheduling method that adapts the schedule when new jobs are submitted and that may change the processors assigned to a job during its execution. The scheduling method is divided into a scheduling strategy and a scheduling algorithm. We also propose an adaptation of the Heterogeneous Earliest-Finish-Time (HEFT) algorithm, called here P-HEFT, to handle parallel tasks in heterogeneous clusters with good efficiency without compromising the makespan. The results of a comparison of this algorithm with another DAG scheduler using a simulation of several machine configurations and job types shows that P-HEFT gives a shorter makespan for a single DAG but scores worse for multiple DAGs. Finally, the results of the dynamic scheduling of a batch of jobs using the proposed scheduler method showed significant improvements for more heavily loaded machines when compared to the alternative resource reservation approach.     

An Agent-Based Approach for Scheduling Multiple Machines

We present a new agent-based solution approach for the problem of scheduling multiple non-identical machines in the face of sequence dependent setups, job machine restrictions, batch size preferences, fixed costs of assigning jobs to machines and downstream considerations. We consider multiple objectives such as minimizing (weighted) earliness and tardiness, and minimizing job-machine assignment costs. We use an agent-based architecture called Asynchronous Team (A-Team), in which each agent encapsulates a different problem solving strategy and agents cooperate by exchanging results. Computational experiments on large instances of real-world scheduling problems show that the results obtained by this approach are significantly better than any single algorithm or the scheduler alone. This approach has been successfully implemented in an industrial scheduling system.     

基于数字孪生的智能车间系统仿真加速测试方法

为解决当前制造系统软件可靠性仿真测试时间长、测试环境难以搭建等问题,提出采用数字孪生技术与智能车间系统仿真加速测试相结合的方法;建立智能车间高保真数字孪生模型替代现实生产车间系统用于制造系统软件的可靠性仿真测试,首先要构建包含产品、设备资源、工艺流程等系统级仿真模型;同时,为仿真车间生产事件流程,在模型中,还需结合生产实际情况,设置设备间通信协议、通信数据以及生产线事件及队列顺序,真实模拟系统运行环境;通过构建步进电机产线数字孪生模型,仿真加工装配流程,运行智能车间系统软件,采用仿真时钟推进机制开展加速测试,验证了该方法的有效性和实用性,对开展工业系统软件高保真快速测试评估具有一定的借鉴意义。     

Competitive online adaptive scheduling for sets of parallel jobs with fairness and efficiency

We study online adaptive scheduling for multiple sets of parallel jobs, where each set may contain one or more jobs with time-varying parallelism. This two-level scheduling scenario arises naturally when multiple parallel applications are submitted by different users or user groups in large parallel systems, where both user-level fairness and system-wide efficiency are of important concerns. To achieve fairness, we use the well-known equi-partitioning algorithm to distribute the available processors among the active job sets at any time. For efficiency, we apply a feedback-driven adaptive scheduler that periodically adjusts the processor allocations within each set by consciously exploiting the jobs’ execution history. We show that our algorithm achieves asymptotically competitive performance with respect to the set response time, which incorporates two widely used performance metrics, namely, total response time and makespan, as special cases. Both theoretical analysis and simulation results demonstrate that our algorithm improves upon an existing scheduler that provides only fairness but lacks efficiency. Furthermore, we provide a generalized framework for analyzing a family of scheduling algorithms based on feedback-driven policies with provable efficiency. Finally, we consider an extended multi-level hierarchical scheduling model and present a fair and efficient solution that effectively reduces the problem to the two-level model.     

Efficient meta-heuristics based on various dominance criteria for a single-machine bi-criteria scheduling problem with rejection

In this paper, we try to fill in the gap between theory and practice in production scheduling by defining a new term as “rejection” and treating the corresponding scheduling problem with multi-objective optimization approach. We study a bi-objective single machine scheduling problem with rejection. At the beginning of scheduling time horizon, scheduler needs to decide which job shall be rejected due to the resource constraints regarding two objective functions: minimization of total weighted completion time of accepted jobs and total rejection penalty of rejected jobs. We develop different algorithms to find the best estimation of Pareto-optimal front for this problem. In order to improve the quality of the solutions, on the one hand, and facilitate the process of selecting best solution for the final decision maker, on the other hand, we integrate various dominance criteria into our proposed algorithms. Finally we compare the performance of those methods by testing on a large set of instances and highlight the advantages and weak points of each one.     

Dynamic I/O-Aware Scheduling for Batch-Mode Applications on Chip Multiprocessor Systems of Cluster Platforms简

Efficiency of batch processing is becoming increasingly important for many modern commercial service centers, e.g., clusters and cloud computing datacenters. However, periodical resource contentions have become the major performance obstacles for concurrently running applications on mainstream CMP servers. I/O contention is such a kind of obstacle, which may impede both the co-running performance of batch jobs and the system throughput seriously. In this paper, a dynamic I/O-aware scheduling algorithm is proposed to lower the impacts of I/O contention and to enhance the co-running performance in batch processing. We set up our environment on an 8-socket, 64-core server in Dawning Linux Cluster. Fifteen workloads ranging from 8 jobs to 256 jobs are evaluated. Our experimental results show significant improvements on the throughputs of the workloads, which range from 7% to 431%. Meanwhile, noticeable improvements on the slowdown of workloads and the average runtime for each job can be achieved. These results show that a well-tuned dynamic I/O-aware scheduler is beneficial for batch-mode services. It can also enhance the resource utilization via throughput improvement on modern service platforms.     

风电场数据中心Hadoop云平台作业调度算法研究

风电场数据中心包含状态监测、数据采集等实时类作业和非实时类作业,采用C/S结构存在资源利用率不平衡、管理与维护成本高等缺点。设计了一种基于Hadoop云平台的数据中心架构;针对开源Hadoop平台现有FIFO调度器不能满足实时监测系统要求,在原有FIFO调度器的基础上,设计了一种双队列的作业调度器,综合考虑作业的截止时间和优先级来进行作业调度决策,实验结果表明,与FIFO调度器相比,双队列的作业调度器在集群负载较大时能够表现出较好的性能,保证实时类作业能够优先执行,为风电机组的安全运行提供保障。     

Design and implementation of an adaptive job allocation strategy for heterogeneous multi‐cluster computing systems

Cluster computing is an attractive approach to provide high‐performance computing for solving large‐scale applications. Owing to the advances in processor and networking technology, expanding clusters have resulted in the system heterogeneity; thus, it is crucial to dispatch jobs to heterogeneous computing resources for better resource utilization. In this paper, we propose a new job allocation system for heterogeneous multi‐cluster environments named the Adaptive Job Allocation Strategy (AJAS), in which a self‐scheduling scheme is applied in the scheduler to dispatch jobs to the most appropriate computing resources. Our strategy focuses on increasing resource utility by dispatching jobs to computing nodes with similar performance capacities. By doing so, execution times among all nodes can be equalized. The experimental results show that AJAS can improve the system performance. Copyright © 2011 John Wiley & Sons, Ltd.     

A new window-based job scheduling scheme for 2D mesh multicomputers

Allocating submeshes to jobs in mesh-connected multicomputers in a FCFS fashion can lead to poor system performance (e.g., long job waiting delays) because the job at the head of the waiting queue can prevent the allocation of free submeshes to other waiting jobs with smaller submesh requirements. However, serving jobs aggressively out-of-order can lead to excessive waiting delays for jobs with large allocation requests. In this paper, we propose a scheduling scheme that uses a window of consecutive jobs from which it selects jobs for allocation and execution. This window starts with the current oldest waiting job and corresponds to the lookahead of the scheduler. The performance of the proposed window-based scheme has been compared to that of FCFS and other previous job scheduling schemes. Extensive simulation results based on synthetic workloads and real workload traces indicate that the new scheduling strategy exhibits good performance when the scheduling window size is large. In particular, it is substantially superior to FCFS in terms of system utilization, average job turnaround times, and maximum waiting delays under medium to heavy system loads. Also, it is superior to aggressive out-of-order scheduling in terms of maximum job waiting delays. Window-based job scheduling can improve both overall system performance and fairness (i.e., maximum job waiting delays) by adopting large lookahead job scheduling windows.     

Provably Efficient Online Nonclairvoyant Adaptive Scheduling

Multiprocessor scheduling in a shared multiprogramming environment can be structured in two levels, where a kernel-level job scheduler allots processors to jobs and a user-level thread scheduler maps the ready threads of a job onto the allotted processors. We present two provably-efficient two-level scheduling schemes called G-RAD and S-RAD respectively. Both schemes use the same job scheduler RAD for the processor allotments that ensures fair allocation under all levels of workload. In G-RAD, RAD is combined with a greedy thread scheduler suitable for centralized scheduling; in S-RAD, RAD is combined with a work-stealing thread scheduler more suitable for distributed settings. Both G-RAD and S-RAD are non-clairvoyant. Moreover, they provide effective control over the scheduling overhead and ensure efficient utilization of processors. We also analyze the competitiveness of both G-RAD and S-RAD with respect to an optimal clairvoyant scheduler. In terms of makespan, both schemes can achieve O(1)-competitiveness for any set of jobs with arbitrary release time. In terms of mean response time, both schemes are O(1)-competitive for arbitrary batched jobs. To the best of our knowledge, G-RAD and S-RAD are the first non-clairvoyant scheduling algorithms that guarantee provable efficiency, fairness and minimal overhead.     

Balancing and simulating of assembly line with overlapped and stopped operation

In order to avoid undesirable idling or fuss of the manufacturing resources, the assembly workshop’s scheduler evaluates the effect of different online sequence of parts on production cycle, balances workload and utilization ratio, minimizes span of the assembly line. This paper studies modeling and simulation of assembly line with overlapped and stopped operation, builds mathematical model for the assembly line both under certainty and uncertainty environment. Assembly line problem with these two conditions is simulated, and a simulation prototype system is developed. The assembly line simulation at assembly workshop in an auto company of Beijing validates the effectiveness and correctness of the prototype system.