A Tool for Prioritizing DAGMan Jobs and its Evaluation |
| |
Authors: | Grzegorz Malewicz Ian Foster Arnold L Rosenberg Michael Wilde |
| |
Affiliation: | (1) Department of Engineering, Google, Inc., Mountain View, CA 94043, USA;(2) Department of Computer Science, University of Chicago, Chicago, IL 60637, USA;(3) Department of Computer Science, University of Massachusetts Amherst, Amherst, MA 01003, USA;(4) Division of Mathematics and Computer Science, Argonne National Laboratory, Argonne, IL 60439, USA |
| |
Abstract: | It is often difficult to perform efficiently a collection of jobs with complex job dependencies due to temporal unpredictability
of the Grid. One way to mitigate the unpredictability is to schedule job execution in a manner that constantly maximizes the
number of jobs that can be sent to workers. A recently developed scheduling theory provides a basis to meet that optimization
goal. Intuitively, when the number of such jobs is always large, high parallelism can be maintained, even if the number of
workers changes over time in an unpredictable manner. In this paper we present the design, implementation, and evaluation
of a practical scheduling tool inspired by the theory. Given a DAGMan input file with interdependent jobs, the tool prioritizes
the jobs. The resulting schedule significantly outperforms currently used schedules under a wide range of system parameters,
as shown by simulation studies. For example, a scientific data analysis application, AIRSN, was executed at least 13% faster
with 95% confidence. An implementation of the tool was integrated with the Condor high-throughput computing system.
The research of G. Malewicz was supported in part by NSF Grant ITR-800864. The research of I. Foster and M. Wilde was supported
in part by the NSF GriPhyN project and by the Mathematical, Information, and Computational Sciences Division subprogram of
the Office of Advanced Scientific Computing Research, U.S. Department of Energy, under Contract W-31-109-Eng-38. The research
of A.L. Rosenberg was supported in part by NSF Grant CCF-0342417. |
| |
Keywords: | scheduling dags unpredictability systems Grid Condor applications of theory |
本文献已被 SpringerLink 等数据库收录! |
|