基于节点识别的慢任务调度算法

为了降低大数据处理集群在执行任务过程中的慢任务对作业执行效率的影响，提出了一种识别慢任务、备份慢任务、减少慢任务相结合的调度算法——TQST算法。首先，通过判断节点能力和任务执行时间，建立慢节点、非常慢节点和慢任务队列；其次，根据预判备份执行价值确定如何启动慢任务的备份任务，提高了备份执行的作用；然后，在节点识别的基础上，规避为非常慢节点分配任务，从根本上减少慢任务的产生，提高作业执行效率。实验结果表明，TQST算法在作业响应时间等方面优于已有的慢任务调度算法。

Slow task scheduling algorithm based on node identification

In order to reduce the influence of the slow task, produced in big data processing,a recognition slow task, speculative slow task, reduce the combination of slow task scheduling algorithm (TQST)was proposed. First of all, through the judgment of node ability and task execution time, slow node queue, very slow node queue and slow task queue were established; Secondly, according to the anticipation speculative execution value to decide how to start speculative task; Then, in the basis of node identification, avoid distributing tasks to very slow node, radically reduce slow task production, improve job execution efficiency. The experimental results show that TQST algorithm previous existing slow task scheduling algorithm in term of the job response time.