基于任务进度感知的异构Hadoop云平台任务调度方案

针对异构Hadoop云计算平台的任务调度问题,对Hadoop 推测执行调度和LATE调度方案进行研究,提出一种基于任务进度感知的自适应任务调度方案。首先,根据当前计算节点上的任务进度情况,估计任务近似完成时间(ATE),以此确定掉队者(Straggler)任务。然后,以平均节点任务进度的25%为阈值,将节点分为慢节点和快节点。当Straggler后备任务达到一定阈值时,将其优先分配到快节点中执行。实验结果表明,提出的方案能够为异构Hadoop平台合理分配任务,有效降低了任务完成时间和响应延迟,同时提高了平台吞吐量。

A Task Scheduling Scheme in Heterogeneous Hadoop Cloud Platform Based on Task Progress Aware

For the issue that the task scheduling problem in heterogeneous Hadoop cloud computing platform, a task scheduling scheme based on task progress aware is proposed, which is based on the research of Hadoop speculative execution scheduling and LATE scheduling scheme. Firstly. the approximate task completion time (ATE) is estimated according to the task progress on computing nodes, so as to determine the Straggler task. Then, the node is divided into slow and fast nodes by the threshold value, which is 25% of average task progress. When the Straggler backup task reaches a certain threshold, it will be assigned to the fast node priority. The experimental results show that the proposed scheme can reasonably schedule tasks for heterogeneous Hadoop platforms, and effectively reduce the task completion time, response delay, and improve the throughput of the platform.