Hadoop平台下改进的推测任务调度算法

研究对比Hadoop平台下默认的推测任务调度算法和异构环境下LATE调度算法的优势和不足,提出了一种基于Hadoop集群的改进的推测任务调度算法.该算法以节点历史信息对Reduce任务各阶段比例进行动态调整和更新,并对任务实时处理速率进行局部平滑处理来提高预估任务剩余完成时间的准确性,最后采用MCP模型对备份任务有效性进行验证.通过实验结果分析可知:该算法能够有效提升备份任务成功率,减少作业完成时间.

Improved speculative task scheduling algorithm on Hadoop platform

After compare and analysis the advantage and disadvantage of the default speculative task scheduling algorithm on the Hadoop platform and LATE scheduling algorithm in heterogeneous environment,an improved speculative task scheduling algorithm based on Hadoop computer cluster is proposed.The algorithm dynamically adjusts and updates the proportion of stages in the Reduce task based on node history information,smoothes the real-time task progress rate to improve the forecast accuracy of the remaining completion time,and applies MCP model to verify the effectiveness for backup task.The experiment analysis show that the algorithm can improve the success rate of backup task and reduce job completion time.