Hadoop下基于统计最优的资源调度算法

云计算集群中的资源存在异构和节点稳定性问题.异构资源的计算能力不同会导致较突出的作业任务同步问题,而某个节点的不稳定状态会使运行于该节点的任务大量备份或重新计算.针对上述两问题将严重影响集群作业的执行进度,在Hadoop平台下利用统计方法,提出一种资源调度算法,对计算资源较少的节点和不稳定状态的节点进行标志并降权,让集群尽可能调度资源较好的稳定节点.实验结果表明,该算法能够在一定程度上减少作业的周转时间,提高集群的效率和吞吐量.

Resource scheduler algorithm based on statistical optimization under Hadoop

There are node stability problems in a heterogeneous cloud clusters. The heterogeneous resources' differences in computing ability will lead to prominent sync issues of job' task, and a unsteady node will make the task which is running in the node been backup or been recount. The two problems above will seriously affect the execution of the cluster's jobs progress. According to the situation, now using the Hadoop workbench with statistical method, this paper put forward a kind of resource scheduling algorithm which marked and dropped the less computing resources node and instability node rightly , letting cluster scheduling resources better stability node as much as possible. The experimental results show that, the algorithm can reduce the jobs cycle time in a certain degree, and improve the efficiency of the cluster and throughput.