Hadoop平台公平调度算法研究与优化

Hadoop Map Reduce框架的公平调度算法以统一的固定配置文件管理计算节点上计算槽的数量,这不能保障集群负载均衡,亦不能满足不同用户的资源需求。针对公平调度算法配置方式的不足,提出一种动态反馈的调度算法。该算法结合公平调度算法预先分配的特性,能够对计算节点上的计算槽进行动态调整。实验结果表明,基于动态反馈的改进算法有效地提高了集群的执行效率。

Research and improve of fair scheduling algorithms based on Hadoop platform

Unified fixed configuration file is utilized in fair scheduling algorithm of the Hadoop MapReduce framework to calculate the number of slots in computing nodes. It can＇t guarantee the load balancing cluster especially in heterogeneous environment and satisfy the different requirement on the resource of different users. Aiming at the shortcomings of the existing configuration ways in fair scheduling algorithm, a dynamic feedback scheduling algorithm is proposed. Combined with the characteristics of pre-allocated algorithm, the computing nodes on the slots can be adjusted dynamically. The experimental results shows that the improved algorithm based on dynamic feedback can efficiently improve the execution effficiency of the cluster.