一种Hadoop Yarn的资源调度方法研究

针对Hadoop Yarn资源调度问题,为提高集群作业执行效率,提出一种基于蚁群算法与粒子群算法的自适应Hadoop资源调度算法SRSAPH.SRSAPH中,通过Hadoop Yarn跳通信机制获取负载、内存、CPU速度等属性信息初始化信息素矩阵;同时,将粒子群算法的自我认知能力与社会认知能力引入到蚁群算法,提高算法的收敛速度;此外,根据蚁群算法全局最优解的波动趋势动态调整信息素挥发系数,提高解的精度.实验表明,采用SRSAPH进行资源调度,集群的作业执行时间缩短至少10%.

A Study on Scheduling Method of Hadoop Yarn

In view of the resource scheduling problem of Hadoop Yarn,to improve the execution efficiency of the cluster job,we propose a Self-adapt Resource Scheduling algorithm based on Ant Colony Algorithm and Particle Swarm Al-gorithm in Hadoop (SRSAPH).In SRSAPH,we initialize the pheromone matrix of SRSAPH by using the attribute informa-tion of load,memory,and CPU speed obtained through the heartbeat message transfer mechanism.Meanwhile,we introduce the self-cognitive ability and social cognition ability of particle swarm algorithm into the ant colony algorithm to speed up the rate of convergence of the algorithm.Moreover,we dynamically adjust the pheromone evaporation rate based on the fluctua-tion trends of global optimal solution to enhance the accuracy of the solutions.Experimental result shows that by using SR-SAPH in resource scheduling,the execution time of cluster job is shorten by 1 0%.