面向多用户环境的MapReduce集群调度算法研究

针对MapReduce集群现有调度策略在多用户环境下无法根据用户的实际资源需求实现动态资源分配的问题,提出了一种基于历史执行信息(HEI)的MapReduce集群调度算法——HEI Scheduler。该算法通过建立集群作业执行信息的收集和分析机制,得到各用户组资源需求随时间变化的规律,并以作业实际占用slot的时间作为作业占用资源量的衡量标准,进而动态地确定资源池的最小共享资源以及集群剩余资源分配的权值。实验结果表明,执行信息分析机制能够更准确地表征作业对资源的需求,采用集群调度算法HEI Scheduler能够有效地缩短作业的整体执行时间。

Research on a job scheduling algorithm for multi user MapReduce clusters

To solve the problem that the existing MapReduce scheduling strategy cannot realize dynamic resource allocation according to user''s actual resource demand in a multi-user environment, an algorithm for MapReduce clusters scheduling based on historical execution information(HEI), called the HEI scheduler, was proposed.The algorithm obtains the rules of the variation of each user group''s resource demand with time by establishing the mechanism for collection and analysis of cluster operation''s execution information, and uses operation''s actual slot occupying time to measure the occupied resource of operation to dynamically determine the minimum shared resource and weights of cluster remaining resource distribution.The experimental result indicates that the proposed execution information analysis mechanism can describe the resource demand of operation exactly.The MapReduce cluster scheduling algorithm based on historical execution information can effectively reduce the overall operation execution time.