一种异构环境下的基于MapReduce任务调度改进机制

针对在异构环境下采用现有MapReduce任务调度机制可能出现各计算节点间数据迁移和系统资源分配难以管理的问题, 提出一种动态的任务调度机制来改善这些问题。该机制先根据节点的计算能力按比例放置数据, 然后通过资源预测方法估计异构环境下MapReduce任务的完成时间, 并根据完成时间计算任务所需的资源。实验结果表明, 该机制提高了异构环境下任务的数据本地性比例, 且能动态地调整资源分配, 以保证任务在规定时间内完成, 是一种有效可行的任务调度机制。

Improved task scheduling mechanism for MapReduce inheterogeneous environment

The existing MapReduce scheduling mechanism that used in heterogeneous environment may lead to the migration of data between compute nodes, and manage system resource allocation difficulty. This paper proposed a dynamic task scheduling mechanism to improve these problems. First, this mechanism distributed data in proportion according to the computing capacity of each node. Then it estimated MapReduce job completion time in heterogeneous environment by using resource prediction model, and calculated task slot requirements based on its completion time. The experiment results show that this mechanism can improve the proportion of task data locality, and dynamically allocate resources in order to ensure the job is completed in the the specified time. It is demonstrated that this task scheduling mechanism is effective and feasible.