一种多用户MapReduce集群的作业调度算法的设计与实现

随着更多的企业开始使用数据密集型集群计算系统如Hadoop和Dryad实现了更多的应用,多用户间共享MapRe-duce集群这种既减少了建立独立集群的代价,同时又使得多用户间可以共享更多的大数据集资源的需求日益增多。在公平调度算法的基础上,结合槽分配延迟和优先级的技术,本文提出了一种改进算法,可以实现更好的数据本地性,改善整个系统的计算性能如吞吐率、响应时间等;同时为了满足差别化的商业服务,通过对用户设置相应的优先级保证紧急任务的完成。

Design and Implementation of Job Scheduling Algorithm for Multi-User MapReduce Clusters

As more enterprises start to use data-intensive cluster computing systems such as Hadoop and Dryad for more applications,sharing MapReduce clusters among multiple users that reducing the cost of establishing an independent cluster and the demand of sharing common data sets resources for users is increasing.Based on fair scheduling algorithm,combining with slot allocation delay and priority technology,the paper proposes an improved algorithm.It can achieve better data locality,improve the performance of the system,such as throughput,response time.To meet the differentiated business services,it sets the appropriate for users to ensure special tasks.