首页 | 本学科首页   官方微博 | 高级检索  
     

异构环境下自适应的Map-Reduce调度
引用本文:陈全,邓倩妮.异构环境下自适应的Map-Reduce调度[J].计算机工程与科学,2009,31(Z1).
作者姓名:陈全  邓倩妮
作者单位:上海交通大学计算机科学与工程系,上海,200240
基金项目:自然科学基金资助项目 
摘    要:本文针对Hadoop自带的Map-Reduce调度器和LATE调度器的不足,提出了自适应的Map-Reduce调度方式。通过历史信息动态调整Map和Reduce任务各阶段的时间比例,以找出真正需要启动备份任务的任务。并将节点进一步分类,在快速节点上启动落后任务的备份任务,从而减小响应时间,并尽量避免节点空载,以免浪费系统资源。最后通过具体实验验证了自适应的Map-Reduce调度的有效性,从历史信息中学习的方式在实验中能减少15%左右的响应时间,综合各种技术,能够减少25%左右的响应时间。

关 键 词:Map-Reduce  异构环境  调度算法  自适应

Self-Adaptive Map-Reduce Scheduling Under Heterogeneous Environment
CHEN Quan,DENG Qian-ni.Self-Adaptive Map-Reduce Scheduling Under Heterogeneous Environment[J].Computer Engineering & Science,2009,31(Z1).
Authors:CHEN Quan  DENG Qian-ni
Abstract:This paper proposes a self-adaptive Map-Reduce scheduling approach, according to drawbacks of Hadoop's Map-Reduce scheduler and LATE scheduler. For finding the correct tasks which need back-up tasks, in the proposed methods, the scheduler dynamically adjusts the time proportions of each stage of Map and Reduce tasks according to historical information. And the scheduler further classes all the nodes, for launching backup tasks of slow tasks on fast nodes. Thereby, reducing response time and trying best not to leave nodes idle, so as not to waste system resources. Through specific experiments, this paper verifies that the method of learning from historical information can decrease the response time by 15% in the test cluster. And by using all the methods which has proposed in this paper, the response time will decrease about 25%.
Keywords:Map-Reduce
本文献已被 万方数据 等数据库收录!
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号