首页 | 本学科首页   官方微博 | 高级检索  
     

Hadoop平台下自响应故障感知的检测技术研究实现
引用本文:赵序光,张树东,严业平.Hadoop平台下自响应故障感知的检测技术研究实现[J].计算机测量与控制,2016,24(11).
作者姓名:赵序光  张树东  严业平
作者单位:首都师范大学 信息工程学院,首都师范大学信息工程学院,首都师范大学信息工程学院
基金项目:本论文由国家自然科学基金(项目编号:31571563),北京市属高等学校创新团队建设与教师职业发展计划项目资助。高可靠嵌入式系统技术北京市工程研究中心资助。
摘    要:当今云计算环境下,Hadoop已经成为大数据处理的事实标准。然而云计算具有大规模、高复杂和动态性的特点,容易导致故障的发生,影响Hadoop上运行的作业。虽然Hadoop具有内置的故障检测和恢复机制,但云环境中不同节点负载大小的变化,被调度的作业仍然导致失败。针对此问题提出自响应故障感知的检测调度方法,对异构环境负载能力的不同,而做出服务器快节点和慢节点的判断,把作业分配调度到合适的节点上执行,调整任务决策来尽可能的防止任务失败的发生。最后在Hadoop框架下与基本调度器进行实验性能比较,结果显示该方法减少作业失败率最高达19%,并缩短了作业执行时间,同时也减少CPU和内存的使用。

关 键 词:Hadoop  异构环境  负载  调度器  作业
收稿时间:2016/5/9 0:00:00
修稿时间:2016/6/8 0:00:00

Implementation of detection technology of self response fault detection based on Hadoop platform
Affiliation:Institute of Information Engineering , Capital Normal University,,
Abstract:In today''s cloud computing environment, Hadoop has become the fact standard for big data processing. However, cloud computing has the characteristics of large scale, high complexity and dynamic characteristics, fault occurrence is common, but it often affects the operation of jobs on the Hadoop. Although the Hadoop has a built-in fault detection and recovery mechanism, but the cloud environment, the changes in the load of different nodes, the job is scheduled to still lead to failure. The proposed self response of fault aware scheduling detection method, according to different load capacity of heterogeneous environment, and make fast server nodes and slow nodes judgment, the scheduling of job allocated to the appropriate node , adjust the decision task to prevent mission failure occurred. Finally in the Hadoop framework and basic scheduler were experimental performance compared results show that the method reduce job failure rate of up to 19% and shorten job execution time, and also reduce the CPU and memory usage.
Keywords:Hadoop  Heterogeneous environment  load balance  Scheduler  Job
点击此处可从《计算机测量与控制》浏览原始摘要信息
点击此处可从《计算机测量与控制》下载全文
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号