共查询到19条相似文献,搜索用时 200 毫秒
1.
《计算机科学》2014,(Z1)
Hadoop已成为研究云计算的基础平台,MapReduce是其大数据分布式处理的计算模型。针对异构集群下MapReduce数据分布、数据本地性、作业执行流程等问题,提出一种基于DAG的MapReduce调度算法。把集群中的节点按计算能力进行划分,将MapReduce作业转换成DAG模型,改进向上排序值计算方法,使其在异构集群中计算更精准、任务的优先级排序更合理。综合节点的计算能力与数据本地性及集群利用情况,选择合理的数据节点分配和执行任务,减少当前任务完成时间。实验表明,该算法能合理分布数据,有效提高数据本地性,减少通信开销,缩短整个作业集的调度长度,从而提高集群的利用率。 相似文献
2.
针对现有Hadoop难以适应异构资源环境的不足,提出一种自适应MapReduce调度器:CloudMR.基于数据局部性,CloudMR将同一机架内的对进行本地归约合并,减少中间结果中对的数目,从而减少机架间的数据传送.根据资源性能和任务特征,CloudMR动态确定节点任务槽数和数据分配量.对于计算性能高的节点,CloudMR分配较多的任务和数据量,而对于计算性能低的节点,相应地减轻任务和数据量负载.实验表明,在异构环境下,较之现有Hadoop,CloudMR减少了节点间数据传输和备份任务运行,缩短了作业完成时间. 相似文献
3.
4.
5.
针对云计算环境下存在密码服务请求算法种类多、资源需求差异化和节点性能异构等问题。为了提高系统的可靠性,保证服务质量,综合考虑用户请求任务和处理节点等多种因素,在作业包截止时间的基础上,通过任务映射策略完成密码服务的一级调度。设计一个基于用户优先级和任务等待时间的任务优先级调度算法实现二级调度,从而构造一种同时支持多种密码服务请求和任务动态可调整的调度系统框架,以保证云环境下任务的时效性。仿真结果表明,该系统有较好的执行效率和负载分布效果,达到设计目标。与随机法和遗传算法相比,其执行效率分别提高了17%和11%左右。 相似文献
6.
MapReduce编程模型被广泛应用于大数据处理平台,而一个有效的任务调度算法对模型的运行效率至关重要。将MapReduce工作流的Map和Reduce阶段分别拆解为若干个有先后序限定关系的作业,每个作业再拆解为多个任务。之后基于计算集群的可用资源和任务异构性,构建面向作业和任务的2级有向无环图(DAG)模型,同时提出基于2级优先级排序的异构调度算法2-MRHS。算法的第1阶段进行优先级排序,即对作业和任务分别进行优先权值计算,再汇总得到任务的调度队列;第2阶段进行任务分配,即基于最快完成时间将每个任务所包含的数据块子任务分配给最适合的计算结点。采用大批量随机生成的DAG模型进行实验,结果表明与其他相关算法相比,本文算法有更短的调度长度(makespan)且更加稳定。 相似文献
7.
当今云计算环境下,Hadoop已经成为大数据处理的事实标准。然而云计算具有大规模、高复杂和动态性的特点,容易导致故障的发生,影响Hadoop上运行的作业。虽然Hadoop具有内置的故障检测和恢复机制,但云环境中不同节点负载大小的变化,被调度的作业仍然导致失败。针对此问题提出自响应故障感知的检测调度方法,对异构环境负载能力的不同,而做出服务器快节点和慢节点的判断,把作业分配调度到合适的节点上执行,调整任务决策来尽可能的防止任务失败的发生。最后在Hadoop框架下与基本调度器进行实验性能比较,结果显示该方法减少作业失败率最高达19%,并缩短了作业执行时间,同时也减少CPU和内存的使用。 相似文献
8.
9.
MapReduce是一个能够对大规模数据进行分布式处理的框架,目前被各个领域广泛应用。在提供MapReduce服务的集群中,如何保证不同优先级用户的截止时间限定是MapReduce作业调度问题的一个挑战。针对这一问题,提出了一个基于排队网络的多优先级作业调度算法(MPSA)。首先分析和归纳了基于MapReduce模型的算法,提出了三种常见模式,采用Jackson排队网络对基于MapReduce模型的算法建立了数学模型,应用该网络模型可以求出不同优先级队列对资源的需求;随后使用AR(1)模型进行预测,使算法可以动态地适应不同的用户访问量;利用二分查找算法,分步计算出不同优先级在map阶段和reduce阶段分配的槽位数;最后实现了在MapReduce模型中应用的实时调度算法。实验结果表明,与传统的FIFO和公平调度算法相比,本文提出的算法在用户到达率和任务规模变化的情况下,可以更加有效地满足不同优先级用户的截止时间限定。 相似文献
10.
为了解决当前Hadoop集群在异构资源环境下固有的调度分配方法的不足,提出了一种基于节点能力的自适应调度算法NCAS(node capacity adaptive scheduling)。首先,NCAS算法根据节点性能、任务特征计算得到调度因子;然后,由调度因子确定各节点应分得的数据量与任务槽数;最后,将数据和任务多分给快节点同时少分给慢节点。实验结果表明,与传统的调度算法相比,NCAS算法大幅度减少了备份任务的启动数量,明显减少了作业完成时间,提升了任务执行效率。 相似文献
11.
服务器执行任务产生的能耗是云计算系统动态能耗的重要组成部分。为降低云计算系统任务执行的总能耗,提出了一种基于能耗优化的最早完成时间任务调度方法,建立了服务器动态功率计算模型,基于动态功率的服务器执行能耗模型,以及云计算系统的能耗优化模型。调度策略根据任务的截止时间要求和在不同服务器上的执行能耗,选择不同的调度算法,以获得最小任务执行总能耗。实验结果证明,提出的任务调度方法,能够较好地满足任务截止时间的要求,降低云计算系统任务执行的总能耗。 相似文献
12.
MapReduce是云计算中重要的批数据处理框架,多任务共享MapReduce机群并满足任务实时性要求是调度算法急需解决的问题。提出两阶段实时调度算法,将调度划分为任务间调度和任务内调度。对于任务间调度,使用抽样法和经验值法确定子任务执行时间,利用该参数建立资源分配模型,动态确定任务优先级进行调度;对于子任务使用延迟调度策略进行调度,保证计算的本地性。实验结果显示,两阶段实时调度算法相比公平调度算法和FIFO算法,在保证吞吐量的同时能够满足任务实时性要求。 相似文献
13.
为了满足有截止时间限制的MapReduce作业的需求,提出一种基于截止时间限制的动态调度算法(DCDS)。该算法实时监控作业运行状况,并对作业运行时间进行动态估算,从而确定作业优先级;对于时间紧迫的作业,可通过抢占策略来保证在用户要求的截止时间内完成。实验结果表明,与Hadoop平台现有的调度算法相比,该算法不仅能满足作业截止时间的要求,也提高了系统资源的利用率和吞吐量。 相似文献
14.
在商业网格和云计算环境中,作业有到达时间、计算量、预算、截止期等参数,其中,预算是时间的函数。准确区分作业的重要性和紧迫性是作业调度系统的一个关键问题。综合利用这四个参数来定义作业的优先级,并提出基于价值密度和相对截止期的网格作业调度算法。分别对弱实时和强实时网格作业的调度进行仿真。仿真结果显示,所提出的调度算法的性能在两种情况下都优于所有对比算法的性能,且在强实时作业情况下优势更明显。 相似文献
15.
虚拟机上部署容器的双层虚拟化云架构在云数据中心中的使用越来越广泛.为了解决该架构下云数据中心的能耗问题,提出了一种工作流任务调度算法TUMS-RTC.针对有截止时间约束的并行工作流,算法将调度过程划分为时间利用率最大化调度和运行时间压缩两个阶段.时间利用率最大化调度通过充分使用给定的时间范围减少完成工作流所需的虚拟机和... 相似文献
16.
Ashis Kumar Mishra Subasish Mohapatra Pradip Kumar Sahu 《Concurrency and Computation》2024,36(10):e8005
Resource management and job scheduling are essential in today's cloud computing world. Due to task scheduling and users' diverse submission of large-scale requests, co-located VM instances negatively impacted the performance of leased VM instances. This workload further led to resource rivalry across co-located VMs. In order to address the aforementioned problems, numerous strategies have been presented, however, they fail to take the asynchronous nature of the cloud environment into account. To address this issue, a novel “CTA using DLFC-NN model” is proposed. This proposed approach combines the coalition theory and DLFC-NN techniques by including IRT-OPTICS for task size clustering, digital metrology based on ionized information (DMBII) for defect detection in virtue machines (VM), and the dynamic levy flight hamster optimization algorithm for processing time optimization of the clusters. However, the implementation of task scheduling in an online environment is limited by a number of presumptions or oversimplifications made by current scheduling systems. As a result, a unique coalition theory is applied to efficiently schedule activities. In addition, the DLFC-NN model is used to reduce resource consumption, span time, and be highly accurate and energy-efficient when working on both online and offline jobs. Nevertheless, while optimizing the clusters' overall execution time, earlier approaches only decreased the make-span time for task scheduling. However, the DLFC-NN model solves the computation problem by using a fully weighted bipartite graph and the pseudo method to determine the fitness of the least makespan time. The enhanced methodology used in this study reduces the scheduling cost and minimizes job completion times according to different task counts when compared to the existing techniques. 相似文献
17.
An Effective Cloud Workflow Scheduling Approach Combining PSO and Idle Time Slot-Aware Rules
下载免费PDF全文
![点击此处可从《IEEE/CAA Journal of Automatica Sinica》网站下载免费的PDF全文](/ch/ext_images/free.gif)
Workflow scheduling is a key issue and remains a challenging problem in cloud computing.Faced with the large number of virtual machine(VM)types offered by cloud providers,cloud users need to choose the most appropriate VM type for each task.Multiple task scheduling sequences exist in a workflow application.Different task scheduling sequences have a significant impact on the scheduling performance.It is not easy to determine the most appropriate set of VM types for tasks and the best task scheduling sequence.Besides,the idle time slots on VM instances should be used fully to increase resources'utilization and save the execution cost of a workflow.This paper considers these three aspects simultaneously and proposes a cloud workflow scheduling approach which combines particle swarm optimization(PSO)and idle time slot-aware rules,to minimize the execution cost of a workflow application under a deadline constraint.A new particle encoding is devised to represent the VM type required by each task and the scheduling sequence of tasks.An idle time slot-aware decoding procedure is proposed to decode a particle into a scheduling solution.To handle tasks'invalid priorities caused by the randomness of PSO,a repair method is used to repair those priorities to produce valid task scheduling sequences.The proposed approach is compared with state-of-the-art cloud workflow scheduling algorithms.Experiments show that the proposed approach outperforms the comparative algorithms in terms of both of the execution cost and the success rate in meeting the deadline. 相似文献
18.
Apache Hadoop becomes ubiquitous for cloud computing which provides resources as services for multi-tenant applications. YARN (a.k.a. MapReduce 2.0) is one of the key features in the second-generation Hadoop, which provides resource management and scheduling for large-scale MapReduce environments. Two enormous challenges in the YARN scheduler are the abilities to automatically tailor and control resource allocations to different jobs for achieving their Service Level Agreements (SLAs), and minimize energy consumption of the overall cloud computing system. In this work, we propose an SLA-aware energy-efficient scheduling scheme which allocates appropriate amount of resources to MapReduce applications with YARN architecture. In our task scheduling policy, We consider the data locality information to save the MapReduce network traffic. Furthermore, the slack time between the actual execution time of completed tasks and expected completion time of the application is utilized to improve the energy-efficiency of the system. An online userspace governor-based dynamic voltage and frequency scaling (DVFS) scheme is designed in the YARN per-application ApplicationMaster to dynamically change the CPU frequency for upcoming tasks given the slack time from previous completed tasks. Experimental evaluation shows that our proposed scheme outperforms the existing MapReduce scheduling policies in terms of both resource ultization and energy-efficiency. 相似文献