Hadoop集群作业调度算法优化技术研究

针对当前云计算技术的广泛使用,提出对Hadoop集群作业调度算法进行研究的构想。在对Hadoop新版本中提出的Hadoop Map Reduce V2(Yarn)框架进行深入研究的基础上,设计一个详细完整的对比实验,针对各种作业调度算法的优势与不足进行全面的测试。通过使用不同的作业调度算法运行相同的作业,进行细致横向的对比,并得到各种作业调度算法之间以及计算框架之间在计算能力、运行时间、资源占用等方面的优劣。实验结果表明,公平调度算法和计算能力调度算法相较于传统的FIFO算法具有更灵活、更高效的特点。     

同步卫星转发器二维时频资源蚁群调度算法

针对地球同步卫星转发器多任务时频资源调度问题,考虑任务执行时间和占用带宽需求,建立了以卫星系统总收益为目标的多约束规划模型,提出了基于任务频率时间窗口更新的蚁群调度算法.该算法综合考虑了任务优先级和时间灵活度,以增强蚁群在状态转移规则下的搜索能力;同时设计了虚拟任务、伪随机状态转移规则和信息素参数,保证算法向最优解逐步收敛.实例仿真表明,该算法相对于传统蚁群算法、遗传算法和启发式算法,在算法结果、寻优能力和稳定性方面具有显著优势.     

粒子群蚁群融合算法的火灾救援路径研究

为获取最优的救援路径,以提高救援的有效性和实时性,文中提出了一种粒子群蚁群融合算法。该算法在分析影响路径选择因素的基础上,运用模糊数学中的层次分析法评定了道路的权重,建立了消防灭火救援模型;使用粒子群算法快速获取次优解,将此次优解作为蚁群算法的初始信息素增量,并将求解出各段路径权重矩阵引入到优化后的蚁群算法状态转移概率的求解模型中来,再利用这种改进后的状态转移规则,且考虑行车速度时变性的基础上求解出模型的最优解。实验结果表明,该方法可以完成最佳救援路径的规划。     

基于遗传和粒子群算法的天基资源调度策略

以提高天基资源调度效率为目标,提出一种基于遗传和粒子群算法的天基资源调度策略。将资源调度问题抽象为任务排序模型,设计适用于任务排序的十进制编码规则;通过重新定义遗传算法中选择、交叉和变异算子的进化行为以及粒子群算法的速度方向,结合遗传算法全局最优搜索、粒子群算法局部快速收敛等优点设计适用于天基资源调度的遗传-粒子群算法。通过实验验证遗传-粒子群算法能够在较短的时间内计算出全局最优解,是一种高效的天基资源调度解决方法。     

云计算资源调度算法的研究与实现

在研究蚁群算法、任务分配和资源调度的基础上,提出了一种改进的蚁群资源调度算法.首先通过引入节点可信度机制在一定程度上增强了云计算资源的搜索能力和节点完成任务的成功率.然后在改进的算法中使用了信息素的局部更新机制和全局更新机制,可以有效地平衡负载.最后通过选取合适的参数利用CloudSim仿真工具对改进的资源调度算法进行实验测试,实验结果表明此算法缩短了任务的执行时间,改善了云计算资源调度的性能.     

基于蚁群算法的云计算资源调度分析

当前,云计算资源调度中常用的算法包括遗传算法、蚁群算法、粒子群算法以及综合优化算法等。不同算法所关注的角度存在一定的差异,遗传算法和粒子群算法主要从资源调度的效率方面进行优化,而蚁群算法则是从云计算资源调度的计算成本方面进行考虑。论文讨论了基于优化蚁群算法的云计算资源调度,在兼顾任务效率的基础上,能够进一步降低计算成本。     

面向密集多波束组网的卫星通信系统资源调度算法

研究了高动态、资源受限条件下的卫星通信系统资源调度问题。以时间窗口、卫星功耗、信道数量、用户优先级以及任务突发性为约束,建立了卫星资源调度模型。考虑到传统的蚁群优化算法存在初期搜索速度过慢、局部搜索能力较弱以及易陷入局部最优等缺点,提出了以初始解集构造、额外信息素沉积为核心的改进蚁群优化算法,来求解资源调度问题。仿真实验评估了所提资源调度算法在完成任务的数量、优先级和调度完成时间方面的性能。实验结果表明,所提算法具有较快的收敛速度,且与同类型优化算法相比具有更高的调度效率,适用于调度面向密集多波束组网需求的卫星通信系统资源。     

一种基于动态资源采集的Hadoop作业调度算法

为了克服Hadoop中基于槽的资源模型对集群资源的浪费,提高Hadoop集群的资源利用率,提出了一种基于动态资源采集的Hadoop作业调度算法.通过在作业运行过程中动态采集部分任务的CPU,内存和IO的资源利用率来评估同一作业中其他任务的资源需求,然后根据任务的实际资源需求以及TaskTracker节点的负载情况进行任务调度,以充分利用各TaskTracker节点的计算资源.通过对比实验表明该调度算法能明显提高集群的资源利用率,缩短作业的完成时间.     

改进DPSO算法求解仿真任务调度问题

针对离散粒子群算法在求解雷达分布式仿真系统中的仿真任务调度时,由于其易陷入局部最优的缺陷导致算法受初始种群的影响较大且结果稳定低的问题,提出基于信息素变异策略的改进离散粒子群算法。文中分析了离散粒子群算法容易陷入局部最优的原因,引入基于信息素的变异策略,充分利用种群中所有粒子的寻优经验信息来累计信息素,以信息素的分布和效率矩阵为依据对基本离散粒子群算法每次迭代后得到的粒子进行变异操作。仿真结果表明,改进算法有效地避免了算法陷于局部最优的问题,且结果的稳定性比基本离散粒子群算法更好,调度跨度和负载平衡度相比离散粒子群算法,蚁群算法,Max-Min算法和Min-Min算法都有明显的改善。      

基于拥挤度因子的动态信息素更新策略蚁群算法

针对蚁群算法易陷入局部最优、收敛速度慢的问题,文中提出了一种基于拥挤度因子的动态信息素更新策略的蚁群算法(CFACS)。引入鱼群算法中拥挤度的思想,扩大种群中蚂蚁分布范围,使其探索更大的解空间,提高算法全局搜索能力;采用动态信息素更新策略,在每一次迭代中,自适应调整当前最优路径所释放的信息素浓度,保证蚁群前期的多样性,同时保证算法在后期的收敛性。求解TSP问题的仿真实验表明,改进算法求得解的质量和求解的收敛速度都明显优于传统蚁群算法,较好地平衡了种群多样性与收敛速度之间的矛盾。     

K-PSO: An improved PSO-based container scheduling algorithm for big data applications

In recent years, Docker container technology is being applied in the field of cloud computing at an explosive speed. The scheduling of Docker container resources has gradually become a research hotspot. Existing big data computing and storage platforms apply with traditional virtual machine technology, which often results in low resource utilization, a long time for flexible scaling and expanding clusters. In this paper, we propose an improved container scheduling algorithm for big data applications named Kubernetes-based particle swarm optimization(K-PSO). Experimental results show that the proposed K-PSO algorithm converges faster than the basic PSO algorithm, and the running time of the algorithm is cut in about half. The K-PSO container scheduling algorithm and algorithm experiment for big data applications are implemented in the Kubernetes container cloud system. Our experimental results show that the node resource utilization rate of the improved scheduling strategy based on K-PSO algorithm is about 20% higher than that of the Kube-scheduler default strategy, balanced QoS priority strategy, ESS strategy, and PSO strategy, while the average I/O performance and average computing performance of Hadoop cluster are not degraded.     

基于蚂蚁算法的网格作业调度研究

网格环境下的作业调度是一个NP难问题,蚂蚁算法内在的并行性和可扩充性使其非常适合网格作业调度。将蚂蚁算法应用于网格环境作业调度,提出一种通过作业代理的移动进行网格作业调度的方案,该蚂蚁算法不仅在分配网格计算资源时进行信息素的局部更新,还在网格计算资源完成作业后进行信息素的整体更新。通过模拟实验测试和选取蚂蚁算法的各种影响参数,取得了比较理想的实验结果。实验证明该算法能够有效地实现作业的合理调度和网格系统的负载平衡。     

基于蚁群的云计算资源调度研究

针对云计算环境中资源调度的问题,提出了一种基于改进蚁群的云计算资源调度算法。在算法中添加了查找表,存储其他蚂蚁推荐的节点。当任务分类比较明确的时候,查找表的优点更加地突出。在信息素的计算中加入了成功率因子,成功率越高的节点被选中的概率就越大。本文使用Cloud Sim对算法进行了仿真,仿真结果表明提出的算法缩短了搜寻资源节点的时间,从而使任务可以更快地获得资源并执行,保证了任务能够按时完成。     

基于改进蚁群算法的自适应云资源调度模型研究

对于传统蚁群算法用于云计算资源分配和调度问题过程中存在的不足,提出了一种可以提高负载均衡度、缩短任务执行时间、降低任务执行成本的改进自适应蚁群算法,改进算法以能够基于用户提交的任务求解出执行时间较短、费用较低,负载率均衡的分配方案为目标,通过CloudSim平台对传统蚁群算法、最新的AC-SFL算法、改进自适应蚁群算法进行仿真实验对比。实验数据表明,改进后的自适应蚁群算法能够快速找出最优的云计算资源调度问题的解决方案,缩短了任务完成时间,降低了执行费用,保持了整个云系统中心的负载均衡。     

Job schedulers for Big data processing in Hadoop environment: testing real-life schedulers using benchmark programs

At present, big data is very popular, because it has proved to be much successful in many fields such as social media, E-commerce transactions, etc. Big data describes the tools and technologies needed to capture, manage, store, distribute, and analyze petabyte or larger-sized datasets having different structures with high speed. Big data can be structured, unstructured, or semi structured. Hadoop is an open source framework that is used to process large amounts of data in an inexpensive and efficient way, and job scheduling is a key factor for achieving high performance in big data processing. This paper gives an overview of big data and highlights the problems and challenges in big data. It then highlights Hadoop Distributed File System (HDFS), Hadoop MapReduce, and various parameters that affect the performance of job scheduling algorithms in big data such as Job Tracker, Task Tracker, Name Node, Data Node, etc. The primary purpose of this paper is to present a comparative study of job scheduling algorithms along with their experimental results in Hadoop environment. In addition, this paper describes the advantages, disadvantages, features, and drawbacks of various Hadoop job schedulers such as FIFO, Fair, capacity, Deadline Constraints, Delay, LATE, Resource Aware, etc, and provides a comparative study among these schedulers.     

Resource scheduling in mobile edge computing using improved ant colony algorithm for space information network

With the development of space information network (SIN), new network applications are emerging. Satellites are not only used for storage and transmission but also gradually used for calculation and analysis, so the demand for resources is increasing. But satellite resources are still limited. Mobile edge computing (MEC) is considered an effective technique to reduce the pressure on satellite resources. To solve the problem of task execution delay caused by limited satellite resources, we designed Space Mobile Edge Computing Network (SMECN) architecture. According to this architecture, we propose a resource scheduling method. First, we decompose the user tasks in SMECN, so that the tasks can be assigned to different servers. An improved ant colony resource scheduling algorithm for SMECN is proposed. The heuristic factors and pheromones of the ant colony algorithm are improved through time and resource constraints, and the roulette algorithm is applied to route selection to avoid falling into the local optimum. We propose a dynamic scheduling algorithm to improve the contract network protocol to cope with the dynamic changes of the SIN and dynamically adjust the task execution to improve the service capability of the SIN. The simulation results show that when the number of tasks reaches 200, the algorithm proposed in this paper takes 17.52% less execution time than the Min-Min algorithm, uses 9.58% less resources than the PSO algorithm, and achieves a resource allocation rate of 91.65%. Finally, introducing dynamic scheduling algorithms can effectively reduce task execution time and improve task availability.     

Performance Improvement of MapReduce for Heterogeneous Clusters Based on Efficient Locality and Replica Aware Scheduling (ELRAS) Strategy

MapReduce is a parallel programming model for processing the data-intensive applications in a cloud environment. The scheduler greatly influences the performance of MapReduce model while utilized in heterogeneous cluster environment. The dynamic nature of cluster environment and computing workloads affect the execution time and computational resource usage in the scheduling process. Further, data locality is essential for reducing total job execution time, cross-rack communication, and to improve the throughput. In the present work, a scheduling strategy named efficient locality and replica aware scheduling (ELRAS) integrated with an autonomous replication scheme (ARS) is proposed to enhance the data locality and performs consistently in the heterogeneous environment. ARS autonomously decides the data object to be replicated by considering its popularity and removes the replica as it is idle. The proposed approach is validated in a heterogeneous cluster environment with various realistic applications that are IO bound, CPU bound and mixed workloads. ELRAS improves the throughput by a factor about 2 as compared with the existing FIFO and it also yields near optimal data locality, reduce the execution time, and effective utilization of resources. The simplicity of ELRAS algorithm proves its feasibility to adopt for a wide range of applications.

     

基于树型网格的混合遗传算法

资源调度算法是网格计算研究的一个重要研究方向。文章讨论了树型网格的逻辑调度模式．提出了基于树型网格的混合遗传算法HGATG。算法利用启发式操作算子加快最优值的收敛速度．并采用随机法和唯一法来避免算法的早熟。实验结果表明HGATG算法在求解速度、成功率和求解问题的规模等方面有较好的效果。