基于性能预测的Map Reduce负载均衡优化

Hadoop作为一个开源云计算框架,因为其优秀的分布式存储能力及其上运行的Map Reduce计算框架,越来越多的公司用Hadoop处理庞大的数据。负载均衡对分布式计算的性能有重要影响,Hadoop基于最简单的hash算法对任务进行分派,并没有对异构环境及多任务运行环境的因素做过多考虑。为了解决上述存在的不足,我们提出一种负载均衡框架,对计算节点的性能情况进行监控,预测节点的计算能力,依据预测结果来对Map Reduce的计算任务进行分配,从而最大化分布式集群的计算能力。     

融合区分服务和速率调整的请求调度策略

针对异构用户服务质量(QoS)需求不同及云数据中心能源浪费等问题,提出了一种融合区分服务和速率调整的请求调度策略。该策略将IaaS云数据中心分为提供速率调整机制的快速云和提供差错恢复机制的可靠云,然后研究新型混合云计算模式。构建了混合排队网络系统模型,从延迟敏感请求的平均响应时间、快速云中虚拟机的节能水平及可靠云中虚拟机的利用率等方面评估了请求调度策略的系统性能。进行了系统实验,验证了策略的有效性及模型的合理性。通过构造系统成本函数,改进Jaya智能寻优算法,给出了请求调度策略的优化方案。     

面向混合负载的集群资源弹性调度

针对目前集群资源调度方法难以适应互联网业务多样化、定制化特征的问题,提出了一种面向混合负载的集群资源弹性调度方法.该方法通过构建作业约束描述语言,允许作业基于自身负载特征提出多维度的资源申请和具有负载意识的资源调度算法,实现在同一集群内各类业务统一部署与管理,及时匹配资源需求的变化;通过建立作业的软约束与硬约束之间的转化机制,满足作业在不同执行阶段对资源的定制化需求.实验表明,该方法相比于Hadoop,可允许作业利用较少资源获得更优性能,在实际生产系统中,基于该方法可将集群资源利用率由62％提升到75％.     

基于超级基站的集中式接入网络架构的物理层虚拟化方法

针对集中式接入网络基带物理层计算资源异构性、基带算法模块间依赖性和物理层通信协议的实时性的存在而导致传统虚拟化技术不再适用的问题,在中科院计算所的超级基站的架构基础上,按照TD-LTE协议提出了一种基于单元级和系统级的二层资源调度策略,从而达到物理层计算资源虚拟化和实现虚拟基站的负载均衡和负载聚合的目的。仿真结果显示,这种实现物理层虚拟化的资源调度策略能够为集中式接入网络的物理层带来1.6的资源复用增益和3.725的功率复用增益。本研究能够为集中式接入网物理层计算资源池虚拟化的实现提供参考。     

A hybrid task scheduling scheme for multicore processors based on improving the EDF algorithm

为解决多核处理器系统中的实时任务调度问题,尤其是实时任务和非实时任务的混合调度问题,在对最早截止时间优先(EDF)算法进行改进的基础上,提出多核处理器混合任务调度算法——EDF-segment算法.EDF-segment算法可以整理调度混合任务时出现的碎片,并通过对碎片的迁移、合并提高处理器的利用率,从而提高系统处理混合任务的性能.通过EDF-segment算法不但可以解决混合任务的调度问题,还可以避免使用EDF算法时造成的多核处理器利用率下降,在保证实时任务处理延迟的前提下提升多核处理器的利用率.经过理论推导和实验分析证明,EDF-segment算法可以有效地应用于多核处理器系统中.     

网络化制造系统联邦集成的关键技术

联邦集成架构(FIA)是异构网络化制造系统集成的新方法.本文按照FIA接口规范,实现了FIA中的联邦执行支撑环境(FEI).对系统的关键技术--分布自治结构的联邦管理技术、支持跨系统任务协作的访问控制与授信技术、跨系统的分布式共享资源调度技术、服务搜索与匹配技术等进行了研究,并给出了相应的解决方案.该联邦集成系统已在北京网络化制造系统集成中得到了实际应用.应用结果表明,联邦模式的集成方案是一种适用于构建大规模服务共享的集成模式,能够适应网络化制造服务异构及运营目标多样化的应用环境.     

OFDMA系统中支持混合业务的跨层调度算法

针对正交频分多址接入(OFDMA)系统下行链路的混合业务调度问题,提出了一种基于队列等待时间的跨层调度算法.该算法联合利用了MAC层的队列等待时间与物理层的信道状态信息作为调度参数,通过队列等待时间反映用户的服务质量要求,并利用多用户分集增益提高系统性能;针对实时和非实时用户的不同服务质量要求,在队列等待时间的计算上采取了不同的策略;在子载波的分配过程中根据分配状态及时更新队列等待时间,使资源的利用更为有效.仿真结果表明,提出的算法可显著降低实时用户的平均时延和最大延时违反概率,同时保证了非实时用户的吞吐量需求,能够有效地支持下一代网络中混合业务的多种服务质量要求.     

面向强一致性的分布式对象存储的I/O并行性优化

在提供数据强一致性保障的分布式对象存储系统中,其I/O并行性受到系统I/O调度算法(主副本优先调度)的限制。本文提出了一种简单高效的I/O调度策略,其基于多副本的主从模型,可在保障强一致性的同时充分挖掘I/O并行空间。其包括如下3个主要步骤:第一,I/O请求被发送至主副本节点进行负载合并;第二,这些请求被送至数据相关性检测器根据相关性分配优先级;第三,根据I/O优先级及负载分布,将I/O请求尽可能均衡地转发至各副本节点上并行处理。本文实现了一个分布式对象存储系统原型用于验证该策略的有效性。实验分别对本文策略的各个环节进行了评估,实验结果表明,较主副本优先调度策略,本文策略使得GET请求吞吐量最大提升41.8%,GET请求平均延迟最大降低42.5%,GET请求99.9~(th)延迟最大降低15.8倍,这使得系统性能达到最终一致性下基准调度策略C3的水平。     

滤光片表面缺陷视觉检测系统多处理器调度优化

针对滤光片表面缺陷视觉检测系统中在线检测实时性需求对检测速度要求较高,研究一种有效利用可用硬件资源并行处理实时工作提高处理速度的调度优化策略。基于AOE图对滤光片表面缺陷视觉检测系统进行任务级分析,优化事件、活动拓扑关系与任务间冗余的数据相关性、资源相关性,建立并行任务模型;采用关联处理器调度算法(arbitrary processor affinities,APAs)进行并行多处理器调度,指定任务只能被某个处理器集合执行,将期限紧迫、缓存敏感的任务限制在单一处理器,提高资源利用率,改进检测系统实时性。试验结果表明:在尺寸为1.20mm×1.20mm、26×28个滤光片组成滤光片面板上,采用多处理器调度可使检测速度极大提升,采用APAs调度算法后,平均缺陷识别完成时间为常规检测系统时间的36.5%,可以满足在线实时要求,证明应用多处理器调度方法,可以极大提升检测仪器实时性能的有效性。     

考虑成本的多CODP混合延迟策略生产系统优化

延迟策略是企业以低成本和高响应客户个性化需求的运作模式来获取竞争优势的重要理论之一。单CODP延迟模式制约生产系统实现不同层次、程度的多样化,约束目标客户的选择范围,为此提出多CODP混合延迟策略。从横、纵向两个纬度构建了实施多CODP混合延迟策略的生产系统成本模型,寻求每个制造中心通用零部件数量最优比例和不同延迟策略的最优组合,并提出与之匹配的求解算法。实例分析表明:在通用零部件数量最优比例和延迟策略的最优组合的支持下,多CODP混合延迟策略的实施不仅释放客户定制空间、扩大目标客户选择范围,而且实现了生产系统总成本的优化。     

An Optimized Resource Scheduling Strategy for Hadoop Speculative Execution Based on Non-cooperative Game Schemes

Hadoop is a well-known parallel computing system for distributed computing and large-scale data processes. “Straggling” tasks, however, have a serious impact on task allocation and scheduling in a Hadoop system. Speculative Execution (SE) is an efficient method of processing “Straggling” Tasks by monitoring real-time running status of tasks and then selectively backing up “Stragglers” in another node to increase the chance to complete the entire mission early. Present speculative execution strategies meet challenges on misjudgement of “Straggling” tasks and improper selection of backup nodes, which leads to inefficient implementation of speculative executive processes. This paper has proposed an Optimized Resource Scheduling strategy for Speculative Execution (ORSE) by introducing non-cooperative game schemes. The ORSE transforms the resource scheduling of backup tasks into a multi-party non-cooperative game problem, where the tasks are regarded as game participants, whilst total task execution time of the entire cluster as the utility function. In that case, the most benefit strategy can be implemented in each computing node when the game reaches a Nash equilibrium point, i.e., the final resource scheduling scheme to be obtained. The strategy has been implemented in Hadoop-2.x. Experimental results depict that the ORSE can maintain the efficiency of speculative executive processes and improve fault-tolerant and computation performance under the circumstances of Normal Load, Busy Load and Busy Load with Skewed Data.     

Task Scheduling Optimization in Cloud Computing by Rao Algorithm

Cloud computing is currently dominated within the space of high-performance distributed computing and it provides resource polling and on-demand services through the web. So, task scheduling problem becomes a very important analysis space within the field of a cloud computing environment as a result of user's services demand modification dynamically. The main purpose of task scheduling is to assign tasks to available processors to produce minimum schedule length without violating precedence restrictions. In heterogeneous multiprocessor systems, task assignments and schedules have a significant impact on system operation. Within the heuristic-based task scheduling algorithm, the different processes will lead to a different task execution time (makespan) on a heterogeneous computing system. Thus, a good scheduling algorithm should be able to set precedence efficiently for every subtask depending on the resources required to reduce (makespan). In this paper, we propose a new efficient task scheduling algorithm in cloud computing systems based on RAO algorithm to solve an important task and schedule a heterogeneous multiple processing problem. The basic idea of this process is to exploit the advantages of heuristic-based algorithms to reduce space search and time to get the best solution. We evaluate our algorithm's performance by applying it to three examples with a different number of tasks and processors. The experimental results show that the proposed approach significantly succeeded in finding the optimal solutions than others in terms of the time of task implementation.     

A New Task Scheduling Scheme Based on Genetic Algorithm for Edge Computing

With the continuous evolution of smart grid and global energy interconnection technology, amount of intelligent terminals have been connected to power grid, which can be used for providing resource services as edge nodes. Traditional cloud computing can be used to provide storage services and task computing services in the power grid, but it faces challenges such as resource bottlenecks, time delays, and limited network bandwidth resources. Edge computing is an effective supplement for cloud computing, because it can provide users with local computing services with lower latency. However, because the resources in a single edge node are limited, resource-intensive tasks need to be divided into many subtasks and then assigned to different edge nodes by resource cooperation. Making task scheduling more efficient is an important issue. In this paper, a two-layer resource management scheme is proposed based on the concept of edge computing. In addition, a new task scheduling algorithm named GA-EC(Genetic Algorithm for Edge Computing) is put forth, based on a genetic algorithm, that can dynamically schedule tasks according to different scheduling goals. The simulation shows that the proposed algorithm has a beneficial effect on energy consumption and load balancing, and reduces time delay.     

面向多用户环境的MapReduce集群调度算法研究

针对MapReduce集群现有调度策略在多用户环境下无法根据用户的实际资源需求实现动态资源分配的问题,提出了一种基于历史执行信息(HEI)的MapReduce集群调度算法——HEI Scheduler。该算法通过建立集群作业执行信息的收集和分析机制,得到各用户组资源需求随时间变化的规律,并以作业实际占用slot的时间作为作业占用资源量的衡量标准,进而动态地确定资源池的最小共享资源以及集群剩余资源分配的权值。实验结果表明,执行信息分析机制能够更准确地表征作业对资源的需求,采用集群调度算法HEI Scheduler能够有效地缩短作业的整体执行时间。     

研发型企业多项目人力资源调度研究——基于蚁群优化的超启发式算法

现阶段,研发型企业的项目处于多项目环境下,为了解决多项目并行时人力资源争夺问题,本文针对该类企业多项目管理中人力资源调度进行优化研究,以考虑项目延期惩罚成本的最小总成本为目标函数,将现实问题抽象建模。基于国内外的研究提出了一种超启发式算法进行求解,该算法将人力资源调度问题分为项目活动分配和人员选择项目活动两个部分,采用蚁群优化作为高层启发式策略搜索低层启发式规则,再进一步根据规则解构造出可行解。最后本研究设计多组仿真实验与启发式规则进行对比,结果表明该算法有较好的搜索性能,为人力资源的调度问题提供了新的解决方案。     

Queueing network modelling and lead time compression of pharmaceutical drug development

Designing an optimized pharmaceutical drug development process is an important problem in itself and is of significant practical and research interest. Drug development lead time is a critical performance metric for a pharmaceutical company. In this paper, we develop a multiclass queueing network model to capture the project dynamics in drug development organizations that involve multiple, concurrent projects with contention for human/technical resources. We explore how drug development lead times can be reduced using efficient scheduling and critical mass-based resource management. The model captures important facets of any typical drug development organization, such as concurrent execution of multiple projects, contention for resources, feedback and reworking of project tasks, variability of new project initiations and task execution times, and certain scheduling issues. First, we show, using a class of fluctuation smoothing scheduling policies, that development lead times can be compressed impressively, without having to commit additional resources. Next, we show that critical mass-based project teams can compress lead times further. The model presented, though stylized, is sufficiently generic and conceptual, and will be of much value in new drug development project planning and management.     

第三方物流联盟中物流任务的优化调度

为了提高物流服务水平、降低物流运作成本,针对由多个第三方物流服务商组建而成的第三方物流联盟中物流任务与物流服务资源的优化调度问题展开研究,综合考虑各第三方物流服务商资源节点提供物流活动服务成本和物流服务总时间,以时间最短和成本最低为优化目标,提出了基于时间和成本的多目标优化调度模型,针对目前物流任务调度优化模型中只考虑各物流服务资源节点本身的服务成本和时间,而未考虑执行各个物流活动之间的物流资源节点之间的衔接时间与衔接成本的问题,提出一种计算不同物流服务资源节点之间的物流服务衔接时间和衔接成本的方法,在模型中,考虑了物流资源服务时间窗限制问题.最后提出了一个改进的遗传算法进行模型求解,并通过算例验证了研究的有效性.     

Fuzzy Control Based Resource Scheduling in IoT Edge Computing

Edge Computing is a new technology in Internet of Things (IoT) paradigm that allows sensitive data to be sent to disperse devices quickly and without delay. Edge is identical to Fog, except its positioning in the end devices is much nearer to end-users, making it process and respond to clients in less time. Further, it aids sensor networks, real-time streaming apps, and the IoT, all of which require high-speed and dependable internet access. For such an IoT system, Resource Scheduling Process (RSP) seems to be one of the most important tasks. This paper presents a RSP for Edge Computing (EC). The resource characteristics are first standardized and normalized. Next, for task scheduling, a Fuzzy Control based Edge Resource Scheduling (FCERS) is suggested. The results demonstrate that this technique enhances resource scheduling efficiency in EC and Quality of Service (QoS). The experimental study revealed that the suggested FCERS method in this work converges quicker than the other methods. Our method reduces the total computing cost, execution time, and energy consumption on average compared to the baseline. The ES allocates higher processing resources to each user in case of limited availability of MDs; this results in improved task execution time and a reduced total task computation cost. Additionally, the proposed FCERS m 1m may more efficiently fetch user requests to suitable resource categories, increasing user requirements.     

A Heterogeneous Virtual Machines Resource Allocation Scheme in Slices Architecture of 5G Edge Datacenter

In the paper, we investigate the heterogeneous resource allocation scheme for virtual machines with slicing technology in the 5G/B5G edge computing environment. In general, the different slices for different task scenarios exist in the same edge layer synchronously. A lot of researches reveal that the virtual machines of different slices indicate strong heterogeneity with different reserved resource granularity. In the condition, the allocation process is a NP hard problem and difficult for the actual demand of the tasks in the strongly heterogeneous environment. Based on the slicing and container concept, we propose the resource allocation scheme named Two-Dimension allocation and correlation placement Scheme (TDACP). The scheme divides the resource allocation and management work into three stages in this paper: In the first stage, it designs reasonably strategy to allocate resources to different task slices according to demand. In the second stage, it establishes an equivalent relationship between the virtual machine reserved resource capacity and the Service-Level Agreement (SLA) of the virtual machine in different slices. In the third stage, it designs a placement optimization strategy to schedule the equivalent virtual machines in the physical servers. Thus, it is able to establish a virtual machine placement strategy with high resource utilization efficiency and low time cost. The simulation results indicate that the proposed scheme is able to suppress the problem of uneven resource allocation which is caused by the pure preemptive scheduling strategy. It adjusts the number of equivalent virtual machines based on the SLA range of system parameter, and reduces the SLA probability of physical servers effectively based on resource utilization time sampling series linear. The scheme is able to guarantee resource allocation and management work orderly and efficiently in the edge datacenter slices.     

Efficient Resource Allocation in Fog Computing Using QTCS Model

Infrastructure of fog is a complex system due to the large number of heterogeneous resources that need to be shared. The embedded devices deployed with the Internet of Things (IoT) technology have increased since the past few years, and these devices generate huge amount of data. The devices in IoT can be remotely connected and might be placed in different locations which add to the network delay. Real time applications require high bandwidth with reduced latency to ensure Quality of Service (QoS). To achieve this, fog computing plays a vital role in processing the request locally with the nearest available resources by reduced latency. One of the major issues to focus on in a fog service is managing and allocating resources. Queuing theory is one of the most popular mechanisms for task allocation. In this work, an efficient model is designed to improve QoS with the efficacy of resource allocation based on a Queuing Theory based Cuckoo Search (QTCS) model which will optimize the overall resource management process.