Logistics-involved QoS-aware service composition in cloud manufacturing with deep reinforcement learning

Cloud manufacturing is a new manufacturing model that aims to provide on-demand manufacturing services to consumers over the Internet. Service composition is an essential issue as well as an important technique in cloud manufacturing (CMfg) that supports construction of larger-granularity, value-added services by combining a number of smaller-granularity services to satisfy consumers’ complex requirements. Meta-heuristics algorithms such as genetic algorithm, particle swarm optimization, and ant colony algorithm are frequently employed for addressing service composition issues in cloud manufacturing. These algorithms, however, require complex design flows and painstaking parameter tuning, and lack adaptability to dynamic environment. Deep reinforcement learning (DRL) provides an alternative approach for solving cloud manufacturing service composition (CMfg-SC) issues. DRL as model-free artificial intelligent methods enables a system to learn optimal service composition solutions through training, which can therefore circumvent the aforementioned problems with meta-heuristics algorithms. This paper is dedicated to exploring possible applications of DRL in CMfg-SC. A logistics-involved QoS-aware DRL-based CMfg-SC is proposed. A dueling Deep Q-Network (DQN) with prioritized replay named PD-DQN is designed as the DRL algorithm. Effectiveness, robustness, adaptability, and scalability of PD-DQN are investigated, and compared with that of the basic DQN and Q-learning. Experimental results indicate that PD-DQN is able to effectively address the CMfg-SC problem.     

Real-time scheduling for distributed permutation flowshops with dynamic job arrivals using deep reinforcement learning

Distributed manufacturing plays an important role for large-scale companies to reduce production and transportation costs for globalized orders. However, how to real-timely and properly assign dynamic orders to distributed workshops is a challenging problem. To provide real-time and intelligent decision-making of scheduling for distributed flowshops, we studied the distributed permutation flowshop scheduling problem (DPFSP) with dynamic job arrivals using deep reinforcement learning (DRL). The objective is to minimize the total tardiness cost of all jobs. We provided the training and execution procedures of intelligent scheduling based on DRL for the dynamic DPFSP. In addition, we established a DRL-based scheduling model for distributed flowshops by designing suitable reward function, scheduling actions, and state features. A novel reward function is designed to directly relate to the objective. Various problem-specific dispatching rules are introduced to provide efficient actions for different production states. Furthermore, four efficient DRL algorithms, including deep Q-network (DQN), double DQN (DbDQN), dueling DQN (DlDQN), and advantage actor-critic (A2C), are adapted to train the scheduling agent. The training curves show that the agent learned to generate better solutions effectively and validate that the system design is reasonable. After training, all DRL algorithms outperform traditional meta-heuristics and well-known priority dispatching rules (PDRs) by a large margin in terms of solution quality and computation efficiency. This work shows the effectiveness of DRL for the real-time scheduling of dynamic DPFSP.     

Logistics service scheduling with manufacturing provider selection in cloud manufacturing

In service-oriented manufacturing models, manufacturing resources in different enterprises are integrated and shared through network, cloud platforms, and logistics. On cloud manufacturing platforms, service providers offer on-demand manufacturing services to service demanders according to supply-demand matching results. As a special type of manufacturing services, logistics services provide transportation capabilities for production services and demanders. It is a critical issue to schedule logistics services efficiently, especially when manufacturer selections have been planned. This research focuses on the logistics scheduling problem in cloud manufacturing with pre-selected manufacturers. We analyze this optimization problem from aspects of tasks, production services, logistics services, and optimization objectives. Then a logistics scheduling method is proposed to reduce the average delivery time from manufacturers to customers. In the proposed method, the total time from start points of logistics to demanders is considered to reduce the average delivery time of all tasks. Based on four different scenarios, we build their scheduling models and run simulations to verify the effectiveness of the proposed method. Results show that the average task delivery time of the proposed method is shorter than three typical strategies.     

Deep reinforcement learning based control for Autonomous Vehicles in CARLA

Nowadays, Artificial Intelligence (AI) is growing by leaps and bounds in almost all fields of technology, and Autonomous Vehicles (AV) research is one more of them. This paper proposes the using of algorithms based on Deep Learning (DL) in the control layer of an autonomous vehicle. More specifically, Deep Reinforcement Learning (DRL) algorithms such as Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) are implemented in order to compare results between them. The aim of this work is to obtain a trained model, applying a DRL algorithm, able of sending control commands to the vehicle to navigate properly and efficiently following a determined route. In addition, for each of the algorithms, several agents are presented as a solution, so that each of these agents uses different data sources to achieve the vehicle control commands. For this purpose, an open-source simulator such as CARLA is used, providing to the system with the ability to perform a multitude of tests without any risk into an hyper-realistic urban simulation environment, something that is unthinkable in the real world. The results obtained show that both DQN and DDPG reach the goal, but DDPG obtains a better performance. DDPG perfoms trajectories very similar to classic controller as LQR. In both cases RMSE is lower than 0.1m following trajectories with a range 180-700m. To conclude, some conclusions and future works are commented.

     

Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network

To benefit from the accurate simulation and high-throughput data contributed by advanced digital twin technologies in modern smart plants, the deep reinforcement learning (DRL) method is an appropriate choice to generate a self-optimizing scheduling policy. This study employs the deep Q-network (DQN), which is a successful DRL method, to solve the dynamic scheduling problem of flexible manufacturing systems (FMSs) involving shared resources, route flexibility, and stochastic arrivals of raw products. To model the system in consideration of both manufacturing efficiency and deadlock avoidance, we use a class of Petri nets combining timed-place Petri nets and a system of simple sequential processes with resources (S³PR), which is named as the timed S³PR. The dynamic scheduling problem of the timed S³PR is defined as a Markov decision process (MDP) that can be solved by the DQN. For constructing deep neural networks to approximate the DQN action-value function that maps the timed S³PR states to scheduling rewards, we innovatively employ a graph convolutional network (GCN) as the timed S³PR state approximator by proposing a novel graph convolution layer called a Petri-net convolution (PNC) layer. The PNC layer uses the input and output matrices of the timed S³PR to compute the propagation of features from places to transitions and from transitions to places, thereby reducing the number of parameters to be trained and ensuring robust convergence of the learning process. Experimental results verify that the proposed DQN with a PNC network can provide better solutions for dynamic scheduling problems in terms of manufacturing performance, computational efficiency, and adaptability compared with heuristic methods and a DQN with basic multilayer perceptrons.     

Multi-phase integrated scheduling of hybrid tasks in cloud manufacturing environment

Cloud manufacturing paradigm aims at gathering distributed manufacturing resources and enterprises to serve for more customized production. Production order which involving several tasks can be taken by distributed suppliers collaboratively at lower cost. The cloud manufacturing platform is responsible for not only arranging reasonable priorities, suitable suppliers, and production processes to multiple orders, but also scheduling hybrid tasks from different orders to manufacturing resources. To maximize the production efficiency and balance the trade-off among different production orders, this paper studies multi-phase integrated scheduling of hybrid tasks in cloud manufacturing environment, which containing order priority assignment, supplier and production process selection, and production line scheduling. Five key objectives are taken into account to analyze the interconnections among different resources and production processes. Six representative multi-objective evolutionary algorithms are adopted to solve the integrated scheduling problem. Experimental results on six production cases show that integrated scheduling is more effective than the traditional step-by-step decision, leading to less production cost and time. In addition, a comparison among the six algorithms is carried out to determine the one best suited for the integrated scheduling problem in different circumstances.     

Workload-based multi-task scheduling in cloud manufacturing

Cloud manufacturing is an emerging service-oriented business model that integrates distributed manufacturing resources, transforms them into manufacturing services, and manages the services centrally. Cloud manufacturing allows multiple users to request services at the same time by submitting their requirement tasks to a cloud manufacturing platform. The centralized management and operation of manufacturing services enable cloud manufacturing to deal with multiple manufacturing tasks in parallel. An important issue with cloud manufacturing is therefore how to optimally schedule multiple manufacturing tasks to achieve better performance of a cloud manufacturing system. Task workload provides an important basis for task scheduling in cloud manufacturing. Based on this idea, we present a cloud manufacturing multi-task scheduling model that incorporates task workload modelling and a number of other essential ingredients regarding services such as service efficiency coefficient and service quantity. Then we investigate the effects of different workload-based task scheduling methods on system performance such as total completion time and service utilization. Scenarios with or without time constraints are separately investigated in detail. Results from simulation experiments indicate that scheduling larger workload tasks with a higher priority can shorten the makespan and increase service utilization without decreasing task fulfilment quality when there is no time constraint. When time constraint is involved, the above strategy enables more tasks to be successfully fulfilled within the time constraint, and task fulfilment quality also does not deteriorate.     

云计算环境下实训教学平台动态迁移策略

云服务环境下最大特点是按需交付,通过虚拟化技术将相关资源构建统一调度池,并且按照用户需求为用户提供服务,因此,云服务具有并行计算、开放性以及按需交付特性.对于实训教学平台来说,在云计算环境下需要面对各种用户需求,如请求任务各种各样,实验任务类型不尽相同,设备资源存在较大差异,通过虚拟化技术来实现规范化管理何资源共享,对云资源进行调度来才能有效满足用户需求,为此,在本文中提出了云计算环境下实训教学平台动态迁移策略.策略设计了三层协同资源调度机制来实现对资源和任务管理,重点研究了任务分割、资源划分、资源调度策略等,在此基础上对系统进行仿真实验,验证云计算环境下实训教学平台动态迁移策略可行与有效性.     

Fast GA-based project scheduling for computing resources allocation in a cloud manufacturing system

Cloud manufacturing is becoming an increasingly popular enterprise model in which computing resources are made available on-demand to the user as needed. Cloud manufacturing aims at providing low-cost, resource-sharing and effective coordination. In this study, we present a genetic algorithm (GA) based resource constraint project scheduling, incorporating a number of new ideas (enhancements and local search) for solving computing resources allocation problems in a cloud manufacturing system. A newly generated offspring may not be feasible due to task precedence and resource availability constraints. Conflict resolutions and enhancements are performed on newly generated offsprings after crossover or mutation. The local search can exploit the neighborhood of solutions to find better schedules. Due to its complex characteristics, computing resources allocation in a cloud manufacturing system is NP-hard. Computational results show that the proposed GA can rapidly provide a good quality schedule that can optimally allocate computing resources and satisfy users’ demands.     

面向云制造平台的模型修复算法及任务调度算法的研究

随着三维打印技术的发展,面向3D打印服务的云制造平台也得到快速发展,改变着传统的制造模式。然而目前的云制造平台也面临着诸多问题,针对目前存在的模型修复难度高及任务不能合理调度的问题,本文提出了保特征的模型修复算法和基于遗传算法的任务调度算法,并通过实验和仿真验证了算法的有效性。为了更好的体现算法效果,本文搭建了一个基于分布式制造的云制造平台,该平台配合模型自动修复算法以及基于遗传算法的任务调度算法,为用户提供低门槛、高效、优质的3D打印服务。     

Cloud workflow scheduling with hybrid resource provisioning

Resource provisioning strategies are crucial for workflow scheduling problems which are widespread in cloud computing. The main challenge lies in determining the amounts of reserved and on-demand resources to meet users’ requirements. In this paper, we consider the cloud workflow scheduling problem with hybrid resource provisioning to minimize the total renting cost, which is NP-hard and has not been studied yet. An iterative population-based meta-heuristic is developed. According to the shift vectors obtained during the search procedure, timetables are computed quickly. The appropriate amounts of reserved and on-demand resources are determined by an incremental optimization method. The utilization of each resource is balanced in a swaying way, in terms of which the probabilistic matrix is updated for the next iteration. The proposed algorithm is compared with modified existing algorithms for similar problems. Experimental results demonstrate effectiveness and efficiency of the proposed algorithm.     

云制造环境下的动态调度

在云制造环境下, 制造资源和制造能力以服务的形式封装起来, 不同的任务通过云端汇集到云平台并通过合适的调度给每个任务分配相应的服务. 由于任务在执行的过程中的不确定性, 会在某个时刻遇到突发状况从而导致对余下任务的重调度问题. 因此, 针对该问题, 考虑到云制造环境下任务的复杂性和多样性会导致在合理的时间段内很难找到最优解, 以所有任务的最大完成时间为优化目标, 提出了一种以改进的遗传算法与邻域搜索技术相结合的元启发式算法, 旨在解决云制造环境下由于任务和资源服务等的不确定性导致的重调度问题. 实验结果表明,本文所提出的算法能够很好地解决动态调度过程中的重调度问题, 并可以快速地获取最优解.     

Application Layer Scheduling in Cloud: Fundamentals,Review and Research Directions

The cloud computing paradigm facilitates a finite pool of on-demand virtualized resources on a pay-per-use basis. For large-scale heterogeneous distributed systems like a cloud, scheduling is an essential component of resource management at the application layer as well as at the virtualization layer in order to deliver the optimal Quality of Services (QoS). The cloud scheduling, in general, is an NP-hard problem due to large solution space, thus, it is difficult to find an optimal solution within a reasonable time. In application layer scheduling, the tasks are mapped to logical resources (i.e., virtual machines), aiming to optimize one or more QoS parameters, and conforming to several constraints. Various algorithms have been proposed in the literature for application layer scheduling, where each of them is based on some fundamental design techniques like simple heuristics, meta-heuristics, and most recently hybrid heuristics. Although ample literature survey exists for cloud scheduling algorithms, none of them present their study exclusively for the application layer. In this survey paper, we present a study on task scheduling algorithms used only at the application layer of the cloud. We classify our study according to various fundamental techniques used in designing such scheduling algorithms. One of the main features of our study is that it covers numerous application type e.g., a set of independent tasks, simple workflow, scientific workflow, and MapReduce jobs. We also provide a comparative analysis of existing algorithms on various parameters like makespan, cost, resource utilization, etc. In the end, research directions for future work have been provided.     

Service-level agreement–aware scheduling and load balancing of tasks in cloud

Cloud computing is an innovative computing paradigm designed to provide a flexible and low-cost way to deliver information technology services on demand over the Internet. Proper scheduling and load balancing of the resources are required for the efficient operations in the distributed cloud environment. Since cloud computing is growing rapidly and customers are demanding better performance and more services, scheduling and load balancing of the cloud resources have become very interesting and important area of research. As more and more consumers assign their tasks to cloud, service-level agreements (SLAs) between consumers and providers are emerging as an important aspect. The proposed prediction model is based on the past usage pattern and aims to provide optimal resource management without the violations of the agreed service-level conditions in cloud data centers. It considers SLA in both the initial scheduling stage and in the load balancing stage, and it looks into different objectives to achieve the minimum makespan, the minimum degree of imbalance, and the minimum number of SLA violations. The experimental results show the effectiveness of the proposed system compared with other state-of-the-art algorithms.     

Fitness rate-based rider optimization enabled for optimal task scheduling in cloud

ABSTRACT

Not long ago, there has been a dramatic augment in the attractiveness of cloud computing systems that depends computing resources on-demand, bill on a pay-as-you-go basis, and multiplex many users on the same physical infrastructure. It is considered as an essential pool of resources, which are offered to users through Internet. Without troubling the fundamental infrastructure, pay-per-use computing resources are provided to the users by the cloud computing technology. Scheduling is a significant dilemma in cloud computing as a cloud provider has to serve multiple users in cloud environment. This proposal plans to implement an optimal task scheduling model in cloud sector as a challenge over the existing technologies. The proposed model solves the task scheduling problem using an improved meta-heuristic algorithm called Fitness Rate-based Rider Optimization Algorithm (FR-ROA), which is the advanced form of conventional Rider Optimization Algorithm (ROA). The objective constraints considered for optimal task scheduling are the maximum makespan or completion time, and the sum of the completion times of entire tasks. Since the proposed FR-ROA has attained the advantageous part of reaching the convergence in a small duration, the proposed model will outperform the other conventional algorithms for accomplishing the optimal task scheduling in cloud environment.     

Assessment of interoperability in cloud manufacturing

Cloud manufacturing is defined as a resource sharing paradigm that provides on-demand access to a pool of manufacturing resources and capabilities aimed at utilising geographically dispersed manufacturing resources in a service-oriented manner. These services are deployed via the Industrial Internet of Things (IIoT) and its underlying IT infrastructure, architecture models, as well as data and information exchange protocols and standards. In this context, interoperability has been identified to be a key enabler for implementing such vertically or horizontally integrated cyber-physical systems for production engineering. Adopting an interoperability framework for cloud manufacturing systems enables an efficient deployment of manufacturing resources and capabilities across the production engineering life-cycle. In this paper, the authors investigate interoperability in the context of cloud manufacturing to identify the key parameters that determine whether or not a change-over from traditional cloud manufacturing to interoperable cloud manufacturing is financially viable for a given scenario of service providers and manufacturing orders. The results obtained confirm that interoperable cloud manufacturing systems cannot be considered a one-size-fits-all option. Rather, its applicability depends on a number of driving parameters that need to be analysed and interpreted to determine whether or not it provides a financially viable alternative to cloud manufacturing without an overarching interoperability framework.     

基于深度强化学习的电力物资配送多目标路径优化

针对现有电力物资车辆路径问题（EVRP）优化时考虑目标函数较为单一、约束不够全面,并且传统求解算法效率不高的问题,提出一种基于深度强化学习（DRL）的电力物资配送多目标路径优化模型和求解算法。首先,充分考虑了电力物资配送区域的加油站分布情况、物资运输车辆的油耗等约束,建立了以电力物资配送路径总长度最短、成本最低、物资需求点满意度最高为目标的多目标电力物资配送模型;其次,设计了一种基于DRL的电力物资配送路径优化算法DRL-EVRP求解所提模型。DRL-EVRP使用改进的指针网络（Ptr-Net）和Q-学习（Q-learning）算法结合的深度Q-网络（DQN）来将累积增量路径长度的负值与满意度之和作为奖励函数。所提算法在进行训练学习后,可直接用于电力物资配送路径规划。仿真实验结果表明,DRL-EVRP求解得到的电力物资配送路径总长度相较于扩展C-W(ECW)节约算法、模拟退火（SA）算法更短,且运算时间在可接受范围内,因此所提算法能更加高效、快速地进行电力物资配送路径优化。     

拟双曲动量梯度的对抗深度强化学习研究

在深度强化学习（Deep Reinforcement Learning,DRL）中,智能体（agent）通过观察通道来观察环境状态。该观察可能包含对抗性攻击的干扰,也即对抗样本,使智能体选择了错误动作。生成对抗样本常用方法是采用随机梯度下降方法。提出使用拟双曲动量梯度算法（QHM）来生成对抗干扰,该方法能够充分利用以前的梯度动量来修正梯度下降方向,因而比采用随机梯度下降方法（SGD）在生成对抗样本上具有更高效率。同时借助这种攻击方法在鲁棒控制框架内训练了DRL鲁棒性。实验效果表明基于QHM训练方法的DRL在进行对抗性训练后,面对攻击和环境参数变化时的鲁棒性显著提高。     

Research on the collaboration of service selection and resource scheduling for IoT simulation workflows

Cloud resources provide a promising way to efficiently perform the needed simulation tasks for a complex manufacturing process. Most of the existing work focuses only on how to effectively schedule computing resources to execute computing requirements of simulation workflows in Internet of Things (IoT) applications. Research on the scheduling of simulation workflows in consideration of task ordering, service selection, and resource allocation altogether has not been lacking. To fill in this void, this paper proposes a cloud-based 3-stage workflow scheduling model. Before scheduling computing resources to complete task requirements, the order of the tasks is determined and the services that can meet the task requirements are selected. In this model, the workload to satisfy task requirements is not fixed and takes on a different value depending upon the service selected with its unique complexity and accuracy. An optimization function that transforms and integrates makespan, cost, and accuracy in a unique way is proposed. For its solution, the relatively new symbiotic organisms search (SOS) algorithm is modified and two SOS-based optimization strategies are developed, i.e., joint optimization-based SOS (JOSOS) and split optimization-based SOS (SOSOS). The simulation results reveal that SOS-based algorithms, especially the SOSOS method, outperform all compared algorithms. Based on the proposed method, simulation services and computing resources can be rationally selected and scheduled to ensure the requirements of IoT applications.     

Normalization-Based Task Scheduling Algorithms for Heterogeneous Multi-Cloud Environment

Cloud computing is one of the most successful technologies that offer on-demand services through the Internet. However, datacenters of the clouds may not have unlimited capacity which can fulfill the demanded services in peak hours. Therefore, scheduling workloads across multiple clouds in a federated manner has gained a significant attention in the recent years. In this paper, we present four task scheduling algorithms, called CZSN, CDSN, CDN and CNRSN for heterogeneous multi-cloud environment. The first two algorithms are based on traditional normalization techniques, namely z-score and decimal scaling respectively which are hired from data mining. The next two algorithms are based on two newly proposed normalization techniques, called distribution scaling and nearest radix scaling respectively. All the proposed algorithms are shown to work on-line. We perform rigorous experiments on the proposed algorithms using various synthetic as well as benchmark datasets. Their performances are evaluated through simulation run by measuring two performance metrics, namely makespan and average cloud utilization. The experimental results are compared with that of existing algorithms to show the efficacy of the proposed algorithms.