Scheduling of decentralized robot services in cloud manufacturing with deep reinforcement learning

Cloud manufacturing is a service-oriented manufacturing model that offers manufacturing resources as cloud services. Robots are an important type of manufacturing resources. In cloud manufacturng, large-scale distributed robots are encapsulated into cloud services and provided to consumers in an on-demand manner. How to effectively and efficiently manage and schedule decentralized robot services in cloud manufacturing to achieve on-demand provisioning is a challenging issue. During the past few years, Deep Reinforcement Learning (DRL) has become very popular and successfully been applied to many different areas such as games, robotics, and manufacturing. DRL also holds tremendous potential for solving scheduling issues in cloud manufacturing. To this end, this paper is devoted to exploring effective approaches for scheduling of decentralized robot manufacturing services in cloud manufacturing with DRL. Specifically, both Deep Q-Networks (DQN) and Dueling Deep Q-Networks (DDQN)-based scheduling algorithms are proposed. Performance of different algorithms, including DQN, DDQN, and other three benchmark algorithms, indicates that DDQN performs the best with respect to each indicator. Effects of different combinations of weight coefficients and influencing degrees of different indicators on the overall scheduling objective are analyzed. Results indicate that the DDQN-based scheduling algorithm is able to generate scheduling solutions efficiently.     

Deep reinforcement learning based control for Autonomous Vehicles in CARLA

Nowadays, Artificial Intelligence (AI) is growing by leaps and bounds in almost all fields of technology, and Autonomous Vehicles (AV) research is one more of them. This paper proposes the using of algorithms based on Deep Learning (DL) in the control layer of an autonomous vehicle. More specifically, Deep Reinforcement Learning (DRL) algorithms such as Deep Q-Network (DQN) and Deep Deterministic Policy Gradient (DDPG) are implemented in order to compare results between them. The aim of this work is to obtain a trained model, applying a DRL algorithm, able of sending control commands to the vehicle to navigate properly and efficiently following a determined route. In addition, for each of the algorithms, several agents are presented as a solution, so that each of these agents uses different data sources to achieve the vehicle control commands. For this purpose, an open-source simulator such as CARLA is used, providing to the system with the ability to perform a multitude of tests without any risk into an hyper-realistic urban simulation environment, something that is unthinkable in the real world. The results obtained show that both DQN and DDPG reach the goal, but DDPG obtains a better performance. DDPG perfoms trajectories very similar to classic controller as LQR. In both cases RMSE is lower than 0.1m following trajectories with a range 180-700m. To conclude, some conclusions and future works are commented.

     

Multitask-oriented manufacturing service composition in an uncertain environment using a hyper-heuristic algorithm

One of the most important issues in cloud manufacturing involves obtaining an optimal manufacturing service composition solution. However, traditional manufacturing service composition methods either focused on single-task-oriented service composition or optimized solutions under a deterministic environment. In the study, a multitask-oriented manufacturing service composition (MMSC) model with two stages in uncertain environment is proposed. It handles the problem of multitask scheduling and also deals with the inherent uncertainty and ambiguity in cloud manufacturing including the occurrence of urgent task requests and the delayed delivery time of raw materials. In order to solve the MMSC model, a new genetic based hyper-heuristic algorithm (GA-HH) with adjustable length of chromosome is proposed. The GA-HH contains a set of low-level heuristics that directly operate on the solution domain that are organized by the high-level heuristic (i.e., genetic algorithm). Finally, the proposed GA-HH is proved as an efficient, effective, and robust algorithm to solve the MMSC model with considerations of multitask and uncertainty, by comparing it with other well-known meta-heuristic algorithms such as the genetic algorithm and particle swarm optimization.     

基于动态匹配网络的制造服务组合自适应方法

云制造（cloud manufacturing,CMfg）模式下,制造任务和制造服务都处于动态变化的环境中,制造服务组合的动态适应能力问题亟待解决.针对这一问题,以制造任务和制造服务的匹配关系为基础,构建了制造任务-制造服务动态匹配网络（dynamic matching network,DMN）理论模型,在此基础上提出了一种三阶段的制造服务组合自适应方法（three-phase manufacturing service composition self-adaptive approach,TPMSCSAA）.第一阶段通过负载队列模型对QoS进行动态评估,以负载和动态QoS为优化目标,将最优制造服务组合问题转化为制造服务网络中最短路径的搜索,实现制造服务的动态调度;第二阶段对不同类型的制造任务和制造服务变更进行实时获取,同步更新制造任务网络和制造服务网络;第三阶段触发动态调度算法,完成动态匹配边的重构.最后,通过对电梯设计服务组合的实验仿真,验证了方法的可行性和有效性.     

多目标事务模糊关联聚类的云制造服务组合算法

目前的云制造服务组合方法单纯从某个角度研究服务组合问题,对基于多目标事务的云制造服务组合的考虑不足,服务组合质量不高。为实现敏捷、智能、平稳的云制造服务组合,基于开展多目标事务的云模式通用解析、多目标事务模糊关联特征的云模式通用表示、云制造服务组合多目标事务模糊关联聚类算法等方面研究,改进反向学习算法、可替换服务推荐算法、三角模糊函数、非支配排序遗传公式,设计一种敏捷、智能、平稳的云制造服务组合算法。最后,实施实验验证,与传统算法进行性能对比分析。实验结果表明,相比传统算法,该算法组合响应时间短、误差小,且收敛性、敏捷性、智能性、动态演化性、平稳性高。因此,该算法实现了基于多目标事务模糊关联聚类的云制造服务的有效组合,具有较高的应用价值。     

基于LG-MMAS算法的制造云服务优化组合研究

制造云服务组合是一种提高云制造资源利用率,实现制造资源增值的新技术,对云制造产业的快速发展具有重要的支撑作用。随着云制造技术的日益成熟,网络上出现了大量具有相同制造功能和不同服务质量的制造云服务,如何通过这些制造云服务构建出既能满足用户制造需求,又具有最优服务质量的组合服务是云制造领域面临的难题。针对这一问题,将协作学习、变异和精英保留机制引入最大最小蚁群算法,构造了具有学习和变异能力的最大最小蚁群算法,并使用该算法求解服务质量感知的制造云服务优化组合问题。仿真实验结果验证了算法的有效性。     

面向云制造的服务组合执行引擎框架研究

云制造模式下,制造资源被封装成制造服务,通常以服务组合的方式满足用户复 杂的制造需求。针对云制造服务组合执行过程中高效性、准确性和动态性等要求,提出了面向 云制造的服务组合执行引擎框架,并详细阐述了引擎运行机制。首先,通过解析云制造服务组 合描述文件,构造服务组合节点和服务组合依赖边,建立服务组合节点参数关联关系,归纳推 倒出相应的执行规则。然后,基于执行状态变更的服务组合执行算法实现了制造服务组合的动 态执行。最后,给出了面向电梯产业联盟的云制造服务组合执行引擎实例,验证了框架的可行 性和有效性。     

Real-time scheduling for distributed permutation flowshops with dynamic job arrivals using deep reinforcement learning

Distributed manufacturing plays an important role for large-scale companies to reduce production and transportation costs for globalized orders. However, how to real-timely and properly assign dynamic orders to distributed workshops is a challenging problem. To provide real-time and intelligent decision-making of scheduling for distributed flowshops, we studied the distributed permutation flowshop scheduling problem (DPFSP) with dynamic job arrivals using deep reinforcement learning (DRL). The objective is to minimize the total tardiness cost of all jobs. We provided the training and execution procedures of intelligent scheduling based on DRL for the dynamic DPFSP. In addition, we established a DRL-based scheduling model for distributed flowshops by designing suitable reward function, scheduling actions, and state features. A novel reward function is designed to directly relate to the objective. Various problem-specific dispatching rules are introduced to provide efficient actions for different production states. Furthermore, four efficient DRL algorithms, including deep Q-network (DQN), double DQN (DbDQN), dueling DQN (DlDQN), and advantage actor-critic (A2C), are adapted to train the scheduling agent. The training curves show that the agent learned to generate better solutions effectively and validate that the system design is reasonable. After training, all DRL algorithms outperform traditional meta-heuristics and well-known priority dispatching rules (PDRs) by a large margin in terms of solution quality and computation efficiency. This work shows the effectiveness of DRL for the real-time scheduling of dynamic DPFSP.     

Manufacturing in the cloud: A human factors perspective

Cloud manufacturing adopts a cloud computing paradigm as the basis for delivering shared, on-demand manufacturing services. The result is customer-centric supply chains that can be configured for cost, quality, speed and customisation. While the technical capabilities required for cloud manufacturing are a current focus, there are many emerging questions relating to the impact, both positive and negative, on the people consuming or supporting cloud manufacturing services. Human factors can have a pivotal role in enabling the success and adoption of cloud manufacturing, while ensuring the safety, well-being and optimum user experience of those involved in a cloud manufacturing environment. This paper presents these issues, structured around groups of users (service providers, application providers and consumers). We also consider the issues of collaboration that are likely to arise from the manufacturing cloud. From this analysis we discuss the central role of human factors as an enabler of cloud manufacturing, and the opportunities that emerge.     

Multi-population parallel self-adaptive differential artificial bee colony algorithm with application in large-scale service composition for cloud manufacturing

As a new service-oriented smart manufacturing paradigm, cloud manufacturing (CMfg) aims at fully sharing and circulation of manufacturing capabilities towards socialization, in which composite CMfg service optimal selection (CCSOS) involves selecting appropriate services to be combined as a composite complex service to fulfill a customer need or a business requirement. Such composition is one of the most difficult combination optimization problems with NP-hard complexity. For such an NP-hard CCSOS problem, this study proposes a new approach, called multi-population parallel self-adaptive differential artificial bee colony (MPsaDABC) algorithm. The proposed algorithm adopts multiple parallel subpopulations, each of which evolves according to different mutation strategies borrowed from the differential evolution (DE) to generate perturbed food sources for foraging bees, and the control parameters of each mutation strategy are adapted independently. Moreover, the size of each subpopulation is dynamically adjusted based on the information derived from the search process. Different scales of the CCSOS problems are conducted to validate the effectiveness of the proposed algorithm, and the experimental results show that the proposed algorithm has superior performance over other hybrid and single population algorithms, especially for complex CCSOS problems.     

Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network

To benefit from the accurate simulation and high-throughput data contributed by advanced digital twin technologies in modern smart plants, the deep reinforcement learning (DRL) method is an appropriate choice to generate a self-optimizing scheduling policy. This study employs the deep Q-network (DQN), which is a successful DRL method, to solve the dynamic scheduling problem of flexible manufacturing systems (FMSs) involving shared resources, route flexibility, and stochastic arrivals of raw products. To model the system in consideration of both manufacturing efficiency and deadlock avoidance, we use a class of Petri nets combining timed-place Petri nets and a system of simple sequential processes with resources (S³PR), which is named as the timed S³PR. The dynamic scheduling problem of the timed S³PR is defined as a Markov decision process (MDP) that can be solved by the DQN. For constructing deep neural networks to approximate the DQN action-value function that maps the timed S³PR states to scheduling rewards, we innovatively employ a graph convolutional network (GCN) as the timed S³PR state approximator by proposing a novel graph convolution layer called a Petri-net convolution (PNC) layer. The PNC layer uses the input and output matrices of the timed S³PR to compute the propagation of features from places to transitions and from transitions to places, thereby reducing the number of parameters to be trained and ensuring robust convergence of the learning process. Experimental results verify that the proposed DQN with a PNC network can provide better solutions for dynamic scheduling problems in terms of manufacturing performance, computational efficiency, and adaptability compared with heuristic methods and a DQN with basic multilayer perceptrons.     

基于改进北极熊算法的制造云服务组合优化

为了提高云制造环境下制造服务组合优化的效率,提出了一种基于改进北极熊算法的制造云服务组合优化方法。该方法对制造服务进行实数编码,并以服务功能和服务质量为评价指标,使用改进的北极熊算法对制造云服务组合优化问题进行求解,得到最优的服务组合方案。同时通过引入动态视野,对算法的局部搜索进行调整,并与遗传算法中的变异策略相结合,以提高求解多目标问题的效率,同时降低因初始参数影响而导致算法陷入局部最优的可能。算例分析表明,改进的北极熊算法在求解制造云服务组合优化问题上比原始北极熊算法、标准遗传算法、改进的灰狼优化算法和改进的粒子群优化算法具有更高的效率。     

基于深度强化学习的电力物资配送多目标路径优化

针对现有电力物资车辆路径问题（EVRP）优化时考虑目标函数较为单一、约束不够全面,并且传统求解算法效率不高的问题,提出一种基于深度强化学习（DRL）的电力物资配送多目标路径优化模型和求解算法。首先,充分考虑了电力物资配送区域的加油站分布情况、物资运输车辆的油耗等约束,建立了以电力物资配送路径总长度最短、成本最低、物资需求点满意度最高为目标的多目标电力物资配送模型;其次,设计了一种基于DRL的电力物资配送路径优化算法DRL-EVRP求解所提模型。DRL-EVRP使用改进的指针网络（Ptr-Net）和Q-学习（Q-learning）算法结合的深度Q-网络（DQN）来将累积增量路径长度的负值与满意度之和作为奖励函数。所提算法在进行训练学习后,可直接用于电力物资配送路径规划。仿真实验结果表明,DRL-EVRP求解得到的电力物资配送路径总长度相较于扩展C-W（ECW）节约算法、模拟退火（SA）算法更短,且运算时间在可接受范围内,因此所提算法能更加高效、快速地进行电力物资配送路径优化。     

Efficient Multiagent Policy Optimization Based on Weighted Estimators in Stochastic Cooperative Environments

Multiagent deep reinforcement learning (MA-DRL) has received increasingly wide attention. Most of the existing MA-DRL algorithms, however, are still inefficient when faced with the non-stationarity due to agents changing behavior consistently in stochastic environments. This paper extends the weighted double estimator to multiagent domains and proposes an MA-DRL framework, named Weighted Double Deep Q-Network (WDDQN). By leveraging the weighted double estimator and the deep neural network, WDDQN can not only reduce the bias effectively but also handle scenarios with raw visual inputs. To achieve efficient cooperation in multiagent domains, we introduce a lenient reward network and scheduled replay strategy. Empirical results show that WDDQN outperforms an existing DRL algorithm (double DQN) and an MA-DRL algorithm (lenient Q-learning) regarding the averaged reward and the convergence speed and is more likely to converge to the Pareto-optimal Nash equilibrium in stochastic cooperative environments.     

基于业务流程的制造云服务组合模型

为了提高云制造系统中制造云服务的组合成功率,实现组合云服务与用户业务需求的准确匹配,在对制造云服务、流程节点任务、云服务的可组合性和流程匹配进行形式化描述的基础上,提出一种基于业务流程的制造云服务组合模型。该模型由业务流程引擎、业务流程、选择逻辑、评估逻辑、监控逻辑、知识库和原子云服务集构成,在功能匹配的基础上,对候选服务的可组合性进行检查,结合负载、服务质量(QoS)和业务流程信息,选择合适的云服务,并将其挂接在业务流程上实现制造云服务的组合。对制造云服务的组合流程进行了详细描述,并给出云服务组合的实现方法。实例分析表明,该模型能够有效地选择满足业务需求的云服务实体并进行组合,从而提高制造云服务的组合成功率,保障用户制造活动的顺利进行。     

Multi-objective service composition model based on cost-effective optimization

The widespread application of cloud computing results in the exuberant growth of services with the same functionality. Quality of service (QoS) is mostly applied to represent nonfunctional properties of services, and has become an important basis for service selection. The object of most existing optimization methods is to maximize the QoS, which restricts the diversity of users’ requirements. In this paper, instead of optimization for the single object, we take maximization of QoS and minimization of cost as two objects, and a novel multi-objective service composition model based on cost-effective optimization is designed according to the complicated QoS requirements of users. Furthermore, to solve this complex optimization problem, the Elite-guided Multi-objective Artificial Bee Colony (EMOABC) algorithm is proposed from the addition of fast nondominated sorting method, population selection strategy, elite-guided discrete solution generation strategy and multi-objective fitness calculation method into the original ABC algorithm. The experiments on two datasets demonstrate that EMOABC has an advantage both on the quality of solution and efficiency as compared to other algorithms. Therefore, the proposed method can be better applicable to the cloud services selection and composition.     

深度强化学习及在路径规划中的研究进展

路径规划的目的是让机器人在移动过程中既能避开障碍物,又能快速规划出最短路径。在分析基于强化学习的路径规划算法优缺点的基础上,引出能够在复杂动态环境下进行良好路径规划的典型深度强化学习DQN（Deep Q-learning Network）算法。深入分析了DQN算法的基本原理和局限性,对比了各种DQN变种算法的优势和不足,进而从训练算法、神经网络结构、学习机制、AC（Actor-Critic）框架的多种变形四方面进行了分类归纳。提出了目前基于深度强化学习的路径规划方法所面临的挑战和亟待解决的问题,并展望了未来的发展方向,可为机器人智能路径规划及自动驾驶等方向的发展提供参考。     

基于自适应粒子群算法的制造云服务组合研究

针对云制造系统中制造云服务组合的多目标规划问题,研究建立了问题模型并提出了求解方法。首先引入了网格制造模式的制造资源服务组合技术,探讨并描述了云制造模式中基于服务质量（QoS）的制造云服务组合过程;接着通过分析云制造模式下制造云服务的特征并基于制造领域知识,研究定义了制造云服务的八维QoS评估标准及计算表达式,推导出制造组合云服务的QoS表达,进而建立了制造云服务组合的多目标规划问题模型。最终设计了自适应粒子群算法来解决该多目标规划问题。仿真实验表明,该算法能有效并高效地解决该问题,且求解效率优于传统粒子群算法。     

云制造服务综合评价方法研究

作为云制造平台中的关键技术之一,云制造服务的综合评价对于整个云制造资源配置至关重要,在云制造服务综合评价中,服务资源的评价是综合评价的基础。针对云制造服务评价问题,根据云制造服务评价体系中评价指标的多样性与云制造服务特点相匹配,基于模糊理论中的模糊数学,建立了模糊综合评价模型,并确定了由层次分析法和熵权法组合确定权重的方法。再根据云制造环境下评价指标的构建原则,选取了七个一级指标和十六个二级评价指标,以上述指标构建了多层次云制造服务评价因素集和对象评语集,从而建立了云制造服务资源综合评价指标体系。最后,通过云制造服务实例验证了该云制造服务综合评价模型及模糊综合评价算法的操作简便性及实用性。