Real-time scheduling for distributed permutation flowshops with dynamic job arrivals using deep reinforcement learning

Distributed manufacturing plays an important role for large-scale companies to reduce production and transportation costs for globalized orders. However, how to real-timely and properly assign dynamic orders to distributed workshops is a challenging problem. To provide real-time and intelligent decision-making of scheduling for distributed flowshops, we studied the distributed permutation flowshop scheduling problem (DPFSP) with dynamic job arrivals using deep reinforcement learning (DRL). The objective is to minimize the total tardiness cost of all jobs. We provided the training and execution procedures of intelligent scheduling based on DRL for the dynamic DPFSP. In addition, we established a DRL-based scheduling model for distributed flowshops by designing suitable reward function, scheduling actions, and state features. A novel reward function is designed to directly relate to the objective. Various problem-specific dispatching rules are introduced to provide efficient actions for different production states. Furthermore, four efficient DRL algorithms, including deep Q-network (DQN), double DQN (DbDQN), dueling DQN (DlDQN), and advantage actor-critic (A2C), are adapted to train the scheduling agent. The training curves show that the agent learned to generate better solutions effectively and validate that the system design is reasonable. After training, all DRL algorithms outperform traditional meta-heuristics and well-known priority dispatching rules (PDRs) by a large margin in terms of solution quality and computation efficiency. This work shows the effectiveness of DRL for the real-time scheduling of dynamic DPFSP.     

Hybrid genetic algorithm for multi-time period production/distribution planning

In this paper, we deal with a production/distribution problem to determine an efficient integration of production, distribution and inventory system so that products are produced and distributed at the right quantities, to the right customers, and at the right time, in order to minimize system wide costs while satisfying all demand required. This problem can be viewed as an optimization model that integrates facility location decisions, distribution costs, and inventory management for multi-products and multi-time periods. To solve the problem, we propose a new technique called spanning tree-based genetic algorithm (hst-GA). In order to improve its efficiency, the proposed method is hybridized with the fuzzy logic controller (FLC) concept for auto-tuning the GA parameters. The proposed method is compared with traditional spanning tree-based genetic algorithm approach. This comparison shows that the proposed method gives better results.     

Multi-objective optimal scheduling of reconfigurable assembly line for cloud manufacturing

We propose a multi-objective optimization scheduling model to improve the production efficiency of a reconfigurable assembly line. We aim to minimize the costs of assembly line reconstruction, achieve the production load equalization, and minimize the delayed workload using this model. However, the proposed multi-objective optimization model is significantly complex for conventional mathematical optimization methods. Thus, we present an efficient solution approach based on a distance sorting particle swarm optimization. Finally, a case study is conducted to illustrate the feasibility and efficiency of the proposed method. Experimental results indicate that our proposed approach can significantly improve the production efficiency (i.e. increased production load balance, minimized reconstruction cost, and minimized delayed workload).     

考虑转包的平行机供应链排序

研究一类考虑转包的供应链排序问题, 即工厂从客户处接受一批订单, 这些订单既可以由工厂完成, 也可以通过支付一定费用进行转包. 工厂需要确定被转包的订单集并安排未被转包订单的生产和运输. 针对工厂为平行机生产环境的情况, 以交货期限内完成所有订单的转包成本、生产成本与运输成本之和最小化为目标, 构建了问题的数学模型, 并设计了启发式算法. 最后通过数值实验结果表明了算法的有效性.

     

Scheduling Optimization Modelling: A Case Study of a Woven Label Manufacturing Company

Production scheduling involves all activities of building production schedules, including coordinating and assigning activities to each person, group of people, or machine and arranging work orders in each workplace. Production scheduling must solve all problems such as minimizing customer wait time, storage costs, and production time; and effectively using the enterprise’s human resources. This paper studies the application of flexible job shop modelling on scheduling a woven labelling process. The labelling process includes several steps which are handled in different work-stations. Each workstation is also comprised of several identical parallel machines. In this study, job splitting is allowed so that the power of work stations can be utilized better. The final objective is to minimize the total completion time of all jobs. The results show a significant improvement since the new planning may save more than 60% of lead time compared to the current schedule. The contribution of this research is to propose a flexible job shop model for scheduling a woven labelling process. The proposed approach can also be applied to support complex production scheduling processes under fuzzy environments in different industries. A practical case study demonstrates the effectiveness of the proposed model.     

软计算求解分布式多工厂单件制造业的提前/拖期生产计划问题

描述了分布式多工厂单件制造企业准时化生产计划问题, 以实现最小化提前/拖期惩罚费用、生产成本、产品运输费用之和为目标建立了0-1规划数学模型; 设计了基于模糊规则量化的方法求解模糊决策, 并将模糊决策嵌入到遗传算法中的软计算方法求解模型, 使得算法具有比分枝定界法更快速的寻找优解的能力以及更广泛的适应范围. 结果表明了该模型和算法的有效性和应用潜力.     

软计算求解分布式多工厂多顾客的供应链准时化生产计划问题

描述了分布式多工厂、多顾客的供应链准时化生产计划问题,以实现最小化提前／拖期惩罚费用、生产成本、产品运输费用之和为目标建立了数学模型,将遗传算法与模糊逻辑相结合,设计了软计算方法求解模型,采用基于规则方法的模糊规则量化方法求解模糊决策,并将模糊决策嵌入遗传算法,使得算法具有比分枝定界法更快的寻优能力和更广的适应范围。实例计算结果表明了该模型和算法的有效性和应用潜力。     

Reverse engineering approach to appearance-based design of metallic and pearlescent paints

We propose a new approach to interactive design of metallic and pearlescent coatings, such as automotive paints and plastic finishes of electronic appliances. This approach includes solving the inverse problem, that is, finding pigment composition of a paint from its bidirectional reflectance distribution function (BRDF) based on a simple paint model. The inverse problem is solved by two consecutive optimizations calculated in real-time on a contemporary PC. Such reverse engineering can serve as a starting point for subsequent design of new paints in terms of appearance attributes that are directly connected to the physical parameters of our model. This allows the user to have a paint composition in parallel with the appearance being designed.     

A large neighborhood search approach for the paint shop scheduling problem

Journal of Scheduling - Minimizing the setup costs caused by color changes is one of the main concerns for paint shop scheduling in the automotive industry. Yet, finding an optimized color sequence...     

可重入混合流水车间调度的拉格朗日松弛算法

为了有效提升多重入车间的生产效率,考虑了实际生产中检查和修复过程对于逐层制造的可重入生产系统的重要性,提出了基于拉格朗日松弛算法的可重入混合流水车间的调度方法.首先进行了问题域的描述,并在此基础上以最小化加权完成时间为调度目标,建立数学规划模型.针对该调度问题提出了基于松弛机器能力约束的拉格朗日松弛算法,使松弛问题分解成工件级子问题,并使用动态规划方法建立递归公式,求解工件级子问题.随后,使用次梯度算法求解拉格朗日对偶问题.最后,对各种不同问题规模进行了仿真实验,结果表明,所提出的调度算法能够在合理的时间内获得满意的近优解.     

Paint batching problem on M-to-1 conveyor systems

An M-to-1 conveyor system consists of multiple upstream conveyors and a single downstream conveyor. In this paper, we investigate the paint batching problem on M-to-1 conveyor systems with the objective of minimizing setup costs. Our research is motivated by a vehicle re-sequencing problem at a major Korean automotive manufacturer. Setup costs are incurred when two consecutive jobs in the downstream conveyor do not share the same feature. Re-sequencing flexibility is limited by the precedence relationship among jobs in the upstream conveyors. First, we develop a mixed integer linear programming model and propose an efficient dynamic programming (DP) algorithm for a 2-to-1 conveyor system. However, because the suggested DP cannot guarantee optimality in general settings, we propose two efficient genetic algorithms (GAs) to find near optimal solutions. Specifically, we design the reordering operation for making offspring to satisfy the precedence condition. We show that the proposed GAs perform prominently with respect to optimality gap and computation time; thus, they are amenable to environments where solutions must be obtained within tight time constraints.     

基于平均神经网络参数的DQN算法

在深度强化学习领域,如何有效地探索环境是一个难题。深度Q网络(Deep Q-Network,DQN)使用ε-贪婪策略来探索环境,ε的大小和衰减需要人工进行调节,而调节不当会导致性能变差。这种探索策略不够高效,不能有效解决深度探索问题。针对DQN的ε-贪婪策略探索效率不够高的问题,提出一种基于平均神经网络参数的DQN算法(Averaged Parameters DQN,AP-DQN)。该算法在回合开始时,将智能体之前学习到的多个在线值网络参数进行平均,得到一个扰动神经网络参数,然后通过扰动神经网络进行动作选择,从而提高智能体的探索效率。实验结果表明,AP-DQN算法在面对深度探索问题时的探索效率优于DQN,在5个Atari游戏环境中相比DQN获得了更高的平均每回合奖励,归一化后的得分相比DQN最多提升了112.50%,最少提升了19.07%。     

分布式多工厂、多分销商的供应链生产计划模型

描述了分布式多工厂、多分销商的供应链生产 计划,以实现最小化提前/拖期惩罚费用、生产成本、产品运输费用的总额为目标建立了模型 ,通过模型转换,求解得到了其生产计划调度方案.计算结果证明了模型的有效性和可行性.     

基于DQN的多类型拦截装备复合式反无人机任务分配方法

针对当前反无人系统无法有效压制无人机的问题,使用多种拦截装备构建一种新的反无人机方法.传统多目标优化算法无法解决动态的任务分配问题,对此,提出一种基于深度Q网络(DQN)的多类型拦截装备复合式反无人机任务分配模型. DQN模块对任务分配问题进行初期决策.为了提高算法收敛速度和学习效率,该方法未采用下一时刻的状态来预测Q值,而是采用当前时刻的状态来预测Q值,消除训练过程中Q值过估计的影响.之后采用进化算法对决策结果进行优化,输出多个拦截方案.以国内某机场跑道周围区域开阔地为防护对象,构建反无人机系统的任务分配仿真环境,仿真结果验证了所提出方法的有效性.同时,将DQN与Double DQN方法相比,所提出改进DQN算法训练的智能体表现更为精确,并且算法的收敛性和所求解的表现更为优异.所提出方法为反无人机问题提供了新的思路.     

基于深度强化学习的节能工艺路线发现方法

由于传统基于固定加工环境的工艺路线制定规则,无法快速响应加工环境的动态变化制定节能工艺路线。因此提出了基于深度Q网络(deep Q network,DQN)的节能工艺路线发现方法。基于马尔可夫决策过程,定义状态向量、动作空间、奖励函数,建立节能工艺路线模型,并将加工环境动态变化的节能工艺路线规划问题,转化为DQN智能体决策问题,利用决策经验的可复用性和可扩展性,进行求解,同时为了提高DQN的收敛速度和解的质量,提出了基于S函数探索机制和加权经验池,并使用了双Q网络。仿真结果表明,相比较改进前,改进后的算法在动态加工环境中能够更快更好地发现节能工艺路线;与遗传算法、模拟退火算法以及粒子群算法相比,改进后的算法不仅能够以最快地速度发现节能工艺路线,而且能得到相同甚至更高精度的解。     

A new algorithm for optimally determining lot-sizing policies for a deteriorating item in an integrated production-inventory system

In this study, we focus on optimally determining lot-sizing policies for a deteriorating item among all the partners in a supply chain system with a single-vendor and multiple-buyers so as to minimize the average total costs. We revise Yang and Wee's [1] model using the Fourier series to precisely estimate the vendor's inventory holding costs. Also, we transform our revised model into a more concise version by applying anapproximation to the exponential terms in the objective function. In order to solve this problem, we analyze the optimality structure of our revised model and derive several interesting properties. By utilizing our theoretical results, we propose a search algorithm that can efficiently solve the optimal solution. Based on our numerical experiments, we show that the proposed algorithm outperforms the existing solution approach in the literature, especially when the number of buyers is larger in the supply chain system.     

基于二阶时序差分误差的双网络DQN算法

针对深度Q网络(DQN)算法因过估计导致收敛稳定性差的问题,在传统时序差分(TD)的基础上提出N阶TD误差的概念,设计基于二阶TD误差的双网络DQN算法。构造基于二阶TD误差的值函数更新公式,同时结合DQN算法建立双网络模型,得到两个同构的值函数网络分别用于表示先后两轮的值函数,协同更新网络参数,以提高DQN算法中值函数估计的稳定性。基于Open AI Gym平台的实验结果表明,在解决Mountain Car和Cart Pole问题方面,该算法较经典DQN算法具有更好的收敛稳定性。     

Flow shop scheduling for separation model of set-up and net process based on Branch-and-Bound method

Lots of research reports on flow shop scheduling problems have been reported. Generally speaking, these models are applicable to a simple model with no separation of set-up processes and net ones. In many production lines, we cannot ignore the set-up times in comparison with the net processing times. We can expect to shorten the total processing time by executing the set-up processes and net ones in parallel. We need a parallel operation model to improve schedule results.We will propose a new scheduling method for multi-stage flow shops. The aim of the method is to shorten the total processing time by operating the set-up processes and the net ones of each job in parallel. We applied the Branch-and-Bound method and developed a new calculation algorithm for the lower bound estimation of the total processing time. Finally, we will evaluate our proposed method by some numerical experiments using actual production line data.     

基于灾变机制的预防性维护和生产调度集成优化方法

为了解决生产车间中多品种任务的生产调度与预防性维护集成优化问题,综合考虑其加工顺序、生产批量及预防性维护策略等要素,在订单充足的前提下,以总制造成本和加工时间最小化为联合优化目标,建立了生产调度与预防性维护集成优化模型。针对模型特点,在非支配排序遗传算法框架的基础上,基于灾变机制和荣誉空间,引入截断和拼接操作算子,提出一种变长度染色体单亲遗传算法对模型进行求解,并在不同参数条件和问题规模下,通过仿真实验验证了该算法解决复杂生产任务调度和预防性维护集成优化问题的有效性。     

基于动态融合目标的深度强化学习算法研究

针对深度强化学习算法中存在的过估计问题,提出了一种目标动态融合机制,在Deep [Q] Networks（DQN）算法基础上进行改进,通过融合Sarsa算法的在线更新目标,来减少DQN算法存在的过估计影响,动态地结合了DQN算法和Sarsa算法各自优点,提出了DTDQN（Dynamic Target Deep [Q] Network）算法。利用公测平台OpenAI Gym上Cart-Pole控制问题进行仿真对比实验,结果表明DTDQN算法能够有效地减少值函数过估计,具有更好的学习性能,训练稳定性有明显提升。