Intelligent fault diagnosis for rotating machinery using deep Q-network based health state classification: A deep reinforcement learning approach

Fault diagnosis methods for rotating machinery have always been a hot research topic, and artificial intelligence-based approaches have attracted increasing attention from both researchers and engineers. Among those related studies and methods, artificial neural networks, especially deep learning-based methods, are widely used to extract fault features or classify fault features obtained by other signal processing techniques. Although such methods could solve the fault diagnosis problems of rotating machinery, there are still two deficiencies. (1) Unable to establish direct linear or non-linear mapping between raw data and the corresponding fault modes, the performance of such fault diagnosis methods highly depends on the quality of the extracted features. (2) The optimization of neural network architecture and parameters, especially for deep neural networks, requires considerable manual modification and expert experience, which limits the applicability and generalization of such methods. As a remarkable breakthrough in artificial intelligence, AlphaGo, a representative achievement of deep reinforcement learning, provides inspiration and direction for the aforementioned shortcomings. Combining the advantages of deep learning and reinforcement learning, deep reinforcement learning is able to build an end-to-end fault diagnosis architecture that can directly map raw fault data to the corresponding fault modes. Thus, based on deep reinforcement learning, a novel intelligent diagnosis method is proposed that is able to overcome the shortcomings of the aforementioned diagnosis methods. Validation tests of the proposed method are carried out using datasets of two types of rotating machinery, rolling bearings and hydraulic pumps, which contain a large number of measured raw vibration signals under different health states and working conditions. The diagnosis results show that the proposed method is able to obtain intelligent fault diagnosis agents that can mine the relationships between the raw vibration signals and fault modes autonomously and effectively. Considering that the learning process of the proposed method depends only on the replayed memories of the agent and the overall rewards, which represent much weaker feedback than that obtained by the supervised learning-based method, the proposed method is promising in establishing a general fault diagnosis architecture for rotating machinery.     

基于强化学习的交通情景问题决策优化

在复杂交通情景中求解出租车路径规划决策问题和交通信号灯控制问题时,传统强化学习算法在收敛速度和求解精度上存在局限性;因此提出一种改进的强化学习算法求解该类问题。首先,通过优化的贝尔曼公式和快速Q学习（SQL）机制,以及引入经验池技术和直接策略,提出一种改进的强化学习算法GSQL-DSEP;然后,利用GSQL-DSEP算法分别优化出租车路径规划决策问题中的路径长度与交通信号灯控制问题中的车辆总等待时间。相较于Q学习、快速Q学习（SQL）、、广义快速Q学习（GSQL）、Dyna-Q算法,GSQL-DSEP算法在性能测试中降低了至少18.7%的误差,在出租车路径规划决策问题中使决策路径长度至少缩短了17.4%,在交通信号灯控制问题中使车辆总等待时间最多减少了51.5%。实验结果表明,相较于对比算法,GSQL-DSEP算法对解决交通情景问题更具优势。     

交互协调强化学习下的城市交通信号配时决策

针对应用传统强化学习进行城市自适应交通信号配时决策时存在维数灾难和缺乏协调机制等问题,提出引入交互协调机制的强化学习算法。以车均延误为性能指标设计了针对城市交通信号配时决策的独立Q-强化学习算法。在此基础上,通过引入直接交互机制对独立强化学习算法进行了延伸,即相邻交叉口交通信号控制agent间直接交换配时动作和交互点值。通过仿真实验分析表明,引入交互协调机制的强化学习的控制效果明显优于独立强化学习算法,协调更有效,并且其学习算法具有较好的收敛性能,交互点值趋向稳定。     

基于学习的鲁棒自适应评判控制研究进展

在作为人工智能核心技术的机器学习领域,强化学习是一类强调机器在与环境的交互过程中进行学习的方法,其重要分支之一的自适应评判技术与动态规划及最优化设计密切相关.为了有效地求解复杂动态系统的优化控制问题,结合自适应评判,动态规划和人工神经网络产生的自适应动态规划方法已经得到广泛关注,特别在考虑不确定因素和外部扰动时的鲁棒自适应评判控制方面取得了很大进展,并被认为是构建智能学习系统和实现真正类脑智能的必要途径.本文对基于智能学习的鲁棒自适应评判控制理论与主要方法进行梳理,包括自学习鲁棒镇定,自适应轨迹跟踪,事件驱动鲁棒控制,以及自适应H_∞控制设计等,并涵盖关于自适应评判系统稳定性、收敛性、最优性以及鲁棒性的分析.同时,结合人工智能、大数据、深度学习和知识自动化等新技术,也对鲁棒自适应评判控制的发展前景进行探讨.     

模仿排序学习模型

文档排序一直是信息检索(IR)领域的关键任务之一。受益于马尔科夫决策过程强大的建模能力,以及强化学习方法强大的求解能力,近年来基于强化学习的排序模型被提出并取得了良好效果。然而,由于候选文档中会包含大量的不相关文档,导致基于“试错”的强化学习方法存在效率低下的问题。为解决上述问题,该文提出了一种基于模仿学习的排序学习算法IR-DAGGER,其基于文档标注信息构建专家策略,在保证文档排序精度的同时提高了算法的学习效率。为了测试IR-DAGGER的性能,该文基于面向相关性排序任务的OHSUMED数据集和面向多样化排序的TREC数据集进行了实验,实验结果表明IR-DAGGER在上述两个数据集上均提升了文档排序的精度和效率。     

仿生机器人运动步态控制:强化学习方法综述

仿生机器人是一类典型的多关节非线性欠驱动系统，其步态控制是一个非常具有挑战性的问题。对于该问题，传统的控制和规划方法需要针对具体的运动任务进行专门设计，需要耗费大量时间和精力，而且所设计出来的控制器往往没有通用性。基于数据驱动的强化学习方法能对不同的任务进行自主学习，且对不同的机器人和运动任务具有良好的通用性。因此，近年来这种基于强化学习的方法在仿生机器人运动步态控制方面获得了不少应用。针对这方面的研究，本文从问题形式化、策略表示方法和策略学习方法3个方面对现有的研究情况进行了分析和总结，总结了强化学习应用于仿生机器人步态控制中尚待解决的问题，并指出了后续的发展方向。     

结合状态预测的深度强化学习交通信号控制

深度强化学习（deep reinforcement learning,DRL）可广泛应用于城市交通信号控制领域,但在现有研究中,绝大多数的DRL智能体仅使用当前的交通状态进行决策,在交通流变化较大的情况下控制效果有限。提出一种结合状态预测的DRL信号控制算法。首先,利用独热编码设计简洁且高效的交通状态;然后,使用长短期记忆网络（long short-term memory,LSTM）预测未来的交通状态;最后,智能体根据当前状态和预测状态进行最优决策。在SUMO（simulation of urban mobility）仿真平台上的实验结果表明,在单交叉口、多交叉口的多种交通流量条件下,与三种典型的信号控制算法相比,所提算法在平均等待时间、行驶时间、燃油消耗、CO2排放等指标上都具有最好的性能。     

School-based optimization for performance-based optimum seismic design of steel frames

The performance-based optimum seismic design of steel frames is one of the most complicated and computationally demanding structural optimization problems. Metaheuristic optimization methods have been successfully used for solving engineering design problems over the last three decades. A very recently developed metaheuristic method called school-based optimization (SBO) will be utilized in the performance-based optimum seismic design of steel frames for the first time in this study. The SBO actually is an improved/enhanced version of teaching–learning-based optimization (TLBO), which mimics the teaching and learning process in a class where learners interact with the teacher and between themselves. Ad hoc strategies are adopted in order to minimize the computational cost of SBO results. The objective of the optimization problem is to minimize the weight of steel frames under interstory drift and strength constraints. Three steel frames previously designed by different metaheuristic methods including particle swarm optimization, improved quantum particle swarm optimization, firefly and modified firefly algorithms, teaching–learning-based optimization, and JAYA algorithm are used as benchmark optimization examples to verify the efficiency and robustness of the present SBO algorithm. Optimization results are compared with those of other state-of-the-art metaheuristic algorithms in terms of minimum structural weight, convergence speed, and several statistical parameters. Remarkably, in all test problems, SBO finds lighter designs with less computational effort than the TLBO and other methods available in metaheuristic optimization literature.

     

基于深度强化学习的有轨电车信号优先控制

现有的有轨电车信号优先控制系统存在诸多问题, 如无法适应实时交通变化、优化求解较为复杂等. 本文提出了一种基于深度强化学习的有轨电车信号优先控制策略. 不依赖于交叉口复杂交通建模, 采用实时交通信息作为输入, 在有轨电车整个通行过程中连续动态调整交通信号. 协同考虑有轨电车与社会车辆的通行需求, 在尽量保证有轨电车无需停车的同时, 降低社会车辆的通行延误. 采用深度Q网络算法进行问题求解, 并利用竞争架构、双Q网络和加权样本池改善学习性能. 基于SUMO的实验表明, 该模型能够有效地协同提高有轨电车与社会车辆的通行效率.     

基于再励学习与遗传算法的交通信号自组织控制

提出一种基于再励学习和遗传算法的交通信号自组织控制方法.再励学习针对每一个 道路交叉口交通流的优化,修正每个信号灯周期的绿信比.遗传算法则产生局部学习过程的全 局优化标准,修正信号灯周期的大小.这种方法将局部优化和全局优化统一起来,克服了现有的 控制方法需要大量数据传输通讯、准确的交通模型等缺陷.     

Effect of external pressure on the vibration analysis of higher order shear deformable FG-CNTRC spherical panels

This paper presents an alternative optimization algorithm to the literature optimizers by introducing global best-guided oppositional-based learning method. The procedure at hand uses the active and recent manipulation schemes of oppositional learning procedure by applying some modifications to them. The first part of the algorithm deals with searching the optimum solution around the current best solution by means of the ensemble learning-based strategy through which unfeasible and semi-optimum solutions have been straightforwardly eliminated. The second part of the algorithm benefits the useful merits of the quasi-oppositional learning strategy to not only improve the solution diversity but also enhance the convergence speed of the whole algorithm. A set of 22 optimization benchmark functions have been solved and corresponding results have been compared with the outcomes of the well-known literature optimization algorithms. Then, a bunch of parameter estimation problem consisting of hard-to-solve real world applications has been analyzed by the proposed method. Following that, eight widely applied constrained benchmark problems along with well-designed 12 constrained test cases proposed in CEC 2006 session have been solved and evaluated in terms of statistical analysis. Finally, a heat exchanger design problem taken from literature study has been solved through the proposed algorithm and respective solutions have been benchmarked against the prevalent optimization algorithms. Comparison results show that optimization procedure dealt with in this study is capable of achieving the utmost performance in solving multidimensional optimization algorithms.

     

面向机器人系统的虚实迁移强化学习综述

近年来,基于环境交互的强化学习方法在机器人相关应用领域取得巨大成功,为机器人行为控制策略优化提供一个现实可行的解决方案.但在真实世界中收集交互样本存在高成本以及低效率等问题,因此仿真环境被广泛应用于机器人强化学习训练过程中.通过在虚拟仿真环境中以较低成本获取大量训练样本进行策略训练,并将学习策略迁移至真实环境,能有效缓解真实机器人训练中存在的安全性、可靠性以及实时性等问题.然而,由于仿真环境与真实环境存在差异,仿真环境中训练得到的策略直接迁移到真实机器人往往难以获得理想的性能表现.针对这一问题,虚实迁移强化学习方法被提出用以缩小环境差异,进而实现有效的策略迁移.按照迁移强化学习过程中信息的流动方向和智能化方法作用的不同对象,提出一个虚实迁移强化学习系统的流程框架,并基于此框架将现有相关工作分为3大类:基于真实环境的模型优化方法、基于仿真环境的知识迁移方法、基于虚实环境的策略迭代提升方法,并对每一分类中的代表技术与关联工作进行阐述.最后,讨论虚实迁移强化学习研究领域面临的机遇和挑战.     

Empirical evaluation methods for multiobjective reinforcement learning algorithms

While a number of algorithms for multiobjective reinforcement learning have been proposed, and a small number of applications developed, there has been very little rigorous empirical evaluation of the performance and limitations of these algorithms. This paper proposes standard methods for such empirical evaluation, to act as a foundation for future comparative studies. Two classes of multiobjective reinforcement learning algorithms are identified, and appropriate evaluation metrics and methodologies are proposed for each class. A suite of benchmark problems with known Pareto fronts is described, and future extensions and implementations of this benchmark suite are discussed. The utility of the proposed evaluation methods are demonstrated via an empirical comparison of two example learning algorithms.     

双Q网络学习的迁移强化学习算法

深度强化学习在训练过程中会探索大量环境样本,造成算法收敛时间过长,而重用或传输来自先前任务(源任务)学习的知识,对算法在新任务(目标任务)的学习具有提高算法收敛速度的潜力.为了提高算法学习效率,提出一种双Q网络学习的迁移强化学习算法,其基于actor-critic框架迁移源任务最优值函数的知识,使目标任务中值函数网络对策略作出更准确的评价,引导策略快速向最优策略方向更新.将该算法用于Open AI Gym以及在三维空间机械臂到达目标物位置的实验中,相比于常规深度强化学习算法取得了更好的效果,实验证明提出的双Q网络学习的迁移强化学习算法具有较快的收敛速度,并且在训练过程中算法探索更加稳定.     

Human pose,hand and mesh estimation using deep learning: a survey

Human pose estimation is one of the issues that have gained many benefits from using state-of-the-art deep learning-based models. Human pose, hand and mesh estimation is a significant problem that has attracted the attention of the computer vision community for the past few decades. A wide variety of solutions have been proposed to tackle the problem. Deep Learning-based approaches have been extensively studied in recent years and used to address several computer vision problems. However, it is sometimes hard to compare these methods due to their intrinsic difference. This paper extensively summarizes the current deep learning-based 2D and 3D human pose, hand and mesh estimation methods with a single or multi-person, single or double-stage methodology-based taxonomy. The authors aim to make every step in the deep learning-based human pose, hand and mesh estimation techniques interpretable by providing readers with a readily understandable explanation. The presented taxonomy has clearly illustrated current research on deep learning-based 2D and 3D human pose, hand and mesh estimation. Moreover, it also provided dataset and evaluation metrics for both 2D and 3D HPE approaches.

     

Fuzzy adaptive teaching–learning-based optimization for global numerical optimization

Teaching–learning-based optimization (TLBO) is one of the latest metaheuristic algorithms being used to solve global optimization problems over continuous search space. Researchers have proposed few variants of TLBO to improve the performance of the basic TLBO algorithm. This paper presents a new variant of TLBO called fuzzy adaptive teaching–learning-based optimization (FATLBO) for numerical global optimization. We propose three new modifications to the basic scheme of TLBO in order to improve its searching capability. These modifications consist, namely of a status monitor, fuzzy adaptive teaching–learning strategies, and a remedial operator. The performance of FATLBO is investigated on four experimental sets comprising complex benchmark functions in various dimensions and compared with well-known optimization methods. Based on the results, we conclude that FATLBO is able to deliver excellence and competitive performance for global optimization.

     

城市区域交通信号迭代学习控制策略

城市交通流具有复杂的非线性动态特性, 在交通控制中难以对其进行精确的数学建模; 同时, 以天为周期, 宏观交通流又呈现出明显的周期性特征. 鉴于此, 提出一种基于迭代学习的城市区域交通信号控制策略, 通过对交通信号的迭代控制, 使路段的平均占有率收敛于期望占有率, 从而使绿灯时间得到充分利用并防止交通拥堵的发生, 保证了交通流在路网中的高效平稳运行. 严格的理论推导证明了该方法的收敛性, 仿真结果验证了该方法的有效性.

     

Reinforcement learning-based dynamic obstacle avoidance and integration of path planning

Deep reinforcement learning has the advantage of being able to encode fairly complex behaviors by collecting and learning empirical information. In the current study, we have proposed a framework for reinforcement learning in decentralized collision avoidance where each agent independently makes its decision without communication with others. In an environment exposed to various kinds of dynamic obstacles with irregular movements, mobile robot agents could learn how to avoid obstacles and reach a target point efficiently. Moreover, a path planner was integrated with the reinforcement learning-based obstacle avoidance to solve the problem of not finding a path in a specific situation, thereby imposing path efficiency. The robots were trained about the policy of obstacle avoidance in environments where dynamic characteristics were considered with soft actor critic algorithm. The trained policy was implemented in the robot operating system (ROS), tested in virtual and real environments for the differential drive wheel robot to prove the effectiveness of the proposed method. Videos are available at https://youtu.be/xxzoh1XbAl0.

     

Traffic information interpolation method based on traffic flow emergence using swarm intelligence

Traffic congestion has become one of the most pressing social problems in today’s society, and research into appropriate traffic signal control is actively underway. At present, most traffic signal control methods define traffic signal parameters on the basis of traffic information such as the number of passing vehicles. Installing sensors at a vast number of intersections is necessary for more precise and real-time adaptive control, but this is unrealistic from the viewpoint of cost. As an alternative, we propose a swarm intelligence-based methodology that creates routes with a similar traffic volume using the traffic information from intersections already equipped with sensors and interpolates this information in the intersections without sensors in real time. Our simulation results show that the proposed methodology can effectively create similar traffic routes for main traffic flows with high traffic volumes. The results also show that it has an excellent interpolation performance for heavy traffic flows and can adapt and interpolate to situations where traffic flow changes suddenly. Moreover, the interpolation results are highly accurate at a road link where traffic flows confluence. We also developed an interpolation algorithm that is adaptable to traffic patterns with confluence traffic flows. Experiments were conducted with a simulation of merging traffic flows and the proposed method showed good results.