State-chain sequential feedback reinforcement learning for path planning of autonomous mobile robots

This paper deals with a new approach based on Q-learning for solving the problem of mobile robot path planning in complex unknown static environments.As a computational approach to learning through interaction with the environment,reinforcement learning algorithms have been widely used for intelligent robot control,especially in the field of autonomous mobile robots.However,the learning process is slow and cumbersome.For practical applications,rapid rates of convergence are required.Aiming at the problem of slow convergence and long learning time for Q-learning based mobile robot path planning,a state-chain sequential feedback Q-learning algorithm is proposed for quickly searching for the optimal path of mobile robots in complex unknown static environments.The state chain is built during the searching process.After one action is chosen and the reward is received,the Q-values of the state-action pairs on the previously built state chain are sequentially updated with one-step Q-learning.With the increasing number of Q-values updated after one action,the number of actual steps for convergence decreases and thus,the learning time decreases,where a step is a state transition.Extensive simulations validate the efficiency of the newly proposed approach for mobile robot path planning in complex environments.The results show that the new approach has a high convergence speed and that the robot can find the collision-free optimal path in complex unknown static environments with much shorter time,compared with the one-step Q-learning algorithm and the Q(λ)-learning algorithm.     

融合动态障碍物运动信息的路径规划算法

传统的路径规划算法只能在障碍物不发生位置变化的环境中计算最优路径。但是随着机器人在商场、医院、银行等动态环境下的普及,传统的路径规划算法容易与动态障碍物发生碰撞等危险。因此,关于随机动态障碍物条件下的机器人路径规划算法需要得到进一步改善。为了解决在动态环境下的机器人路径规划问题,提出了一种融合机器人与障碍物运动信息的改进动态窗口法来解决机器人在动态环境下的局部路径规划问题,并且与优化A*算法相结合来实现全局最优路径规划。主要内容体现为：在全局路径规划上,采用优化A*算法求解最优路径。在局部路径规划上,以动态障碍物的速度作为先验信息,通过对传统动态窗口法的评价函数进行扩展,实现机器人在动态环境下的自主智能避障。实验证明,该算法可以实现基于全局最优路径的实时动态避障,具体表现为可以在不干涉动态障碍物的条件下减少碰撞风险、做出智能避障且路径更加平滑、长度更短、行驶速度更快。     

A hierarchical reinforcement learning approach for optimal path tracking of wheeled mobile robots

Robust motion control is fundamental to autonomous mobile robots. In the past few years, reinforcement learning (RL) has attracted considerable attention in the feedback control of wheeled mobile robot. However, it is still difficult for RL to solve problems with large or continuous state spaces, which is common in robotics. To improve the generalization ability of RL, this paper presents a novel hierarchical RL approach for optimal path tracking of wheeled mobile robots. In the proposed approach, a graph Laplacian-based hierarchical approximate policy iteration (GHAPI) algorithm is developed, in which the basis functions are constructed automatically using the graph Laplacian operator. In GHAPI, the state space of an Markov decision process is divided into several subspaces and approximate policy iteration is carried out on each subspace. Then, a near-optimal path-tracking control strategy can be obtained by GHAPI combined with proportional-derivative (PD) control. The performance of the proposed approach is evaluated by using a P3-AT wheeled mobile robot. It is demonstrated that the GHAPI-based PD control can obtain better near-optimal control policies than previous approaches.     

Path planning approach in unknown environment

This paper presents a new algorithm of path planning for mobile robots, which utilises the characteristics of the obstacle border and fuzzy logical reasoning. The environment topology or working space is described by the time-variable grid method that can be further described by the moving obstacles and the variation of path safety. Based on the algorithm, a new path planning approach for mobile robots in an unknown environment has been developed. The path planning approach can let a mobile robot find a safe path from the current position to the goal based on a sensor system. The two types of machine learning: advancing learning and exploitation learning or trial learning are explored, and both are applied to the learning of mobile robot path planning algorithm. Comparison with A* path planning approach and various simulation results are given to demonstrate the efficiency of the algorithm. This path planning approach can also be applied to computer games.     

Planning smooth paths for mobile robots in an unknown environment

This paper addresses the path planning problem for autonomous mobile robots operating in an unknown environment with obstacles. Paths are formed based on third-order Bezier splines and, then, are corrected on the move as a robot detects obstacles with its onboard sensors. During this correction, the initial path between two reference points is divided into two segments (described by Bezier splines) in such a way as to allow the robot to move at a safe distance from a detected obstacle along a smooth resultant trajectory. In this case, the use of smooth paths ensures a high levels of accuracy and velocity of mobile robots during their operation.     

异构多目标差分-动态窗口算法 及其在移动机器人中的应用

为了实现在多移动机器人和多窄通道的复杂动态环境中机器人的节能运动规划,提出异构多目标差分-动态窗口法(heterogeneous multi-objective differential evolution-dynamic window algorithm,HMODE-DWA).首先,建立行驶时间、执行器作用力和平滑度的3目标优化模型,设计具有碰撞约束的异构多目标差分进化算法来获得3个目标函数的最优解,进而在已知的静态环境中获得帕累托前沿,利用平均隶属度函数获得起点与终点间最优的全局路径;其次,定义基于环境缓冲区域的模糊动态窗口法使机器人完成动态复杂环境中避障,利用所提出的HMODE-DWA算法动态避障的同时实现节能规划.仿真和实验结果表明,所提出的混合路径规划控制策略能够有效降低移动机器人动态避障过程中的能耗.     

基于自适应动态窗口改进细菌算法与移动机器人路径规划

针对移动机器人在复杂环境下的路径规划问题,提出一种新的自适应动态窗口改进细菌算法,并将新算法应用于移动机器人路径规划。改进细菌算法继承了细菌算法与动态窗口算法（dynamic window algorithm, DWA）在避障时的优点,能较好实现复杂环境中移动机器人静态和动态避障。该改进算法主要分三步完成移动机器人路径规划。首先,利用改进细菌趋化算法在静态环境中得到初始参考规划路径。接着,基于参考路径,机器人通过自身携带的传感器感知动态障碍物进行动态避障并利用自适应DWA完成局部动态避障路径规划。最后,根据移动机器人局部动态避障完成情况选择算法执行步骤,如果移动机器人能达到最终目标点,结束该算法,否则移动机器人再重回初始路径,直至到达最终目标点。仿真比较实验证明,改进算法无论在收敛速度还是路径规划精确度方面都有明显提升。     

一种车型机器人路径规划方法

在自主移动机器人的许多应用中,路径规划技术顺序地设置一套分散的路径点来引导机器人以最短的时间从起始位置到达目标点。针对移动机器人路径规划问题,提出了一种非完整型机器人路径规划技术,该技术采用基本原子操纵方法来解决车型机器人路径规划问题,并采用平滑路径规划方法来产生更多的连续路径用以解决基本原子操纵技术在做路径规划时具有很不连续的缺点从而为机器人获得最优路径。仿真结果证明了该方法的有效性和实用性。     

双轮驱动移动机器人的学习控制器设计方法*

提出一种基于增强学习的双轮驱动移动机器人路径跟随控制方法,通过将机器人运动控制器的优化设计问题建模为Markov决策过程,采用基于核的最小二乘策略迭代算法(KLSPI)实现控制器参数的自学习优化。与传统表格型和基于神经网络的增强学习方法不同,KLSPI算法在策略评价中应用核方法进行特征选择和值函数逼近,从而提高了泛化性能和学习效率。仿真结果表明,该方法通过较少次数的迭代就可以获得优化的路径跟随控制策略,有利于在实际应用中的推广。     

自主机器人的强化学习研究进展

虽然基于行为控制的自主机器人具有较高的鲁棒性,但其对于动态环境缺乏必要的自 适应能力．强化学习方法使机器人可以通过学习来完成任务,而无需设计者完全预先规定机 器人的所有动作,它是将动态规划和监督学习结合的基础上发展起来的一种新颖的学习方法 ,它通过机器人与环境的试错交互,利用来自成功和失败经验的奖励和惩罚信号不断改进机 器人的性能,从而达到目标,并容许滞后评价．由于其解决复杂问题的突出能力,强化学习 已成为一种非常有前途的机器人学习方法．本文系统论述了强化学习方法在自主机器人中的 研究现状,指出了存在的问题,分析了几种问题解决途径,展望了未来发展趋势．     

A PATH PLANNING METHOD BASED ON CELLULAR AUTOMATA FOR COOPERATIVE ROBOTS

A Cellular Automaton-based technique suitable for solving the path planning problem in a distributed robot team is outlined. Real-time path planning is a challenging task that has many applications in the fields of artificial intelligence, moving robots, virtual reality, and agent behavior simulation. The problem refers to finding a collision-free path for autonomous robots between two specified positions in a configuration area. The complexity of the problem increases in systems of multiple robots. More specifically, some distance should be covered by each robot in an unknown environment, avoiding obstacles found on its route to the destination. On the other hand, all robots must adjust their actions in order to keep their initial team formation immutable. Two different formations were tested in order to study the efficiency and the flexibility of the proposed method. Using different formations, the proposed technique could find applications to image processing tasks, swarm intelligence, etc. Furthermore, the presented Cellular Automaton (CA) method was implemented and tested in a real system using three autonomous mobile minirobots called E-pucks. Experimental results indicate that accurate collision-free paths could be created with low computational cost. Additionally, cooperation tasks could be achieved using minimal hardware resources, even in systems with low-cost robots.     

人工免疫算法在机器人路径规划中的应用

人工免疫算法是一种新兴的优化方法,在计算、控制等各方面都已得到应用.将免疫算法应用于移动机器人路径规划,提出一种任意多边形障碍物复杂布局环境下的机器人路径规划的人工免疫算法,仿真证明该算法可以准确地找到全局最优路径,而且能够适应各种复杂的环境.     

移动机器人路径规划综述

移动机器人是目前科学技术发展最活跃的领域之一,在工业、农业、医疗等行业广泛应用,还在城市安全、国防和空间探测领域得到更广的应用。要实现机器人在未知环境下自主作业,具备实时、自主、识别高风险区域和风险管理的能力,路径规划是一个重要环节,规划水平的高低,在一定程度上标志着机器人的智能水平,因此研究机器人路径规划对提高机器人的智能化水平、加快工程化应用具有重要的意义。文章重点分别从全局路径规划和局部路径规划角度对机器人路径规划的研究方法进行了分析与总结,同时分析研究了基于仿生学的智能算法的遗传算法、蚁群算法、粒子群算法,最后展望了移动机器人的未来发展趋势。     

Neural networks based reinforcement learning for mobile robots obstacle avoidance

This study proposes a new approach for solving the problem of autonomous movement of robots in environments that contain both static and dynamic obstacles. The purpose of this research is to provide mobile robots a collision-free trajectory within an uncertain workspace which contains both stationary and moving entities. The developed solution uses Q-learning and a neural network planner to solve path planning problems. The algorithm presented proves to be effective in navigation scenarios where global information is available. The speed of the robot can be set prior to the computation of the trajectory, which provides a great advantage in time-constrained applications. The solution is deployed in both Virtual Reality (VR) for easier visualization and safer testing activities, and on a real mobile robot for experimental validation. The algorithm is compared with Powerbot's ARNL proprietary navigation algorithm. Results show that the proposed solution has a good conversion rate computed at a satisfying speed.     

基于连续动态运动基元的移动机器人路径规划

针对二维动态场景下的移动机器人路径规划问题，提出了一种新颖的路径规划方法——连续动态运动基元（continuous dynamic movement primitives， CDMPs）.该方法将传统的单一动态运动基元推广到连续动态运动基元，通过对演示运动轨迹的学习，获得各运动基元的权重序列，利用相位变量的更新，实现对未知动态目标的追踪.该方法克服了移动机器人对环境模型的依赖，解决了动态场景下追踪运动目标和躲避动态障碍物的路径规划问题.最后通过一系列仿真实验，验证了算法的可行性.仿真实验结果表明，对于动态场景下移动机器人路径规划问题， CDMPs算法比传统的DMPs方法在连续性能和规划效率上具有更好的表现.     

A Bioinspired Neural Network for Real-Time Concurrent Map Building and Complete Coverage Robot Navigation in Unknown Environments

Complete coverage navigation (CCN) requires a special type of robot path planning, where the robots should pass every part of the workspace. CCN is an essential issue for cleaning robots and many other robotic applications. When robots work in unknown environments, map building is required for the robots to effectively cover the complete workspace. Real-time concurrent map building and complete coverage robot navigation are desirable for efficient performance in many applications. In this paper, a novel neural-dynamics-based approach is proposed for real-time map building and CCN of autoxnomous mobile robots in a completely unknown environment. The proposed model is compared with a triangular-cell-map-based complete coverage path planning method (Oh , 2004) that combines distance transform path planning, wall-following algorithm, and template-based technique. The proposed method does not need any templates, even in unknown environments. A local map composed of square or rectangular cells is created through the neural dynamics during the CCN with limited sensory information. From the measured sensory information, a map of the robot's immediate limited surroundings is dynamically built for the robot navigation. In addition, square and rectangular cell map representations are proposed for real-time map building and CCN. Comparison studies of the proposed approach with the triangular-cell-map-based complete coverage path planning approach show that the proposed method is capable of planning more reasonable and shorter collision-free complete coverage paths in unknown environments.      

移动机器人路径规划算法的研究综述

路径规划是移动机器人的热门研究之一,是实现机器人自主导航的关键技术。针对移动机器人路径规划的算法进行研究,以了解不同条件下路径规划算法的发展与应用,系统性地总结了路径规划的研究现状和发展。针对移动机器人路径规划的特点,将其划分为智能搜索算法、基于人工智能算法、基于几何模型算法和用于局部避障算法。基于上述分类,介绍了近年来具有代表性的研究成果,重点分析各类规划算法的优缺点,对移动机器人路径规划的未来发展趋势进行展望,为移动机器人路径规划研究提供一定的思路。     

Dynamic path planning of mobile robots with improved genetic algorithm

In this study, a new mutation operator is proposed for the genetic algorithm (GA) and applied to the path planning problem of mobile robots in dynamic environments. Path planning for a mobile robot finds a feasible path from a starting node to a target node in an environment with obstacles. GA has been widely used to generate an optimal path by taking advantage of its strong optimization ability. While conventional random mutation operator in simple GA or some other improved mutation operators can cause infeasible paths, the proposed mutation operator does not and avoids premature convergence. In order to demonstrate the success of the proposed method, it is applied to two different dynamic environments and compared with previous improved GA studies in the literature. A GA with the proposed mutation operator finds the optimal path far too many times and converges more rapidly than the other methods do.     

A dynamic priority based path planning for cooperation of multiple mobile robots in formation forming

Robots that work in a proper formation show several advantages compared to a single complex robot, such as a reduced cost, robustness, efficiency and improved performance. Existing researches focused on the method of keeping the formation shape during the motion, but usually neglect collision constraints or assume a simplified model of obstacles. This paper investigates the path planning of forming a target robot formation in a clutter environment containing unknown obstacles. The contribution lies in proposing an efficient path planner for the multiple mobile robots to achieve their goals through the clutter environment and developing a dynamic priority strategy for cooperation of robots in forming the target formation. A multirobot system is set up to verify the proposed method of robot path planning. Simulations and experiments results demonstrate that the proposed method can successfully address the collision avoidance problem as well as the formation forming problem.     

Real-Time Robot Path Planning Based on a Modified Pulse-Coupled Neural Network Model

This paper presents a modified pulse-coupled neural network (MPCNN) model for real-time collision-free path planning of mobile robots in nonstationary environments. The proposed neural network for robots is topologically organized with only local lateral connections among neurons. It works in dynamic environments and requires no prior knowledge of target or barrier movements. The target neuron fires first, and then the firing event spreads out, through the lateral connections among the neurons, like the propagation of a wave. Obstacles have no connections to their neighbors. Each neuron records its parent, that is, the neighbor that caused it to fire. The real-time optimal path is then the sequence of parents from the robot to the target. In a static case where the barriers and targets are stationary, this paper proves that the generated wave in the network spreads outward with travel times proportional to the linking strength among neurons. Thus, the generated path is always the global shortest path from the robot to the target. In addition, each neuron in the proposed model can propagate a firing event to its neighboring neuron without any comparing computations. The proposed model is applied to generate collision-free paths for a mobile robot to solve a maze-type problem, to circumvent concave U-shaped obstacles, and to track a moving target in an environment with varying obstacles. The effectiveness and efficiency of the proposed approach is demonstrated through simulation and comparison studies.