Dyna-Q-based vector direction for path planning problem of autonomous mobile robots in unknown environments

Reinforcement learning (RL) is a popular method for solving the path planning problem of autonomous mobile robots in unknown environments. However, the primary difficulty faced by learning robots using the RL method is that they learn too slowly in obstacle-dense environments. To more efficiently solve the path planning problem of autonomous mobile robots in such environments, this paper presents a novel approach in which the robot’s learning process is divided into two phases. The first one is to accelerate the learning process for obtaining an optimal policy by developing the well-known Dyna-Q algorithm that trains the robot in learning actions for avoiding obstacles when following the vector direction. In this phase, the robot’s position is represented as a uniform grid. At each time step, the robot performs an action to move to one of its eight adjacent cells, so the path obtained from the optimal policy may be longer than the true shortest path. The second one is to train the robot in learning a collision-free smooth path for decreasing the number of the heading changes of the robot. The simulation results show that the proposed approach is efficient for the path planning problem of autonomous mobile robots in unknown environments with dense obstacles.     

Obstacle avoidance of redundant manipulators using neural networks based reinforcement learning

This paper proposes a new approach for solving the problem of obstacle avoidance during manipulation tasks performed by redundant manipulators. The developed solution is based on a double neural network that uses Q-learning reinforcement technique. Q-learning has been applied in robotics for attaining obstacle free navigation or computing path planning problems. Most studies solve inverse kinematics and obstacle avoidance problems using variations of the classical Jacobian matrix approach, or by minimizing redundancy resolution of manipulators operating in known environments. Researchers who tried to use neural networks for solving inverse kinematics often dealt with only one obstacle present in the working field. This paper focuses on calculating inverse kinematics and obstacle avoidance for complex unknown environments, with multiple obstacles in the working field. Q-learning is used together with neural networks in order to plan and execute arm movements at each time instant. The algorithm developed for general redundant kinematic link chains has been tested on the particular case of PowerCube manipulator. Before implementing the solution on the real robot, the simulation was integrated in an immersive virtual environment for better movement analysis and safer testing. The study results show that the proposed approach has a good average speed and a satisfying target reaching success rate.     

基于深度强化学习的移动机器人动态路径规划算法

为了在复杂舞台环境下使用移动机器人实现物品搬运或者载人演出,提出了一种基于深度强化学习的动态路径规划算法。首先通过构建全局地图获取移动机器人周围的障碍物信息,将演员和舞台道具分别分类成动态障碍物和静态障碍物。然后建立局部地图,通过LSTM网络编码动态障碍物信息,使用社会注意力机制计算每个动态障碍物的重要性来实现更好的避障效果。通过构建新的奖励函数来实现对动静态障碍物的不同躲避情况。最后通过模仿学习和优先级经验回放技术来提高网络的收敛速度,从而实现在舞台复杂环境下的移动机器人的动态路径规划。实验结果表明,该网络的收敛速度明显提高,在不同障碍物环境下都能够表现出好的动态避障效果。     

Guided Autowave Pulse Coupled Neural Network (GAPCNN) based real time path planning and an obstacle avoidance scheme for mobile robots

Real time path planning for mobile robots requires fast convergence to optimal paths. Most rapid collision free path planning algorithms do not guarantee the optimality of the path. In this paper we present a Guided Autowave Pulse Coupled Neural Network (GAPCNN) approach for mobile robot path planning. The proposed model is a novel approach that improves upon the recently presented Modified PCNN (MPCNN) by introducing directional autowave control and accelerated firing of neurons based on a dynamic thresholding technique. Simulation studies and experimental results in both static as well as dynamic environments confirm GAPCNN to be a robust and time efficient path planning scheme for finding optimal paths.     

基于自适应动态窗口改进细菌算法与移动机器人路径规划

针对移动机器人在复杂环境下的路径规划问题,提出一种新的自适应动态窗口改进细菌算法,并将新算法应用于移动机器人路径规划。改进细菌算法继承了细菌算法与动态窗口算法（dynamic window algorithm, DWA）在避障时的优点,能较好实现复杂环境中移动机器人静态和动态避障。该改进算法主要分三步完成移动机器人路径规划。首先,利用改进细菌趋化算法在静态环境中得到初始参考规划路径。接着,基于参考路径,机器人通过自身携带的传感器感知动态障碍物进行动态避障并利用自适应DWA完成局部动态避障路径规划。最后,根据移动机器人局部动态避障完成情况选择算法执行步骤,如果移动机器人能达到最终目标点,结束该算法,否则移动机器人再重回初始路径,直至到达最终目标点。仿真比较实验证明,改进算法无论在收敛速度还是路径规划精确度方面都有明显提升。     

Path planning approach in unknown environment

This paper presents a new algorithm of path planning for mobile robots, which utilises the characteristics of the obstacle border and fuzzy logical reasoning. The environment topology or working space is described by the time-variable grid method that can be further described by the moving obstacles and the variation of path safety. Based on the algorithm, a new path planning approach for mobile robots in an unknown environment has been developed. The path planning approach can let a mobile robot find a safe path from the current position to the goal based on a sensor system. The two types of machine learning: advancing learning and exploitation learning or trial learning are explored, and both are applied to the learning of mobile robot path planning algorithm. Comparison with A* path planning approach and various simulation results are given to demonstrate the efficiency of the algorithm. This path planning approach can also be applied to computer games.     

A Bioinspired Neural Network for Real-Time Concurrent Map Building and Complete Coverage Robot Navigation in Unknown Environments

Complete coverage navigation (CCN) requires a special type of robot path planning, where the robots should pass every part of the workspace. CCN is an essential issue for cleaning robots and many other robotic applications. When robots work in unknown environments, map building is required for the robots to effectively cover the complete workspace. Real-time concurrent map building and complete coverage robot navigation are desirable for efficient performance in many applications. In this paper, a novel neural-dynamics-based approach is proposed for real-time map building and CCN of autoxnomous mobile robots in a completely unknown environment. The proposed model is compared with a triangular-cell-map-based complete coverage path planning method (Oh , 2004) that combines distance transform path planning, wall-following algorithm, and template-based technique. The proposed method does not need any templates, even in unknown environments. A local map composed of square or rectangular cells is created through the neural dynamics during the CCN with limited sensory information. From the measured sensory information, a map of the robot's immediate limited surroundings is dynamically built for the robot navigation. In addition, square and rectangular cell map representations are proposed for real-time map building and CCN. Comparison studies of the proposed approach with the triangular-cell-map-based complete coverage path planning approach show that the proposed method is capable of planning more reasonable and shorter collision-free complete coverage paths in unknown environments.      

Sensor-based path planning for nonholonomic mobile robots subject to dynamic constraints

It is generally not easy to achieve smooth path planning in an unknown environment for nonholonomic mobile robots, which are subject to various robot constraints. In this paper, a hybrid approach is proposed for smooth path planning with global convergence for differential drive nonholonomic robots. We first investigate the use of a polar polynomial curve (PPC) to produce a path changing continuously in curvature and satisfying dynamic constraints. In order to achieve path generation in real-time, a computationally effective method is proposed for collision test of the complex curve. Then, a hybrid path planning approach is presented to guide the robot to move forward along the boundary of an obstacle of arbitrary shape, by generating a proper “Instant Goal” (and a series of deliberate motions through PPC curve based path generation) and planning reactively when needed using a fuzzy controller for wall following. The choice of an Instant Goal is limited to the set of candidates that are practically reachable by the robot and that enable the robot to continue following the obstacle. The effectiveness of the proposed approach is verified by simulation experiments.     

异构多目标差分-动态窗口算法 及其在移动机器人中的应用

为了实现在多移动机器人和多窄通道的复杂动态环境中机器人的节能运动规划,提出异构多目标差分-动态窗口法(heterogeneous multi-objective differential evolution-dynamic window algorithm,HMODE-DWA).首先,建立行驶时间、执行器作用力和平滑度的3目标优化模型,设计具有碰撞约束的异构多目标差分进化算法来获得3个目标函数的最优解,进而在已知的静态环境中获得帕累托前沿,利用平均隶属度函数获得起点与终点间最优的全局路径;其次,定义基于环境缓冲区域的模糊动态窗口法使机器人完成动态复杂环境中避障,利用所提出的HMODE-DWA算法动态避障的同时实现节能规划.仿真和实验结果表明,所提出的混合路径规划控制策略能够有效降低移动机器人动态避障过程中的能耗.     

基于深度Q网络和人工势场的移动机器人路径规划研究

随着移动机器人在各个领域的研究与发展,人们对移动机器人路径规划的能力提出了更高的要求;为了解决传统的深度Q网络算法在未知环境下,应用于自主移动机器人路径规划时存在的收敛速度慢、训练前期产生较大迭代空间、迭代的次数多等问题,在传统DQN算法初始化Q值时,加入人工势场法的引力势场来协助初始化环境先验信息,进而可以引导移动机器人向目标点运动,来减少算法在最初几轮探索中形成的大批无效迭代,进而减少迭代次数,加快收敛速度;在栅格地图环境中应用pytorch框架验证加入初始引力势场的改进DQN算法路径规划效果;仿真实验结果表明,改进算法能在产生较小的迭代空间且较少的迭代次数后,快速有效地规划出一条从起点到目标点的最优路径。     

基于增强学习的摄像机网络节点动态选择方法

摄像机节点动态选择问题是摄像机网络应用中的一个难点.提出了一种基于增强学习的节点动态选择方法.采用视觉信息评分作为单步回报设计了节点选择策略的Q-学习算法,为了加速算法收敛速度,利用摄像机空间拓扑关系初始化Q值表,并基于Gibbs分布进行非贪心尝试.从目标可见性、朝向、清晰度和切换次数设计视觉评价函数反映视频信息丰富程度和视觉舒适度.实验结果表明,该节点动态选择方法能够有效地反映视频中的目标状态信息,选择结果切换平滑,满足实际应用需要.     

Incremental Q-learning strategy for adaptive PID control of mobile robots

Expert and intelligent systems are being developed to control many technological systems including mobile robots. However, the PID (Proportional-Integral-Derivative) controller is a fast low-level control strategy widely used in many control engineering tasks. Classic control theory has contributed with different tuning methods to obtain the gains of PID controllers for specific operation conditions. Nevertheless, when the system is not fully known and the operative conditions are variable and not previously known, classical techniques are not entirely suitable for the PID tuning. To overcome these drawbacks many adaptive approaches have been arisen, mainly from the field of artificial intelligent. In this work, we propose an incremental Q-learning strategy for adaptive PID control. In order to improve the learning efficiency we define a temporal memory into the learning process. While the memory remains invariant, a non-uniform specialization process is carried out generating new limited subspaces of learning. An implementation on a real mobile robot demonstrates the applicability of the proposed approach for a real-time simultaneous tuning of multiples adaptive PID controllers for a real system operating under variable conditions in a real environment.     

VARIABLE PATROL PLANNING OF MULTI-ROBOT SYSTEMS BY A COOPERATIVE AUCTION SYSTEM

A cooperative auction system (CAS) is proposed to solve the large-scale multi-robot patrol planning problem. Each robot picks its own patrol points via the cooperative auction system and the system continuously re-auctions, based on the team work performance. The proposed method not only works in static environments but also considers variable path planning when the number of mobile robots increases or decreases during patrol. From the results of the simulation, the proposed approach demonstrates decreased time complexity, a lower routing path cost, improved balance of workload among robots, and the potential to scale to a large number of robots and is adaptive to environmental perturbations when the number of robots changes during patrol.     

未知环境下移动机器人遍历路径规划

为提高未知环境下移动机器人遍历路径规划的效率,提出了一种可动态调节启发式规则的滚动路径规划算法.该算法以生物激励神经网络为环境模型,通过在线识别环境信息特征,动态调用静态搜索算法和环绕障碍搜索算法,有效减少了路径的转弯次数.引入虚拟障碍和直接填充算法,解决了u型障碍区域的连续遍历问题.最后通过仿真实验表明了该方法在未知复杂环境下的有效性.     

New robot navigation algorithm for arbitrary unknown dynamic environments based on future prediction and priority behavior

This study focuses on existing drawbacks and inefficiencies of the available path planning approaches within unknown dynamic environments. The drawbacks are the inability to plan under uncertain dynamic environments, non-optimality, failure in crowded complex situations, and difficulty in predicting the velocity vector of obstacles. This study aims (1) to develop a new predictive method to avoid static and dynamic obstacles in planning the path of a mobile robot in unknown dynamic environments in which the obstacles are moving and their speed profiles are not pre-identified, to find a safe path and to react rapidly and (2) to integrate a decision-making process with the predictive behavior of the velocity vector of obstacles by using the sensory system information of the robot. Information on the locations, shapes, and velocities of static and dynamic obstacles is presumed to be unavailable. Such information is determined online using rangefinder sensors. Thus, the robot recognizes free directions that lead it toward its destination and keep it safe and prevent collision with obstacles. Extensive simulations confirm the efficiency of the suggested approach and its success in handling complex and extremely dynamic environments that contain various obstacle shapes. Findings indicate that the proposed method exhibits attractive features, such as high optimality, high stability, low running time, and zero failure rates. The failure rate is zero for all test problems. The average path length for all test environments is 16.51 with a standard deviation of 0.49, which provides an average optimality rate of 89.79%. The average running time is 4.74 s (the standard deviation is 0.26).     

不确定动态环境下移动机器人的完全遍历路径规划

基于生物激励神经网络、滚动窗口和启发式搜索,提出了一种新的完全遍历路径规划方法．该方法用Grossberg的生物神经网络实现移动机器人的局部环境建模,将滚动窗口的概念引入到局部路径规划,由启发式算法决定滚动窗口内的局域路径规划目标．该方法能在不确定动态环境中有效地实现机器人自主避障的完全遍历路径规划．仿真研究证明了该方法的可用性和有效性．     

未知环境下移动机器人实时路径规划

针对在未知环境下实现移动机器人实时的路径规划问题,提出了一种将快速扩展随机树（RRT）算法与视野域自适应的滚动窗口相结合的路径规划算法。该方法实时获取滚动窗口内的局部环境信息,根据环境的变化,滚动窗口视野域进行自适应调整,通过分析滚动窗口内传感器获取的信息,结合改进后的RRT算法筛选出可行的路径,控制移动机器人到达子目标点,在此过程中动态监测规划好的路径,确保路径合理,并重复上述过程,直至到达目标区域。实验对比分析表明,该方法能实时并有效实现未知环境下移动机器人的路径规划。     

A global motion planner that learns from experience for autonomous mobile robots

A new technique for enhancing global path planning for mobile robots working in partially known as indoor environments is presented in this paper. The method is based on a graph approach that adapts the cost of the paths by incorporating travelling time from real experiences. The approach uses periodical measurements of time and position reached by the robot while moving to the goal to modify the costs of the branches. Consequently, the search of a feasible path from a static global map in dynamic environments is more realistic than employing a distance metric. Our approach has been tested in simulation as well on an autonomous robot. Results from both simulation and real experiences are discussed.     

Motion planning of multiple mobile robots by a combination of learned visibility graphs and virtual impedance

A motion planning algorithm for multiple mobile robots is proposed in this paper. A hierarchical architecture with two layers 'learned visibility graph layer (upper layer)' and 'virtual impedance layer (lower layer)' (one of the potential field planning method) is presented. This system has the following characteristics: (1) is applicable to unknown dynamic environments, (2) is applicable to distributed multiple robot systems and (3) is capable of adequate path generation and motion. At the upper layer, efficient exploration of environments makes it possible to generate sub-shortest paths that avoid static obstacles. At the lower layer, on-line avoidance can be made with virtual impedance against moving obstacles such as other robots. Simulation results show the validity of the proposed method.     

基于改进SAC算法的移动机器人路径规划

为解决SAC算法在移动机器人局部路径规划中训练时间长、收敛速度慢等问题,通过引入优先级经验回放（PER）技术,提出了PER-SAC算法。首先从等概率从经验池中随机抽取样本变为按优先级抽取,使网络优先训练误差较大的样本,从而提高了机器人训练过程的收敛速度和稳定性;其次优化时序差分（TD）误差的计算,以降低训练偏差;然后利用迁移学习,使机器人从简单环境到复杂环境逐步训练,从而提高训练速度;另外,设计了改进的奖励函数,增加机器人的内在奖励,从而解决了环境奖励稀疏的问题;最后在ROS平台上进行仿真测试。仿真结果表明,在不同的障碍物环境中,PER-SAC算法均比原始算法收敛速度更快、规划的路径长度更短,并且PER-SAC算法能够减少训练时间,在路径规划性能上明显优于原始算法。