Neural networks based reinforcement learning for mobile robots obstacle avoidance

This study proposes a new approach for solving the problem of autonomous movement of robots in environments that contain both static and dynamic obstacles. The purpose of this research is to provide mobile robots a collision-free trajectory within an uncertain workspace which contains both stationary and moving entities. The developed solution uses Q-learning and a neural network planner to solve path planning problems. The algorithm presented proves to be effective in navigation scenarios where global information is available. The speed of the robot can be set prior to the computation of the trajectory, which provides a great advantage in time-constrained applications. The solution is deployed in both Virtual Reality (VR) for easier visualization and safer testing activities, and on a real mobile robot for experimental validation. The algorithm is compared with Powerbot's ARNL proprietary navigation algorithm. Results show that the proposed solution has a good conversion rate computed at a satisfying speed.     

基于支持向量机和Q学习的移动机器人导航

基于神经网络的连续状态空间Q学习已应用在机器人导航领域。针对神经网络易陷入局部极小,提出了将支持向量机与Q学习相结合的移动机器人导航方法。首先以研制的CASIA-I移动机器人和它的工作环境为实验平台,确定出Q学习的回报函数;然后利用支持向量机对Q学习的状态——动作对的Q值进行在线估计,同时,为了提高估计速度,引入滚动时间窗机制;最后对所提方法进行了实验,实验结果表明所提方法能够使机器人无碰撞的到达目的地。     

State-chain sequential feedback reinforcement learning for path planning of autonomous mobile robots

This paper deals with a new approach based on Q-learning for solving the problem of mobile robot path planning in complex unknown static environments.As a computational approach to learning through interaction with the environment,reinforcement learning algorithms have been widely used for intelligent robot control,especially in the field of autonomous mobile robots.However,the learning process is slow and cumbersome.For practical applications,rapid rates of convergence are required.Aiming at the problem of slow convergence and long learning time for Q-learning based mobile robot path planning,a state-chain sequential feedback Q-learning algorithm is proposed for quickly searching for the optimal path of mobile robots in complex unknown static environments.The state chain is built during the searching process.After one action is chosen and the reward is received,the Q-values of the state-action pairs on the previously built state chain are sequentially updated with one-step Q-learning.With the increasing number of Q-values updated after one action,the number of actual steps for convergence decreases and thus,the learning time decreases,where a step is a state transition.Extensive simulations validate the efficiency of the newly proposed approach for mobile robot path planning in complex environments.The results show that the new approach has a high convergence speed and that the robot can find the collision-free optimal path in complex unknown static environments with much shorter time,compared with the one-step Q-learning algorithm and the Q(λ)-learning algorithm.     

基于深度Q学习的移动机器人路径规划

针对传统Q-learning算法在复杂环境下移动机器人路径规划问题中容易产生维数灾难的问题,提出一种改进方法。该方法将深度学习融于Q-learming框架中,以网络输出代替Q值表,解决维数灾难问题。通过构建记忆回放矩阵和双层网络结构打断数据相关性,提高算法收敛性。最后,通过栅格法建立仿真环境建模,在不同复杂程度上的地图上进行仿真实验,对比实验验证了传统Q-learming难以在大状态空间下进行路径规划,深度强化学习能够在复杂状态环境下进行良好的路径规划。     

基于行为优先与模糊控制的移动机器人导航

由于动态未知环境下自主移动机器人的导航具有较大困难,为实现自主机器人在动态未知环境下的无碰撞运行,文中将行为优先级控制与模糊逻辑控制相结合,提出4种基本行为控制策略:目标寻找、避障、跟踪和解锁.针对'U'型和'V'型障碍物运行解锁问题,提出了行走路径记忆方法,并通过构建虚拟墙来避免机器人再次走入此类区域.仿真实验表明,所提出的控制策略可有效地运用于复杂和未知环境下自主移动机器人的导航,且具有较好的鲁棒性和适应性.     

Real-time map building and navigation for autonomous robots inunknown environments

An algorithmic solution method is presented for the problem of autonomous robot motion in completely unknown environments. Our approach is based on the alternate execution of two fundamental processes: map building and navigation. In the former, range measures are collected through the robot exteroceptive sensors and processed in order to build a local representation of the surrounding area. This representation is then integrated in the global map so far reconstructed by filtering out insufficient or conflicting information. In the navigation phase, an A*-based planner generates a local path from the current robot position to the goal. Such a path is safe inside the explored area and provides a direction for further exploration. The robot follows the path up to the boundary of the explored area, terminating its motion if unexpected obstacles are encountered. The most peculiar aspects of our method are the use of fuzzy logic for the efficient building and modification of the environment map, and the iterative application of A*, a complete planning algorithm which takes full advantage of local information. Experimental results for a NOMAD 200 mobile robot show the real-time performance of the proposed method, both in static and moderately dynamic environments.     

一种改进的未知动态环境下机器人混合路径规划方法

动态未知环境下的机器人路径规划是机器人导航领域的重要课题之一,采用传统的方法求解并不理想。针对这个问题,提出一种改进的机器人混合路径规划方法。首先利用改进的文化基因算法规划出较优的全局路径,指引机器人沿着全局路径行走,然后根据传感器探测到的局部环境信息,利用Morphin算法进行局部路径实时规划,使机器人有效地躲避动态障碍物。仿真实验表明,该算法在未知动态路径规划中具有良好的效果。     

未知环境下机器人路径规划的粒子群优化

研究了全局静态环境未知时机器人的路径规划问题,提出了一种新颖的基于粒子群算法的滚动规划算法。该方法在机器人视野域内产生若干个同心圆进行环境建模,然后利用粒子群优化算法规划出一条导航路径,机器人每前进一步,都由粒子群优化算法重新规划导航路径,因此,机器人前进路径不断动态修改,从而能使机器人沿一条全局优化的路径接近终点。仿真实验结果表明,即使在障碍物非常复杂的地理环境,用该算法也能迅速规划出一条优化路径,且能安全避碰,效果令人满意。     

A Novel Approach for Mobile Robot Navigation with Dynamic Obstacles Avoidance

This paper proposes a new approach for trajectory optimization of a mobile robot in a general dynamic environment. The new method combines the static and dynamic modes of trajectory planning to provide an algorithm that gives fast and optimal solutions for static environments, and generates a new path when an unexpected situation occurs. The particularity of the method is in the representation of the static environment in a judicious way facilitating the path planning and reducing the processing time. Moreover, when an unexpected obstacle blocks the robot trajectory, the method uses the robot sensors to detect the obstacle, finds a best way to circumvent it and then resumes its path toward the desired destination. Experimental results showed the effectiveness of the proposed approach.     

Dyna-Q-based vector direction for path planning problem of autonomous mobile robots in unknown environments

Reinforcement learning (RL) is a popular method for solving the path planning problem of autonomous mobile robots in unknown environments. However, the primary difficulty faced by learning robots using the RL method is that they learn too slowly in obstacle-dense environments. To more efficiently solve the path planning problem of autonomous mobile robots in such environments, this paper presents a novel approach in which the robot’s learning process is divided into two phases. The first one is to accelerate the learning process for obtaining an optimal policy by developing the well-known Dyna-Q algorithm that trains the robot in learning actions for avoiding obstacles when following the vector direction. In this phase, the robot’s position is represented as a uniform grid. At each time step, the robot performs an action to move to one of its eight adjacent cells, so the path obtained from the optimal policy may be longer than the true shortest path. The second one is to train the robot in learning a collision-free smooth path for decreasing the number of the heading changes of the robot. The simulation results show that the proposed approach is efficient for the path planning problem of autonomous mobile robots in unknown environments with dense obstacles.     

Navigation with constraints for an autonomous mobile robot

This paper describes a navigation planning algorithm for a robot capable of autonomous navigation in a structured, partially known and dynamic environment. This algorithm is applied to a discrete workspace composed of a network of places and roads. The environment specification associates temporal constraints with any element of the network, and recharge or relocalisation possibilities with places. A mission specification associates several constraints with each navigation task (energy, time, position uncertainty and distance).

The algorithm computes an optimal path for each navigation task according to the optimization criterion and constraints. We introduce the notion of efficient path applied to a new best first search algorithm solving a multiple constraints problem. The path determination relies on a state representation adapted to deal with environment constraints. We then prove that the complexity chracteristics of our algorithm are similar to those of the A* algorithm.

The planner described in this paper has been implemented on a Spare station for a Robuter mobile platform equipped with ultra-sonic range sensors and an active stereo vision system. It was developed for the MITHRA family of autonomous surveillance robots as part of project EUREKA EU 110.     

引入势场及陷阱搜索的强化学习路径规划算法

移动机器人在复杂环境中移动难以得到较优的路径,基于马尔可夫过程的Q学习（Q-learning）算法能通过试错学习取得较优的路径,但这种方法收敛速度慢,迭代次数多,且试错方式无法应用于真实的环境中。在Q-learning算法中加入引力势场作为初始环境先验信息,在其基础上对环境进行陷阱区域逐层搜索,剔除凹形陷阱区域[Q]值迭代,加快了路径规划的收敛速度。同时取消对障碍物的试错学习,使算法在初始状态就能有效避开障碍物,适用于真实环境中直接学习。利用python及pygame模块建立复杂地图,验证加入初始引力势场和陷阱搜索的改进Q-learning算法路径规划效果。仿真实验表明,改进算法能在较少的迭代次数后,快速有效地到达目标位置,且路径较优。     

越野环境建模与动态路径规划

针对越野环境下移动机器人的导航与控制问题，提出了一种越野环境下环境建模与动态路径规划方法。该方法能够针对地形的数字高程模型，在综合考虑机器人的性能约束和地形特征等因素的基础上有效地实现越野环境的建模；在此基础上的路径规划采用全局信息与局部信息、前期规划结果与当前规划相结合的方法，满足了越野环境下动态路径规划的要求。实验结果表明，该方法能够很好地适应各种复杂的越野环境。     

基于ART2的Q学习算法研究

为了解决Q学习应用于连续状态空间的智能系统所面临的"维数灾难"问题,提出一种基于ART2的Q学习算法.通过引入ART2神经网络,让Q学习Agent针对任务学习一个适当的增量式的状态空间模式聚类,使Agent无需任何先验知识,即可在未知环境中进行行为决策和状态空间模式聚类两层在线学习,通过与环境交互来不断改进控制策略,从而提高学习精度.仿真实验表明,使用ARTQL算法的移动机器人能通过与环境交互学习来不断提高导航性能.     

基于未知环境状态新定义及知识启发的机器人导航Q学习算法

由于强大的自主学习能力, 强化学习方法逐渐成为机器人导航问题的研究热点, 但是复杂的未知环境对算法的运行效率和收敛速度提出了考验。提出一种新的机器人导航Q学习算法, 首先用三个离散的变量来定义环境状态空间, 然后分别设计了两部分奖赏函数, 结合对导航达到目标有利的知识来启发引导机器人的学习过程。实验在Simbad仿真平台上进行, 结果表明本文提出的算法很好地完成了机器人在未知环境中的导航任务, 收敛性能也有其优越性。     

Navigation integration of a mobile robot in dynamic environments

This article presents a design and experimental study of navigation integration of an intelligent mobile robot in dynamic environments. The proposed integration architecture is based on the virtual‐force concept, by which each navigation resource is assumed to exert a virtual force on the robot. The resultant force determines how the robot will move. Reactive behavior and proactive planning can both be handled in a simple and uniform manner using the proposed integration method. A real‐time motion predictor is employed to enable the mobile robot to deal in advance with moving obstacles. A grid map is maintained using on‐line sensory data for global path planning, and a bidirectional algorithm is proposed for planning the shortest path for the robot by using updated grid‐map information. Therefore, the mobile robot has the capacity to both learn and adapt to variations. To implement the whole navigation system efficiently, a blackboard model is used to coordinate the computation on board the vehicle. Simulation and experimental results are presented to verify the proposed design and demonstrate smooth navigation behavior of the intelligent mobile robot in dynamic environments. ©1999 John Wiley & Sons, Inc.     

An optimized modular neural network controller based on environment classification and selective sensor usage for mobile robot reactive navigation

A new approach to the design of a neural network (NN) based navigator is proposed in which the mobile robot travels to a pre-defined goal position safely and efficiently without any prior map of the environment. This navigator can be optimized for any user-defined objective function through the use of an evolutionary algorithm. The motivation of this research is to develop an efficient methodology for general goal-directed navigation in generic indoor environments as opposed to learning specialized primitive behaviors in a limited environment. To this end, a modular NN has been employed to achieve the necessary generalization capability across a variety of indoor environments. Herein, each NN module takes charge of navigating in a specialized local environment, which is the result of decomposing the whole path into a sequence of local paths through clustering of all the possible environments. We verify the efficacy of the proposed algorithm over a variety of both simulated and real unstructured indoor environments using our autonomous mobile robot platform.     

未知环境下移动机器人遍历路径规划

为提高未知环境下移动机器人遍历路径规划的效率,提出了一种可动态调节启发式规则的滚动路径规划算法.该算法以生物激励神经网络为环境模型,通过在线识别环境信息特征,动态调用静态搜索算法和环绕障碍搜索算法,有效减少了路径的转弯次数.引入虚拟障碍和直接填充算法,解决了u型障碍区域的连续遍历问题.最后通过仿真实验表明了该方法在未知复杂环境下的有效性.     

基于自适应动态窗口改进细菌算法与移动机器人路径规划

针对移动机器人在复杂环境下的路径规划问题,提出一种新的自适应动态窗口改进细菌算法,并将新算法应用于移动机器人路径规划。改进细菌算法继承了细菌算法与动态窗口算法（dynamic window algorithm, DWA）在避障时的优点,能较好实现复杂环境中移动机器人静态和动态避障。该改进算法主要分三步完成移动机器人路径规划。首先,利用改进细菌趋化算法在静态环境中得到初始参考规划路径。接着,基于参考路径,机器人通过自身携带的传感器感知动态障碍物进行动态避障并利用自适应DWA完成局部动态避障路径规划。最后,根据移动机器人局部动态避障完成情况选择算法执行步骤,如果移动机器人能达到最终目标点,结束该算法,否则移动机器人再重回初始路径,直至到达最终目标点。仿真比较实验证明,改进算法无论在收敛速度还是路径规划精确度方面都有明显提升。     

室外自主移动机器人AMOR的导航技术

在非结构化环境,移动机器人行驶运动规划和自主导航是非常挑战性的问题。基于实时的动态栅格地图,提出了一个快速的而又实效的轨迹规划算法,实现机器人在室外环境的无碰撞运动导航。AMOR是自主研发的室外运动移动机器人,它在2007年欧洲C-ELROB大赛中赢得了野外自主侦察比赛的冠军。它装备了SICK的激光雷达,用来获取机器人运动前方的障碍物体信息,建立实时动态的环境地图。以A*框架为基础的改造算法,能够在众多的路径中快速地找到最佳的安全行驶路径,实现可靠的自主导航。所有的测试和比赛结果表明所提方案是可行的、有效的。