A combined reactive and reinforcement learning controller for an autonomous tracked vehicle

Unmanned ground vehicles currently exhibit simple autonomous behaviours. This paper presents a control algorithm developed for a tracked vehicle to autonomously climb obstacles by varying its front and back track orientations. A reactive controller computes a desired geometric configuration based on terrain information. A reinforcement learning algorithm enhances vehicle mobility by finding effective exit strategies in deadlock situations. It is capable of incorporating complex information including terrain and vehicle dynamics through learned experiences. Experiments illustrate the effectiveness of the proposed approach for climbing various obstacles.     

Reinforcement learning for autonomous vehicle movements in wireless multimedia applications

We develop a Deep Reinforcement Learning (DeepRL)-based, multi-agent algorithm to efficiently control autonomous vehicles that are typically used within the context of Wireless Sensor Networks (WSNs), in order to boost application performance. As an application example, we consider wireless acoustic sensor networks where a group of speakers move inside a room. In a traditional setup, microphones cannot move autonomously and are, e.g., located at fixed positions. We claim that autonomously moving microphones improve the application performance. To control these movements, we compare simple greedy heuristics against a DeepRL solution and show that the latter achieves best application performance.As the range of audio applications is broad and each has its own (subjective) performance metric, we replace those application metrics by two immediately observable ones: First, quality of information (QoI), which is used to measure the quality of sensed data (e.g., audio signal strength). Second, quality of service (QoS), which is used to measure the network’s performance when forwarding data (e.g., delay). In this context, we propose two multi-agent solutions (where one agent controls one microphone) and show that they perform similarly to a single-agent solution (where one agent controls all microphones and has a global knowledge). Moreover, we show via simulations and theoretical analysis how other parameters such as the number of microphones and their speed impacts performance.     

Neurofuzzy identification of an autonomous underwater vehicle

Neurofuzzy modelling is ideally suited to many nonlinear system identification and data modelling applications. By combining the attractive attributes of fuzzy systems and neural networks transparent models of ill-defined systems can be identified. Available expert a priori knowledge is used to construct an initial model. Data modelling techniques from the neural network, statistical and conventional system identification communities are then used to adapt these models. As a result accurate parsimonious models which are transparent and easy to validate are identified. Recent advances in the datadriven identification algorithms have now made neurofuzzy modelling appropriate for high-dimensional problems for which the expert knowledge and data may be of a poor quality. In this paper neurofuzzy modelling techniques are presented. This powerful approach to system identification is demonstrated by its application to the identification of an Autonomous Underwater Vehicle (AUV).     

MARIUS: an autonomous underwater vehicle for coastal oceanography

An autonomous underwater vehicle (AW), named MARIUS, has been developed under the MAST Programme of the Commission of the European Communities. The primary envisioned missions of the prototype AW are environmental surveying and oceanographic data acquisition in coastal waters. The authors describe the design and implementation of the AW systems for vehicle and mission control, and report the results of the sea trials conducted with the vehicle in Sines, Portugal     

A distributed fuzzy logic controller for an autonomous vehicle

Autonomous vehicles can be used in a variety of applications such as hazardous environments or intelligent highway systems. Fuzzy logic is an appropriate choice for this application as it can describe human behavior well. This paper proposes two fuzzy logic controllers for the steering and the velocity control of an autonomous vehicle. The two controllers are divided into separate modules to mimic the way humans think while driving. The steering controller is divided into four modules; one module drives the vehicle toward the target while another module avoids collision with obstacles. A third module drives the vehicle through mazes. The fourth module adjusts the final orientation of the target. The velocity controller is divided into three modules; the first module speeds up the vehicle to reach the target and slows it down as it moves toward the target. The second module controls the velocity in the neighborhood of obstacles. A third module controls the velocity of the vehicle as it turns sharp corners. A method for automatic tuning of the first module of the velocity controller is proposed to stabilize the velocity of the vehicle as it approaches the target. Two examples to demonstrate the interaction among the seven control modules are included. Results of the simulation are compared with those in the literature. © 2004 Wiley Periodicals, Inc.     

Adaptive tracking control of an autonomous underwater vehicle

This paper presents the trajectory tracking control of an autonomous underwater vehicle(AUV). To cope with parametric uncertainties owing to the hydrodynamic effect, an adaptive control law is developed for the AUV to track the desired trajectory. This desired state-dependent regressor matrix-based controller provides consistent results under hydrodynamic parametric uncertainties.Stability of the developed controller is verified using the Lyapunov s direct method. Numerical simulations are carried out to study the efficacy of the proposed adaptive controller.     

基于准最大最小模型预测控制的AUV视觉对接

针对六自由度自主式水下机器人(autonomous underwater vehicle, AUV)视觉对接这一重要课题,提出一种基于融合深度信息的改进准最大最小模型预测控制(quasi-min-max model predictive control, QMM-MPC)方法,有效提高复杂水下视觉伺服对接系统性能.首先,针对水下AUV视觉由于能见度低导致深度信息存在不确定性的影响,建立新的六自由度AUV视觉伺服模型;然后,结合AUV运动和图像特征运动的测量数据,设计在线深度估计器,同时提出结合多李雅普诺夫函数的QMM-MPC算法,通过求取凸多面体中各顶点不同上界值,降低传统QMM-MPC算法中单李雅普诺夫函数上界所带来的强保守性;最后,通过仿真验证所提出方法的有效性和优越性.     

基于分层强化学习的自动驾驶车辆掉头问题研究

调头任务是自动驾驶研究的内容之一,大多数在城市规范道路下的方案无法在非规范道路上实施。针对这一问题,建立了一种车辆掉头动力学模型,并设计了一种多尺度卷积神经网络提取特征图作为智能体的输入。另外还针对调头任务中的稀疏奖励问题,结合分层强化学习和近端策略优化算法提出了分层近端策略优化算法。在简单和复杂场景的实验中,该算法相比于其他算法能够更快地学习到策略,并且具有更高的掉头成功率。     

Inverse optimal self-tuning PID control design for an autonomous underwater vehicle

This paper presents a new approach to path following control design for an autonomous underwater vehicle (AUV). A NARMAX model of the AUV is derived first and then its parameters are adapted online using the recursive extended least square algorithm. An adaptive Propotional-Integral-Derivative (PID) controller is developed using the derived parameters to accomplish the path following task of an AUV. The gain parameters of the PID controller are tuned using an inverse optimal control technique, which alleviates the problem of solving Hamilton–Jacobian equation and also satisfies an error cost function. Simulation studies were pursued to verify the efficacy of the proposed control algorithm. From the obtained results, it is envisaged that the proposed NARMAX model-based self-tuning adaptive PID control provides good path following performance even in the presence of uncertainty arising due to ocean current or hydrodynamic parameter.     

Visual navigation of an autonomous vehicle using white linerecognition

A method is presented for the autonomous visual navigation of a vehicle by using a white guide for path recognition. The vehicle moves over a white guide line on the flat ground or floor, which is contrasted with its background. The vehicle uses a forward-looking TV camera for sensing. The white-line recognition algorithm presented uses a state transition algorithm. A field pattern monitoring method is also presented for vehicle guidance using a state transition scheme. When the vehicle comes to branches, it selects a suitable direction according to a path planner output. The vehicle can also avoid collisions with obstacles in front of it by monitoring the field patterns and their changes. An experimental moving vehicle system was constructed. Tests conforming the effectiveness of these approaches are described     

Observer Kalman filter identification of an autonomous underwater vehicle

This paper deals with the identification of linear discrete-time multivariable models of an autonomous underwater vehicle (AUV). The observer Kalman filter identification (OKID) method is applied with the main objective of evaluating its effectiveness to the experimental identification of the dynamic behaviour of an AUV. After presenting the mathematical background of the OKID algorithm, the proposed method is first validated on the basis of simulated data of both the linearized and nonlinear yaw dynamics of an AUV. Subsequently, the identification algorithm is applied to a set of experimental data. Results suggest that the method can be an efficient tool for the experimental identification of AUV dynamics.     

Chaotic neurodynamics for autonomous agents

Mesoscopic level neurodynamics study the collective dynamical behavior of neural populations. Such models are becoming increasingly important in understanding large-scale brain processes. Brains exhibit aperiodic oscillations with a much more rich dynamical behavior than fixed-point and limit-cycle approximation allow. Here we present a discretized model inspired by Freeman's K-set mesoscopic level population model. We show that this version is capable of replicating the important principles of aperiodic/chaotic neurodynamics while being fast enough for use in real-time autonomous agent applications. This simplification of the K model provides many advantages not only in terms of efficiency but in simplicity and its ability to be analyzed in terms of its dynamical properties. We study the discrete version using a multilayer, highly recurrent model of the neural architecture of perceptual brain areas. We use this architecture to develop example action selection mechanisms in an autonomous agent.     

Three-dimensional optimal path planning for waypoint guidance of an autonomous underwater vehicle

In this paper, optimal three-dimensional paths are generated offline for waypoint guidance of a miniature Autonomous Underwater Vehicle (AUV). Having the starting point, the destination point, and the position and dimension of the obstacles, the AUV is intended to systematically plan an optimal path toward the target. The path is defined as a set of waypoints to be passed by the vehicle. Four criteria are considered for evaluation of an optimal path; they are “total length of path”, “margin of safety”, “smoothness of the planar motion” and “gradient of diving”. A set of Pareto-optimal solutions is found where each solution represents an optimal feasible path that cannot be outrun by any other path considering all four criteria. Then, a proposed three-dimensional guidance system is used for guidance of the AUV through selected optimal paths. This system is inspired from the Line-of-Sight (LOS) guidance strategy; the idea is to select the desired depth, presumed proportional to the horizontal distance of the AUV and the target. To develop this guidance strategy, the dynamic modeling of this novel miniature AUV is also derived. The simulation results show that this guidance system efficiently guides the AUV through the optimal paths.     

Moth-inspired chemical plume tracing on an autonomous underwater vehicle

This paper presents a behavior-based adaptive mission planner (AMP)to trace a chemical plume to its source and reliably declare the source location. The proposed AMP is implemented on a REMUS autonomous underwater vehicle (AUV)equipped with multiple types of sensors that measure chemical concentration,the flow velocity vector, and AUV position, depth, altitude, attitude, and speed. This paper describes the methods and results from experiments conducted in November 2002 on San Clemente Island, CA, using a plume of Rhodamine dye developed in a turbulent fluid flow (i.e., near-shore ocean conditions). These experiments demonstrated chemical plume tracing over 100 m and source declaration accuracy relative to the nominal source location on the order of tens of meters. The designed maneuvers are divided into four behavior types: finding a plume,tracing the plume, reacquiring the plume, and declaring the source location. The tracing and reacquiring behaviors are inspired by male moths flying up wind along a pheromone plume to locate a sexually receptive female. All behaviors are formulated by perception and action modules and translated into chemical plume-tracing algorithms suitable for implementation on a REMUS AUV. To coordinate the different behaviors, the subsumption architecture is adopted to define and arbitrate the behavior priorities. AUVs capable of such feats would have applicability in searching for environmentally interesting phenomena, unexploded ordnance, undersea wreckage, and sources of hazardous chemicals or pollutants.     

Stable nonlinear adaptive controller for an autonomous underwater vehicle using neural networks

In general, the dynamics of autonomous underwater vehicles (AUVs) are highly nonlinear and their hydrodynamic coefficients vary with different operating conditions. For this reason, high performance control system for an AUV usually should have the capacities of learning and adaptation to the time-varying dynamics of the vehicle. In this article, we present a robust adaptive nonlinear control scheme for an AUV, where a linearly parameterized neural network (LPNN) is introduced to approximate the uncertainties of the vehicle's dynamics, and the basis function vector of the network is constructed according to the vehicle's physical properties. The proposed control scheme can guarantee that all of the signals in the closed-loop system are uniformly ultimately bounded (UUB). Numerical simulation studies are performed to illustrate the effectiveness of the proposed control scheme.     

Developmental learning for autonomous robots

Developmental robotics is concerned with the design of algorithms that promote robot adaptation and learning through qualitative growth of behaviour and increasing levels of competence.This paper uses ideas and inspiration from early infant psychology (up to three months of age) to examine how robot systems could discover the structure of their local sensory-motor spaces and learn how to coordinate these for the control of action.An experimental learning model is described and results from robotic experiments using the model are presented and discussed.     

Development of an autonomous aerial vehicle: A case study

This paper describes the design, development, and deployment of an unmanned autonomous aerial vehicle developed at the Georgia Institute of Technology during 1990–1991. The approach taken, the system architecture, and the embedded intelligence of the project as conceived by a team of students, faculty, and industrial affiliates is reported. The project focused on engineering a vehicle which performed an intended mission in the time, space, and weight restrictions specified as part of an AUVS 1991 Competition. This paper documents the system and its various components and also provides a discussion of integration issues.The project demonstrated capabilities of existing and new technologies, but also highlighted many serious integration issues, particularly when using prototype components. The project also demonstrated the utility and mutual benefits of academic-industry projects. All members of the team benefited by working on a real and tangible project. Industrial participates gained first hand experience integrating their products with other components and many saw potential for their products and services in new markets.     

A new approach to vision-based unsupervised learning of unexplored indoor environment for autonomous land vehicle navigation

A vision-based approach to unsupervised learning of the indoor environment for autonomous land vehicle (ALV) navigation is proposed. The ALV may, without human's involvement, self-navigate systematically in an unexplored closed environment, collect the information of the environment features, and then build a top-view map of the environment for later planned navigation or other applications. The learning system consists of three subsystems: a feature location subsystem, a model management subsystem, and an environment exploration subsystem. The feature location subsystem processes input images, and calculates the locations of the local features and the ALV by model matching techniques. To facilitate feature collection, two laser markers are mounted on the vehicle which project laser light on the corridor walls to form easily detectable line and corner features. The model management subsystem attaches the local model into a global one by merging matched corner pairs as well as line segment pairs. The environment exploration subsystem guides the ALV to explore the entire navigation environment by using the information of the learned model and the current ALV location. The guidance scheme is based on the use of a pushdown transducer derived from automata theory. A prototype learning system was implemented on a real vehicle, and simulations and experimental results in real environments show the feasibility of the proposed approach.     

Using shorelines for autonomous air vehicle guidance

We suggest the use of extended landmarks, such as shorelines, creeks, tree lines, and railroads, as well as roads for autonomous navigation of an unmanned air vehicle (UAV). In particular, we recommend the use of shorelines, because of their common availability, their ease of detection, and their significance in terms of events happening along them. Monitoring coastlines and waterways from low flying UAVs has many applications for military and civilian use. We report the development of a vision system that has enabled a prototype UAV to follow shorelines autonomously (without requiring maps or GPS). Using a near-infrared sensor the vision system distinguishes water from land (irrespective of water’s color) and issues commands to the autopilot to follow the coastline or the riverbank. One insight of this problem is that the control algorithm could be integrated deeply with the vision system. This has the benefit of delaying smoothing/regularization so that it could occur in the context of the control coordinate system rather than the image or ground coordinate system. The algorithm itself is simple, but it possibly points the way to future algorithms which could more closely couple image processing and control. Furthermore, the experience gained in this work may be of value in the development of vision systems for following other types of paths.     

Foraging theory for autonomous vehicle speed choice

We consider the optimal control design of an abstract autonomous vehicle (AAV). The AAV searches an area for tasks that are detected with a probability that depends on vehicle speed, and each detected task can be processed or ignored. Both searching and processing are costly, but processing also returns rewards that quantify designer preferences. We generalize results from the analysis of animal foraging behavior to model the AAV. Then, using a performance metric common in behavioral ecology, we explicitly find the optimal speed and task processing choice policy for a version of the AAV problem. Finally, in simulation, we show how parameter estimation can be used to determine the optimal controller online when density of task types is unknown.