An adaptive vision-based autopilot for mini flying machines guidance,navigation and control

The design of reliable navigation and control systems for Unmanned Aerial Vehicles (UAVs) based only on visual cues and inertial data has many unsolved challenging problems, ranging from hardware and software development to pure control-theoretical issues. This paper addresses these issues by developing and implementing an adaptive vision-based autopilot for navigation and control of small and mini rotorcraft UAVs. The proposed autopilot includes a Visual Odometer (VO) for navigation in GPS-denied environments and a nonlinear control system for flight control and target tracking. The VO estimates the rotorcraft ego-motion by identifying and tracking visual features in the environment, using a single camera mounted on-board the vehicle. The VO has been augmented by an adaptive mechanism that fuses optic flow and inertial measurements to determine the range and to recover the 3D position and velocity of the vehicle. The adaptive VO pose estimates are then exploited by a nonlinear hierarchical controller for achieving various navigational tasks such as take-off, landing, hovering, trajectory tracking, target tracking, etc. Furthermore, the asymptotic stability of the entire closed-loop system has been established using systems in cascade and adaptive control theories. Experimental flight test data over various ranges of the flight envelope illustrate that the proposed vision-based autopilot performs well and allows a mini rotorcraft UAV to achieve autonomously advanced flight behaviours by using vision.     

Tracking a Ground Moving Target with a Quadrotor Using Switching Control

An unmanned aerial vehicle (UAV) stabilization strategy based on computer vision and switching controllers is proposed. The main goal of this system is to perform tracking of a moving target on ground. The architecture implemented consists of a quadrotor equipped with an embedded camera which provides real-time video to a computer vision algorithm where images are processed. A vision-based estimator is proposed, which makes use of 2-dimensional images to compute the relative 3-dimensional position and translational velocity of the UAV with respect to the target. The proposed estimator provides the required states measurements to a micro-controller for stabilizing the vehicle during flight. The control strategy consists of switching controllers, which allows making decisions when the target is lost temporarily or when it is out of the camera’s field of view. Real time experiments are presented to demonstrate the performance of the target-tracking system proposed.     

Autonomous quadrotor flight with vision-based obstacle avoidance in virtual environment

In this paper, vision-based autonomous flight with a quadrotor type unmanned aerial vehicle (UAV) is presented. Automatic detection of obstacles and junctions are achieved by the use of optical flow velocities. Variation in the optical flow is used to determine the reference yaw angle. Path to be followed is generated autonomously and the path following process is achieved via a PID controller operating as the low level control scheme. Proposed method is tested in the Google Earth® virtual environment for four different destination points. In each case, autonomous UAV flight is successfully simulated without observing collisions. The results show that the proposed method is a powerful candidate for vision based navigation in an urban environment. Claims are justified with a set of experiments and it is concluded that proper thresholding of the variance of the gradient of optical flow difference have a critical effect on the detectability of roads having different widths.     

Vision-guided flight stability and control for micro air vehicles

Substantial progress has been made recently towards designing, building and test-flying remotely piloted micro air vehicles (MAVs) and small unmanned air vehicles. We seek to complement this progress in overcoming the aerodynamic obstacles to flight at very small scales with a visionguided flight stability and autonomy system, based on a robust horizon detection algorithm. In this paper, we first motivate the use of computer vision for MAV autonomy, arguing that given current sensor technology, vision may be the only practical approach to the problem. We then describe our statistical vision-based horizon detection algorithm, which has been demonstrated at 30 Hz with over 99.9% correct horizon identification. Next, we develop robust schemes for the detection of extreme MAV attitudes, where no horizon is visible, and for the detection of horizon estimation errors, due to external factors such as video transmission noise. Finally, we discuss our feedback controller for selfstabilized flight and report results on vision-based autonomous flights of duration exceeding 10 min. We conclude with an overview of our on-going and future MAV-related research.     

UAS Flight Simulation with Hardware-in-the-loop Testing and Vision Generation

UAS flight simulation for research and development is a difficult problem because each airframe requires accurate physical models, control systems, an organized method of testing new control systems, virtual cameras for vision-based control, and methods of testing new control in the transition from simulation to flight tests. In an environment where researchers are temporary, such as a university, a standard research and development platform with these properties expedites prototyping and prevents code loss when an employee leaves. We develop a research simulation which conforms to all of these properties inside a Matlab environment. A series of mex functions provide connections to the autopilot for hardware-in-the-loop testing, graphical interfaces, and vision processing. The option to write C mex functions offers a seamless method of porting code to embedded systems, minimizing coding errors. We demonstrate fast prototyping by showing flight test data where the simulation provided virtual vision data to avoid virtual obstacles.     

An Experimental Test Bed for Small Unmanned Helicopters

This paper introduces a custom experimental test bed for the evaluation of autonomous flight controllers for unmanned helicopters. The development of controllers for unmanned helicopters is a difficult procedure which involves testing through simulation at first, and then actual experimentation on real vehicles. As simulation cannot accurately represent the exact real flight conditions and the dangers involved in them, the suggested test bed fills the gap between simulation runs and experimental flights. The developed system involves a small helicopter mounted on a flying stand, equipped with a set of sensors for real-time flight monitoring and control. For demonstration purposes, the test bed has been used for design and validation of a fuzzy logic based autopilot, able to perform hovering and altitude control. Experimental results are presented and commented for various test cases.     

Nonlinear sliding sector design for multi‐input systems with application to helicopter control

The ability of helicopters to hover and land vertically has spurred an interesting field of research on the development of autonomous flight for these rotatory wing aircrafts. Linear control theory with gain scheduling, which is based on linearizing the system at the equilibrium points, dominated the helicopter autopilot design. Unlike the linear cascaded autopilot structure used in the existing literature, this paper uses state‐dependent linear like structure, including rate‐limited actuator dynamics, with cascaded autopilot topology. This approach allows nonlinear control laws to be implemented throughout the entire flight envelope, providing satisfactory robustness and stability over the various parameter uncertainties and time delays. The cascaded autopilot topology with nonlinear dynamical equations contains a new sliding sector control (SSC) mechanism which is derived for multi‐input nonlinear dynamical systems. The proposed SSC structure for multi‐input nonlinear systems is used in the inner loop of the cascaded autopilot system where the fastest dynamics are required to be controlled for rapid changes in the helicopter dynamical characteristics which enables one to stabilize the helicopter over a wide range of flight conditions. The proposed cascaded autopilot topology with the new SSC mechanism is tested in simulations to assess its robustness and stability properties. To establish its feasibility, the proposed control method is replaced with a suboptimal control method, namely state‐dependent differential Riccati equation (SDDRE) method, for the inner loop and the results of the proposed control architecture are compared with those of SDDRE method.     

Autopilots for small unmanned aerial vehicles: A survey

This paper presents a survey of the autopilot systems for small or micro unmanned aerial vehicles (UAVs). The objective is to provide a summary of the current commercial, open source and research autopilot systems for convenience of potential small UAV users. The UAV flight control basics are introduced first. The radio control system and autopilot control system are then explained from both the hardware and software viewpoints. Several typical off-the-shelf autopilot packages are compared in terms of sensor packages, observation approaches and controller strengths. Afterwards some open source autopilot systems are introduced. Conclusion is made with a summary of the current autopilot market and a remark on the future development.     

A vision-based strategy for autonomous aerial refueling tasks

Autonomous aerial refueling is a key enabling technology for both manned and unmanned aircraft where extended flight duration or range are required. The results presented within this paper offer one potential vision-based sensing solution, together with a unique test environment. A hierarchical visual tracking algorithm based on direct methods is proposed and developed for the purposes of tracking a drogue during the capture stage of autonomous aerial refueling, and of estimating its 3D position. Intended to be applied in real time to a video stream from a single monocular camera mounted on the receiver aircraft, the algorithm is shown to be highly robust, and capable of tracking large, rapid drogue motions within the frame of reference. The proposed strategy has been tested using a complex robotic testbed and with actual flight hardware consisting of a full size probe and drogue. Results show that the vision tracking algorithm can detect and track the drogue at real-time frame rates of more than thirty frames per second, obtaining a robust position estimation even with strong motions and multiple occlusions of the drogue.     

PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision

We describe a novel quadrotor Micro Air Vehicle (MAV) system that is designed to use computer vision algorithms within the flight control loop. The main contribution is a MAV system that is able to run both the vision-based flight control and stereo-vision-based obstacle detection parallelly on an embedded computer onboard the MAV. The system design features the integration of a powerful onboard computer and the synchronization of IMU-Vision measurements by hardware timestamping which allows tight integration of IMU measurements into the computer vision pipeline. We evaluate the accuracy of marker-based visual pose estimation for flight control and demonstrate marker-based autonomous flight including obstacle detection using stereo vision. We also show the benefits of our IMU-Vision synchronization for egomotion estimation in additional experiments where we use the synchronized measurements for pose estimation using the 2pt+gravity formulation of the PnP problem.     

A vision-based landing system for small unmanned aerial vehicles using an airbag

Statistics show that the landing accounts for the largest portion of all mishaps of unmanned aerial vehicles (UAVs) due to many difficulties including limited situational awareness of the external pilot and the limited maneuverability during the low speed flight before touchdown. In this paper, a vision-based automatic landing system using a dome-shaped airbag is proposed for small UAVs. Its isotropic shape allows airplanes to approach in any direction to avoid crosswind unlike net-assisted landing. The dome’s distinctive color improves the detection owing to its strong visual cue. Color- and shape-based detection vision algorithms are applied for robust detection under varying lighting conditions. Due to the insufficient accuracy of navigation sensors, a direct visual servoing is used for terminal guidance. The proposed algorithm is validated in a series of flight tests.     

Dual-eyes Vision-based Docking System for Autonomous Underwater Vehicle: an Approach and Experiments

A critical challenge for autonomous underwater vehicles (AUVs) is the docking operation for applications such as sleeping under the mother ship, recharging batteries, transferring data, and new mission downloading. The final stage of docking at a unidirectional docking station requires the AUV to approach while keeping the pose (position and orientation) of the vehicle within an allowable range. The appropriate pose therefore demands a sensor unit and a control system that have high accuracy and robustness against disturbances existing in a real-world underwater environment. This paper presents a vision-based AUV docking system consisting of a 3D model-based matching method and Real-time Multi-step Genetic Algorithm (GA) for real-time estimation of the robot’s relative pose. Experiments using a remotely operated vehicle (ROV) with dual-eye cameras and a separate 3D marker were conducted in a small indoor pool. The experimental results confirmed that the proposed system is able to provide high homing accuracy and robustness against disturbances that influence not only the captured camera images but also the movement of the vehicle. A successful docking operation using stereo vision that is new and novel to the underwater vehicle environment was achieved and thus proved the effectiveness of the proposed system for AUV.     

Central Processing Unit for an Autopilot: Description and Hardware-In-the-Loop Simulation

This paper describes the architecture of the Central Processing Unit (CPU) of Pegasus AutoPilot, which is an academic autopilot, in the developmental stage, for small Unmanned Aerial Vehicles (UAVs). The data manager process and control laws, implemented on dedicated hardware, are described. In order to verify, validate, and optimize the system a Hardware-In-the-Loop (HIL) simulation, between the CPU and the flight simulator X-Plane is performed. X-Plane simulates the other systems of the autopilot, such as the sensors and actuators. The system is designed to facilitate the disconnection of the flight simulator and the connection of the real navigation hardware and control surface manager drive, as a plug and play device. The described control loops, consisting of inner and outer loops, controls the aircraft’s attitude and maintains a constant altitude, direction, and speed. The work presented can also be used as a guide for those who wants to begin to use Hardware-In-the-Loop Simulations using X-Plane.     

Coning motion stability and decoupling methods analysis of spinning missiles with angle-of-attack feedback autopilot

The angle-of-attack (AOA) three-loop feedback autopilot is an improved three-loop autopilot that has been widely employed in missile control systems. For spinning missiles, however, unstable coning motion can be induced by the cross-couple effect. For spinning missiles with an AOA feedback autopilot, this paper analyzes the coning motion stability with the consideration of the augmentation loop and the position of the accelerometer and compares the performance of three decoupling methods. First, a novel double-channel actuator AOA three-loop autopilot is established, and the sufficient and necessary condition of coning motion stability is proposed analytically on account of the complex system equations. The stability condition shows that the stable region of the design parameters for the autopilot shrinks as a result of the spinning condition. Moreover, methods associated with actuator dynamics, control coupling, static stability, and the decoupling method of setting the lead angle to the control system are presented to improve the stability of spinning missiles. Numerical simulations are implemented to demonstrate the accuracy of the proposed methods, whose results illustrate that the stability conditions can guide AOA autopilot design for the flight stabilization of spinning missiles.     

Web-based mobile robot platform for real-time exercises

This paper introduces a new vision-based and web-based mobile robot platform. The platform consists of control and communication centers, a mobile robot and real-time support libraries. All activities in the platform are achieved by only computer vision techniques. The platform provides monitoring, tele-controlling and programming for real-time educational exercises and helps to the users to achieve these exercises through a standard web browser without any need for additional support software. The results have shown that the proposed, designed and implemented platform provide amazing new facilities and features to the users (students and researchers) in applying their real-time exercises on web.     

Integrated real-time vision system for vehicle control in non-structured environments

An integrated vision-based real-time vehicle control system is proposed, as implemented in the experimental vehicle ROMEO-3R. One of the applications developed, an active object-tracking method using generalized predictive control, is also described. The real-time computer-vision module is based on a multiprocessor, DSP-based system; fast processing is achieved by parallel execution of the main image-processing algorithms. A pan-and-tilt platform allows the tracking process to be decoupled from vehicle motion.     

某小型无人机飞控系统的中断控制器设计

为了提高无人机飞行控制系统对外部事件快速响应及实时处理的能力,中断是一种非常有效的实现手段;文章针对以TI公司DSP芯片TMS320LF2407A为核心的某小型无人机机载计算机系统,提出并实现了基于复杂可编程逻辑单元(CPLD),对无人机飞控系统外部中断端口进行扩展和管理的设计方案,详细给出了中断控制系统的硬件接口设计、程序设计及仿真结果;该设计已成功应用于某小型无人机飞控系统中,提高了无人机飞控系统对外部中断事件的管理和实时处理能力。     

分布交互式实时三维飞行仿真平台的综合设计

针对基于高阶非线性系统数学模型基础上的飞行器控制系统设计难于验证的实际问题,在基于全数字飞机的非线性模型基础上,在分布式局域网的仿真环境下,利用飞行控制律实时算法、面向对象的编程方法、TCP/IP网络通信技术及计算机三维仿真等技术,构建了分布交互式实时三维飞行控制系统仿真平台,研究了该平台的软硬件构成、设计方法及相关的关键技术,并且在网络环境下分别开发了基于增益表和自修复重构的实时仿真算法,成功模拟了飞机在驾驶员在环和全自动飞行的情况,研究结果表明了该仿真平台在飞行仿真训练和教学中的实用性以及验证先进控制律的可移植性.     

Optic flow regulation: the key to aircraft automatic guidance

We have developed a visually based autopilot which is able to make an air vehicle automatically take off, cruise and land, while reacting appropriately to wind disturbances (head wind and tail wind). This autopilot consists of a visual control system that adjusts the thrust so as to keep the downward optic flow (OF) at a constant value. This autopilot is therefore based on an optic flow regulation loop. It makes use of a sensor, which is known as an elementary motion detector (EMD). The functional structure of this EMD was inspired by that of the housefly, which was previously investigated at our Laboratory by performing electrophysiological recordings while applying optical microstimuli to single photoreceptor cells of the insect's compound eye.

We built a proof-of-concept, tethered rotorcraft that circles indoors over an environment composed of contrasting features randomly arranged on the floor. The autopilot, which we have called OCTAVE (Optic flow based Control sysTem for Aerial VEhicles), enables this miniature (100 g) rotorcraft to carry out complex tasks such as ground avoidance and terrain following, to control risky maneuvers such as automatic take off and automatic landing, and to respond appropriately to wind disturbances. A single visuomotor control loop suffices to perform all these reputedly demanding tasks. As the electronic processing system required is extremely light-weight (only a few grams), it can be mounted on-board micro-air vehicles (MAVs) as well as larger unmanned air vehicles (UAVs) or even submarines and autonomous underwater vehicles (AUVs). But the OCTAVE autopilot could also provide guidance and/or warning signals to prevent the pilots of manned aircraft from colliding with shallow terrain, for example.     

Real-time Implementation and Validation of a New Hierarchical Path Planning Scheme of UAVs via Hardware-in-the-Loop Simulation

We present a real-time hardware-in-the-loop simulation environment for the validation of a new hierarchical path planning and control algorithm for a small fixed-wing unmanned aerial vehicle (UAV). The complete control algorithm is validated through on-board, real-time implementation on a small autopilot having limited computational resources. We present two distinct real-time software frameworks for implementing the overall control architecture, including path planning, path smoothing, and path following. We emphasize, in particular, the use of a real-time kernel, which is shown to be an effective and robust way to accomplish real-time operation of small UAVs under non-trivial scenarios. By seamless integration of the whole control hierarchy using the real-time kernel, we demonstrate the soundness of the approach. The UAV equipped with a small autopilot, despite its limited computational resources, manages to accomplish sophisticated unsupervised navigation to the target, while autonomously avoiding obstacles.