Omnidirectional vision applied to Unmanned Aerial Vehicles (UAVs) attitude and heading estimation

This paper presents an aircraft attitude and heading estimator using catadioptric images as a principal sensor for UAV or as a redundant system for IMU (Inertial Measure Unit) and gyro sensors. First, we explain how the unified theory for central catadioptric cameras is used for attitude and heading estimation, explaining how the skyline is projected on the catadioptric image and how it is segmented and used to calculate the UAV’s attitude. Then, we use appearance images to obtain a visual compass, and we calculate the relative rotation and heading of the aerial vehicle. Finally the tests and results using the UAV COLIBRI platform and the validation of them in real flights are presented, comparing the estimated data with the inertial values measured on board.     

A Catadioptric and Pan-Tilt-Zoom Camera Pair Object Tracking System for UAVs

Unmanned aerial vehicles (UAVs) are seeing widespread use in military, scientific, and civilian sectors in recent years. As the mission demands increase, these systems are becoming more complicated. Omnidirectional camera is a vision sensor that can captures 360° view in a single frame. In recent years omnidirectional camera usage has experienced a remarkable increase in many fields, where many innovative research has been done. Although, it is very promising, employment of omnidirectional cameras in UAVs is quite new. In this paper, an innovative sensory system is proposed, that has an omnidirectional imaging device and a pan tilt zoom (PTZ) camera. Such a system combines the advantages of both of the camera systems. The system can track any moving object within its 360° field of view and provide detailed images of it. The detection of the moving object has been accomplished by an adaptive background subtraction method implemented on the lowered resolution images of the catadioptric camera. A novel algorithm has also been developed to estimate the relative distance of the object with respect to the UAV, using tracking information of both of the cameras. The algorithms are implemented on an experimental system to validate the approach.     

Vision Based UAV Attitude Estimation: Progress and Insights

Unmanned aerial vehicles (UAVs) are increasingly replacing manned systems in situations that are dangerous, remote, or difficult for manned aircraft to access. Its control tasks are empowered by computer vision technology. Visual sensors are robustly used for stabilization as primary or at least secondary sensors. Hence, UAV stabilization by attitude estimation from visual sensors is a very active research area. Vision based techniques are proving their effectiveness and robustness in handling this problem. In this work a comprehensive review of UAV vision based attitude estimation approaches is covered, starting from horizon based methods and passing by vanishing points, optical flow, and stereoscopic based techniques. A novel segmentation approach for UAV attitude estimation based on polarization is proposed. Our future insightes for attitude estimation from uncalibrated catadioptric sensors are also discussed.     

Unmanned aerial vehicles UAVs attitude,height, motion estimation and control using visual systems

This paper presents an implementation of an aircraft pose and motion estimator using visual systems as the principal sensor for controlling an Unmanned Aerial Vehicle (UAV) or as a redundant system for an Inertial Measure Unit (IMU) and gyros sensors. First, we explore the applications of the unified theory for central catadioptric cameras for attitude and heading estimation, explaining how the skyline is projected on the catadioptric image and how it is segmented and used to calculate the UAV’s attitude. Then we use appearance images to obtain a visual compass, and we calculate the relative rotation and heading of the aerial vehicle. Additionally, we show the use of a stereo system to calculate the aircraft height and to measure the UAV’s motion. Finally, we present a visual tracking system based on Fuzzy controllers working in both a UAV and a camera pan and tilt platform. Every part is tested using the UAV COLIBRI platform to validate the different approaches, which include comparison of the estimated data with the inertial values measured onboard the helicopter platform and the validation of the tracking schemes on real flights.     

Photometric visual servoing for omnidirectional cameras

2D visual servoing consists in using data provided by a vision sensor for controlling the motions of a dynamic system. Most of visual servoing approaches has relied on the geometric features that have to be tracked and matched in the image acquired by the camera. Recent works have highlighted the interest of taking into account the photometric information of the entire image. This approach was tackled with images of perspective cameras. We propose, in this paper, to extend this technique to central cameras. This generalization allows to apply this kind of method to catadioptric cameras and wide field of view cameras. Several experiments have been successfully done with a fisheye camera in order to control a 6 degrees of freedom robot and with a catadioptric camera for a mobile robot navigation task.     

Bearing-only visual SLAM for small unmanned aerial vehicles in GPS-denied environments

This paper presents a hierarchical simultaneous localization and mapping(SLAM) system for a small unmanned aerial vehicle(UAV) using the output of an inertial measurement unit(IMU) and the bearing-only observations from an onboard monocular camera.A homography based approach is used to calculate the motion of the vehicle in 6 degrees of freedom by image feature match.This visual measurement is fused with the inertial outputs by an indirect extended Kalman filter(EKF) for attitude and velocity estimation.Then,another EKF is employed to estimate the position of the vehicle and the locations of the features in the map.Both simulations and experiments are carried out to test the performance of the proposed system.The result of the comparison with the referential global positioning system/inertial navigation system(GPS/INS) navigation indicates that the proposed SLAM can provide reliable and stable state estimation for small UAVs in GPS-denied environments.     

A Data Fusion System for Attitude Estimation of Low-cost Miniature UAVs

Miniature unmanned aerial vehicles (UAVs) have attracted wide interest from researchers and developers because of their broad applications. In order to make a miniature UAV platform popular for civilian applications, one critical concern is the overall cost. However, lower cost generally means lower navigational accuracy and insufficient flight control performance, mainly due to the low graded avionics on the UAV. This paper introduces a data fusion system based on several low-priced sensors to improve the attitude estimation of a low-cost miniature fixed-wing UAV platform. The characteristics of each sensor and the calculation of attitude angles are carefully studied. The algorithms and implementation of the fusion system are described and explained in details. Ground test results with three sensor fusions are compared and analyzed, and flight test comparison results with two sensor fusions are also presented.     

基于四元数衍生无迹卡尔曼滤波的二段式多旋翼无人机姿态估计算法

对传统多旋翼无人机姿态估计算法难以兼顾高精度、强实时性以及抗干扰能力差的问题,首先基于一种计算量较小的衍生无迹卡尔曼滤波算法,在量测更新中,将加速度数据和磁力计数据分为两个阶段进行姿态四元数校正处理,然后从旋转四元数的本质出发,推测出四元数各元素分别包含着不同的姿态角信息,最后将校正四元数分别乘上为降低校正过程中的相互干扰所设计的系数,提出一种基于四元数衍生无迹卡尔曼滤波的二段式多旋翼无人机姿态估计算法.通过使用PIXHAWK飞控数据,与传统姿态估计算法进行仿真实验对比,实验表明,本文提出算法与传统使用扩展卡尔曼滤波(EKF)或无迹卡尔曼滤波(UKF)的姿态估计算法相比,在实时性、解算精度和抗干扰能力方面有较大提升.     

Nonlinear Kalman Filters and Particle Filters for integrated navigation of unmanned aerial vehicles

The paper studies and compares nonlinear Kalman Filtering methods and Particle Filtering methods for estimating the state vector of Unmanned Aerial Vehicles (UAVs) through the fusion of sensor measurements. Next, the paper proposes the use of the estimated state vector in a control loop for autonomous navigation and trajectory tracking by the UAVs. The proposed nonlinear controller is derived according to the flatness-based control theory. The estimation of the UAV’s state vector is carried out with the use of (i) Extended Kalman Filtering (EKF), (ii) Sigma-Point Kalman Filtering (SPKF), (iii) Particle Filtering (PF), and (iv) a new nonlinear estimation method which is the Derivative-free nonlinear Kalman Filtering (DKF). The performance of the nonlinear control loop which is based on these nonlinear state estimation methods is evaluated through simulation tests. Comparing the aforementioned filtering methods in terms of estimation accuracy and computation speed, it is shown that the Sigma-Point Kalman Filtering is a reliable and computationally efficient approach to state estimation-based control, while Particle Filtering is well-suited to accommodate non-Gaussian measurements. Moreover, it is shown that the Derivative-free nonlinear Kalman Filter is faster than the rest of the nonlinear filters while also succeeding accurate, in terms of variance, state estimates.     

Low‐altitude Terrestrial Spectroscopy from a Pushbroom Sensor

Hyperspectral cameras sample many different spectral bands at each pixel, enabling advanced detection and classification algorithms. However, their limited spatial resolution and the need to measure the camera motion to create hyperspectral images makes them unsuitable for nonsmooth moving platforms such as unmanned aerial vehicles (UAVs). We present a procedure to build hyperspectral images from line sensor data without camera motion information or extraneous sensors. Our approach relies on an accompanying conventional camera to exploit the homographies between images for mosaic construction. We provide experimental results from a low‐altitude UAV, achieving high‐resolution spectroscopy with our system.     

Improved Kalman-based attitude estimation framework for UAVs via an antenna array

Accurate attitude estimation is crucial for Unmanned Aerial Vehicles (UAVs) in order to facilitate automated activities such as landing or trajectory tracking. Recently antenna array based communication systems have been installed in UAVs. This array structure can also be applied for attitude estimation by computing the line-of-sight (LOS) path between the base station and UAV. In this paper, we propose a complete framework for attitude estimation by exploiting 3D LOS vector obtained from the antenna array system. We present all the steps to incorporate the estimated LOS vector into the TRIaxial Attitude Determination (TRIAD), QUaternion ESTimator (QUEST) and Kalman algorithms. As an additional contribution, the error covariance matrix of the LOS vector is analytically calculated by first finding the phase shift mean squared error using the known perturbation model from Singular Value Decomposition and assuming that the antenna array measured data error can be modeled as a circularly symmetric white noise. We evaluate five array configurations via Monte Carlo simulations. We show that array configurations that provide orthogonal components of the LOS vector achieve a better performance. The usage of more than three pairs of antennas to improve the estimation of the LOS vector is also proposed for low and intermediate signal-to-noise ratio regimes.     

Evaluation of the Unified Model of the Sphere for Fisheye Cameras in Robotic Applications

Abstract

A wide field of view is required for many robotic vision tasks. Such an aperture may be acquired by a fisheye camera, which provides a full image compared to catadioptric visual sensors, and does not increase the size and the weakness of the imaging system with respect to perspective cameras. While a unified model exists for all central catadioptric systems, many different models, approximating the radial distortions, exist for fisheye cameras. It is shown in this paper that the unified projection model proposed for central catadioptric cameras is also valid for fisheye cameras in the context of robotic applications. This model consists of a projection onto a virtual unitary sphere followed by a perspective projection onto an image plane. This model is shown equivalent to almost all the fisheye models. Calibration with four cameras and partial Euclidean reconstruction are done using this model, and lead to persuasive results. Finally, an application to a mobile robot navigation task is proposed and correctly executed along a 200-m trajectory.     

一种反射折射摄像机的简易标定方法

Central catadioptric cameras are widely used in virtual reality and robot navigation, and the camera calibration is a prerequisite for these applications. In this paper, we propose an easy calibration method for central catadioptric cameras with a 2D calibration pattern. Firstly, the bounding ellipse of the catadioptric image and field of view (FOV) are used to obtain the initial estimation of the intrinsic parameters. Then, the explicit relationship between the central catadioptric and the pinhole model is used to initialize the extrinsic parameters. Finally, the intrinsic and extrinsic parameters are refined by nonlinear optimization. The proposed method does not need any fitting of partial visible conic, and the projected images of 2D calibration pattern can easily cover the whole image, so our method is easy and robust. Experiments with simulated data as well as real images show the satisfactory performance of our proposed calibration method.     

一种反射折射摄像机的简易标定方法

Central catadioptric cameras are widely used in virtual reality and robot navigation,and the camera calibration is a prerequisite for these applications.In this paper,we propose an easy calibration method for central catadioptric cameras with a 2D calibration pattern.Firstly,the bounding ellipse of the catadioptric image and field of view (FOV) are used to obtain the initial estimation of the intrinsic parameters.Then,the explicit relationship between the central catadioptric and the pinhole model is used to initialize the extrinsic parameters.Finally,the intrinsic and extrinsic parameters are refined by nonlinear optimization.The proposed method does not need any fitting of partial visible conic,and the projected images of 2D calibration pattern can easily cover the whole image,so our method is easy and robust.Experiments with simulated data as well as real images show the satisfactory performance of our proposed calibration method.     

Attitude Determination of Multirotor Aerial Vehicles Using Camera Vector Measurements

The employment of embedded cameras in navigation and guidance of Unmanned Aerial Vehicles (UAV) has attracted the focus of many academic researches. In particular, for the multirotor UAV, the camera is widely employed for applications performed in indoor environments, where the GNSS signal is often unreliable and electromagnetic interference can be a concern. In the literature, images are mostly adopted for position and velocity estimation, rather than attitude estimation. This paper proposes an attitude determination method for multirotor aerial vehicles using pairs of vector measurements taken from a downward-facing strapdown camera. The method is composed of three modules. The first one detects and identifies the visible landmarks by processing the images. The second module computes the vector measurements related to the direction from the camera to the landmarks. The third module estimates attitude from the vector measurements. In the last module, a version of the Multiplicative Extended Kalman Filter (MEKF) with sequential update is proposed as estimation method. The overall method is evaluated via Monte Carlo simulations, showing that it is effective in determining the vehicle’s attitude and revealing its properties.     

Indoor UAV Control Using Multi-Camera Visual Feedback

This paper presents the control of an indoor unmanned aerial vehicle (UAV) using multi-camera visual feedback. For the autonomous flight of the indoor UAV, instead of using onboard sensor information, visual feedback concept is employed by the development of an indoor flight test-bed. The indoor test-bed consists of four major components: the multi-camera system, ground computer, onboard color marker set, and quad-rotor UAV. Since the onboard markers are attached to the pre-defined location, position and attitude of the UAV can be estimated by marker detection algorithm and triangulation method. Additionally, this study introduces a filter algorithm to obtain the full 6-degree of freedom (DOF) pose estimation including velocities and angular rates. The filter algorithm also enhances the performance of the vision system by making up for the weakness of low cost cameras such as poor resolution and large noise. Moreover, for the pose estimation of multiple vehicles, data association algorithm using the geometric relation between cameras is proposed in this paper. The control system is designed based on the classical proportional-integral-derivative (PID) control, which uses the position, velocity and attitude from the vision system and the angular rate from the rate gyro sensor. This paper concludes with both ground and flight test results illustrating the performance and properties of the proposed indoor flight test-bed and the control system using the multi-camera visual feedback.     

Image Based and Hybrid Visual Servo Control of an Unmanned Aerial Vehicle

The use of unmanned aerial vehicles (UAVs) for military, scientific, and civilian sectors are increasing drastically in recent years. This study presents algorithms for the visual-servo control of an UAV, in which a quadrotor helicopter has been stabilized with visual information through the control loop. Unlike previous study that use pose estimation approach which is time consuming and subject to various errors, the visual-servo control is more reliable and fast. The method requires a camera on-board the vehicle, which is already available on various UAV systems. The UAV with a camera behaves like an eye-in-hand visual servoing system. In this study the controller was designed by using two different approaches; image based visual servo control method and hybrid visual servo control method. Various simulations are developed on Matlab, in which the quadrotor aerial vehicle has been visual-servo controlled. In order to show the effectiveness of the algorithms, experiments were performed on a model quadrotor UAV, which suggest successful performance.     

A Theory of Single-Viewpoint Catadioptric Image Formation

Conventional video cameras have limited fields of view which make them restrictive for certain applications in computational vision. A catadioptric sensor uses a combination of lenses and mirrors placed in a carefully arranged configuration to capture a much wider field of view. One important design goal for catadioptric sensors is choosing the shapes of the mirrors in a way that ensures that the complete catadioptric system has a single effective viewpoint. The reason a single viewpoint is so desirable is that it is a requirement for the generation of pure perspective images from the sensed images. In this paper, we derive the complete class of single-lens single-mirror catadioptric sensors that have a single viewpoint. We describe all of the solutions in detail, including the degenerate ones, with reference to many of the catadioptric systems that have been proposed in the literature. In addition, we derive a simple expression for the spatial resolution of a catadioptric sensor in terms of the resolution of the cameras used to construct it. Moreover, we include detailed analysis of the defocus blur caused by the use of a curved mirror in a catadioptric sensor.     

Vision-aided Estimation of Attitude,Velocity, and Inertial Measurement Bias for UAV Stabilization

This paper studies vision-aided inertial navigation of small-scale unmanned aerial vehicles (UAVs) in GPS-denied environments. The objectives of the navigation system are to firstly online estimate and compensate the unknown inertial measurement biases, secondly provide drift-free velocity and attitude estimates which are crucial for UAV stabilization control, and thirdly give relatively accurate position estimation such that the UAV is able to perform at least a short-term navigation when the GPS signal is not available. For the vision system, we do not presume maps or landmarks of the environment. The vision system should be able to work robustly even given low-resolution images (e.g., 160 ×120 pixels) of near homogeneous visual features. To achieve these objectives, we propose a novel homography-based vision-aided navigation system that adopts four common sensors: a low-cost inertial measurement unit, a downward-looking monocular camera, a barometer, and a compass. The measurements of the sensors are fused by an extended Kalman filter. Based on both analytical and numerical observability analyses of the navigation system, we theoretically verify that the proposed navigation system is able to achieve the navigation objectives. We also show comprehensive simulation and real flight experimental results to verify the effectiveness and robustness of the proposed navigation system.     

基于改进扩展卡尔曼滤波的姿态解算算法研究

为了解决标准扩展卡尔曼滤波器(EKF)在多旋翼无人机姿态解算中精度较低的问题,提出了一种改进扩展卡尔曼滤波算法(BPNN-EKF),使得解算精度得到较大提升。针对EKF存在预测模型参数要求具有先验已知性,在工程实践中难以获得准确的参数,以及标准EKF对非线性系统采用线性化模型带来的误差等问题,利用神经网络的非线性映射能力和自适应能力对标准EKF的估计值进行补偿,减小模型以及滤波参数误差对最优估计值的影响,从而提高最优估计精度。仿真实验证明,BPNN-EKF对多旋翼无人机姿态解算精度的提升具有显著作用。