Monocular Vision for Long‐term Micro Aerial Vehicle State Estimation: A Compendium

The recent technological advances in Micro Aerial Vehicles (MAVs) have triggered great interest in the robotics community, as their deployability in missions of surveillance and reconnaissance has now become a realistic prospect. The state of the art, however, still lacks solutions that can work for a long duration in large, unknown, and GPS‐denied environments. Here, we present our visual pipeline and MAV state‐estimation framework, which uses feeds from a monocular camera and an Inertial Measurement Unit (IMU) to achieve real‐time and onboard autonomous flight in general and realistic scenarios. The challenge lies in dealing with the power and weight restrictions onboard a MAV while providing the robustness necessary in real and long‐term missions. This article provides a concise summary of our work on achieving the first onboard vision‐based power‐on‐and‐go system for autonomous MAV flights. We discuss our insights on the lessons learned throughout the different stages of this research, from the conception of the idea to the thorough theoretical analysis of the proposed framework and, finally, the real‐world implementation and deployment. Looking into the onboard estimation of monocular visual odometry, the sensor fusion strategy, the state estimation and self‐calibration of the system, and finally some implementation issues, the reader is guided through the different modules comprising our framework. The validity and power of this framework are illustrated via a comprehensive set of experiments in a large outdoor mission, demonstrating successful operation over flights of more than 360 m trajectory and 70 m altitude change. ¹      

Intuitive 3D Maps for MAV Terrain Exploration and Obstacle Avoidance

Recent development showed that Micro Aerial Vehicles (MAVs) are nowadays capable of autonomously take off at one point and land at another using only one single camera as exteroceptive sensor. During the flight and landing phase the MAV and user have, however, little knowledge about the whole terrain and potential obstacles. In this paper we show a new solution for a real-time dense 3D terrain reconstruction. This can be used for efficient unmanned MAV terrain exploration and yields a solid base for standard autonomous obstacle avoidance algorithms and path planners. Our approach is based on a textured 3D mesh on sparse 3D point features of the scene. We use the same feature points to localize and control the vehicle in the 3D space as we do for building the 3D terrain reconstruction mesh. This enables us to reconstruct the terrain without significant additional cost and thus in real-time. Experiments show that the MAV is easily guided through an unknown, GPS denied environment. Obstacles are recognized in the iteratively built 3D terrain reconstruction and are thus well avoided.     

Towards a swarm of agile micro quadrotors

We describe a prototype 75 g micro quadrotor with onboard attitude estimation and control that operates autonomously with an external localization system. The motivation for designing quadrotors at this scale comes from two observations. First, the agility of the robot increases with a reduction in size, a fact that is supported by experimental results in this paper. Second, smaller robots are able to operate in tight formations in constrained, indoor environments. We describe the hardware and software used to operate the vehicle as well our dynamic model. We also discuss the aerodynamics of vertical flight and the contribution of ground effect to the vehicle performance. Finally, we discuss architecture and algorithms to coordinate a team of these quadrotors, and provide experimental results for a team of 20 micro quadrotors.     

Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. This study covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined “Treasure Hunt,” which required mission planning and multirobot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge‐specific extensions, cover the camera‐based perception, explain control and trajectory‐planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real‐robot experiments during the competition in Abu Dhabi. With our system, we—as part of the larger team NimbRo—won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.     

Autonomous Vision-Based Helicopter Flights Through Obstacle Gates

The challenge for unmanned aerial vehicles to sense and avoid obstacles becomes even harder if narrow passages have to be crossed. An approach to solve a mission scenario that tackles the problem of such narrow passages is presented here. The task is to fly an unmanned helicopter autonomously through a course with gates that are only slightly larger than the vehicle itself. A camera is installed on the vehicle to detect the gates. Using vehicle localization data from a navigation solution, camera alignment and global gate positions are estimated simultaneously. The presented algorithm calculates the desired target waypoints to fly through the gates. Furthermore, the paper presents a mission execution plan that instructs the vehicle to search for a gate, to fly through it after successful detection, and to search for a proceeding one. All algorithms are designed to run onboard the vehicle so that no interaction with the ground control station is necessary, making the vehicle completely autonomous. To develop and optimize algorithms, and to prove the correctness and accuracy of vision-based gate detection under real operational conditions, gate positions are searched in images taken from manual helicopter flights. Afterwards, the integration of visual sensing and mission control is proven. The paper presents results from full autonomous flight where the helicopter searches and flies through a gate without operator actions.     

Multiple‐Robot Simultaneous Localization and Mapping: A Review

Simultaneous localization and mapping (SLAM) in unknown GPS‐denied environments is a major challenge for researchers in the field of mobile robotics. Many solutions for single‐robot SLAM exist; however, moving to a platform of multiple robots adds many challenges to the existing problems. This paper reviews state‐of‐the‐art multiple‐robot systems, with a major focus on multiple‐robot SLAM. Various issues and problems in multiple‐robot SLAM are introduced, current solutions for these problems are reviewed, and their advantages and disadvantages are discussed.     

GPS‐denied Indoor and Outdoor Monocular Vision Aided Navigation and Control of Unmanned Aircraft

GPS‐denied closed‐loop autonomous control of unstable Unmanned Aerial Vehicles (UAVs) such as rotorcraft using information from a monocular camera has been an open problem. Most proposed Vision aided Inertial Navigation Systems (V‐INSs) have been too computationally intensive or do not have sufficient integrity for closed‐loop flight. We provide an affirmative answer to the question of whether V‐INSs can be used to sustain prolonged real‐world GPS‐denied flight by presenting a V‐INS that is validated through autonomous flight‐tests over prolonged closed‐loop dynamic operation in both indoor and outdoor GPS‐denied environments with two rotorcraft unmanned aircraft systems (UASs). The architecture efficiently combines visual feature information from a monocular camera with measurements from inertial sensors. Inertial measurements are used to predict frame‐to‐frame transition of online selected feature locations, and the difference between predicted and observed feature locations is used to bind in real‐time the inertial measurement unit drift, estimate its bias, and account for initial misalignment errors. A novel algorithm to manage a library of features online is presented that can add or remove features based on a measure of relative confidence in each feature location. The resulting V‐INS is sufficiently efficient and reliable to enable real‐time implementation on resource‐constrained aerial vehicles. The presented algorithms are validated on multiple platforms in real‐world conditions: through a 16‐min flight test, including an autonomous landing, of a 66 kg rotorcraft UAV operating in an unconctrolled outdoor environment without using GPS and through a Micro‐UAV operating in a cluttered, unmapped, and gusty indoor environment. © 2013 Wiley Periodicals, Inc.     

On the design and development of attitude stabilization, vision-based navigation, and aerial gripping for a low-cost quadrotor

This paper presents the design and development of autonomous attitude stabilization, navigation in unstructured, GPS-denied environments, aggressive landing on inclined surfaces, and aerial gripping using onboard sensors on a low-cost, custom-built quadrotor. The development of a multi-functional micro air vehicle (MAV) that utilizes inexpensive off-the-shelf components presents multiple challenges due to noise and sensor accuracy, and there are control challenges involved with achieving various capabilities beyond navigation. This paper addresses these issues by developing a complete system from the ground up, addressing the attitude stabilization problem using extensive filtering and an attitude estimation filter recently developed in the literature. Navigation in both indoor and outdoor environments is achieved using a visual Simultaneous Localization and Mapping (SLAM) algorithm that relies on an onboard monocular camera. The system utilizes nested controllers for attitude stabilization, vision-based navigation, and guidance, with the navigation controller implemented using a nonlinear controller based on the sigmoid function. The efficacy of the approach is demonstrated by maintaining a stable hover even in the presence of wind gusts and when manually hitting and pulling on the quadrotor. Precision landing on inclined surfaces is demonstrated as an example of an aggressive maneuver, and is performed using only onboard sensing. Aerial gripping is accomplished with the addition of a secondary camera, capable of detecting infrared light sources, which is used to estimate the 3D location of an object, while an under-actuated and passively compliant manipulator is designed for effective gripping under uncertainty. The quadrotor is therefore able to autonomously navigate inside and outside, in the presence of disturbances, and perform tasks such as aggressively landing on inclined surfaces and locating and grasping an object, using only inexpensive, onboard sensors.     

Flying on point clouds: Online trajectory generation and autonomous navigation for quadrotors in cluttered environments

Micro aerial vehicles (MAVs), especially quadrotors, have been widely used in field applications, such as disaster response, field surveillance, and search‐and‐rescue. For accomplishing such missions in challenging environments, the capability of navigating with full autonomy while avoiding unexpected obstacles is the most crucial requirement. In this paper, we present a framework for online generating safe and dynamically feasible trajectories directly on the point cloud, which is the lowest‐level representation of range measurements and is applicable to different sensor types. We develop a quadrotor platform equipped with a three‐dimensional (3D) light detection and ranging (LiDAR) and an inertial measurement unit (IMU) for simultaneously estimating states of the vehicle and building point cloud maps of the environment. Based on the incrementally registered point clouds, we online generate and refine a flight corridor, which represents the free space that the trajectory of the quadrotor should lie in. We represent the trajectory as piecewise Bézier curves by using the Bernstein polynomial basis and formulate the trajectory generation problem as a convex program. By using Bézier curves, we can constrain the position and kinodynamics of the trajectory entirely within the flight corridor and given physical limits. The proposed approach is implemented to run onboard in real‐time and is integrated into an autonomous quadrotor platform. We demonstrate fully autonomous quadrotor flights in unknown, complex environments to validate the proposed method.     

Development of A Behavior‐Based Cooperative Search Strategy for Distributed Autonomous Mobile Robots Using Zigbee Wireless Sensor Network

To achieve efficient and objective search tasks in an unknown environment, a cooperative search strategy for distributed autonomous mobile robots is developed using a behavior‐based control framework with individual and group behaviors. The sensing information of each mobile robot activates the individual behaviors to facilitate autonomous search tasks to avoid obstacles. An 802.15.4 ZigBee wireless sensor network then activates the group behaviors that enable cooperative search among the mobile robots. An unknown environment is dynamically divided into several sub‐areas according to the locations and sensing data of the autonomous mobile robots. The group behaviors then enable the distributed autonomous mobile robots to scatter and move in the search environment. The developed cooperative search strategy successfully reduces the search time within the test environments by 22.67% (simulation results) and 31.15% (experimental results).     

旋翼空中机器人系统架构及设计

设计并验证了某型旋翼空中机器人的系统架构。整个空中机器人系统由直升机和地面站两部分组成。直升机是空中机器人的主体,可以自主飞行并完成指定任务。地面站用于监控无人直升机的飞行,并实现人机交互等多项功能。此外,地面站还可通过视觉导航系统引导直升机的自主着陆。直升机与地面站之间通过指令数字链路和视频模拟链路进行信息交互和实时通讯。经实际飞行验证,该空中机器人系统具有鲁棒和实时的特点,能实现直升机自主飞行和自主起降功能。     

A distributed architecture for supervision of autonomous multi-robot missions

Realizing long-term autonomous missions involving teams of heterogeneous robots is a challenge. It requires mechanisms to make robots react to disturbances or failures that will arise during the mission, while trying to successfully achieve the mission in cooperation. This paper presents HiDDeN, a distributed deliberative architecture that manages the execution of a hierarchical plan. This plan has initially been computed offline, ensuring some military operational constraints of the mission. Each robot’s supervisor then executes its own part of the plan, and reacts to failures using a hierarchical repair approach. This hierarchical repair has been designed with the sake of ensuring operational constraints, while reducing the need of communication between robots, as communication may be intermittent or even nonexistent when the robots operate in completely separate environments. HiDDeN’s robustness and scalability is evaluated with simulations. Experiments with an autonomous helicopter and an autonomous underwater vehicle have been realized and are presented as the defining point of our contribution.     

Air‐ground Matching: Appearance‐based GPS‐denied Urban Localization of Micro Aerial Vehicles

In this paper, we address the problem of globally localizing and tracking the pose of a camera‐equipped micro aerial vehicle (MAV) flying in urban streets at low altitudes without GPS. An image‐based global positioning system is introduced to localize the MAV with respect to the surrounding buildings. We propose a novel air‐ground image‐matching algorithm to search the airborne image of the MAV within a ground‐level, geotagged image database. Based on the detected matching image features, we infer the global position of the MAV by back‐projecting the corresponding image points onto a cadastral three‐dimensional city model. Furthermore, we describe an algorithm to track the position of the flying vehicle over several frames and to correct the accumulated drift of the visual odometry whenever a good match is detected between the airborne and the ground‐level images. The proposed approach is tested on a 2 km trajectory with a small quadrocopter flying in the streets of Zurich. Our vision‐based global localization can robustly handle extreme changes in viewpoint, illumination, perceptual aliasing, and over‐season variations, thus outperforming conventional visual place‐recognition approaches. The dataset is made publicly available to the research community. To the best of our knowledge, this is the first work that studies and demonstrates global localization and position tracking of a drone in urban streets with a single onboard camera.     

Design and implementation of an autonomous flight control law for a UAV helicopter

In this paper, we present the design and implementation of an autonomous flight control law for a small-scale unmanned aerial vehicle (UAV) helicopter. The approach is decentralized in nature by incorporating a newly developed nonlinear control technique, namely the composite nonlinear feedback control, together with dynamic inversion. The overall control law consists of three hierarchical layers, namely, the kernel control, command generator and flight scheduling, and is implemented and verified in flight tests on the actual UAV helicopter. The flight test results demonstrate that the UAV helicopter is capable of carrying out complicated flight missions autonomously.     

PIXHAWK: A micro aerial vehicle design for autonomous flight using onboard computer vision

We describe a novel quadrotor Micro Air Vehicle (MAV) system that is designed to use computer vision algorithms within the flight control loop. The main contribution is a MAV system that is able to run both the vision-based flight control and stereo-vision-based obstacle detection parallelly on an embedded computer onboard the MAV. The system design features the integration of a powerful onboard computer and the synchronization of IMU-Vision measurements by hardware timestamping which allows tight integration of IMU measurements into the computer vision pipeline. We evaluate the accuracy of marker-based visual pose estimation for flight control and demonstrate marker-based autonomous flight including obstacle detection using stereo vision. We also show the benefits of our IMU-Vision synchronization for egomotion estimation in additional experiments where we use the synchronized measurements for pose estimation using the 2pt+gravity formulation of the PnP problem.     

Team NimbRo at MBZIRC 2017: Autonomous valve stem turning using a wrench

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. In this study, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, grasped, and finally used to turn a valve stem. Mario’s omnidirectional base allows both fast locomotion and precise close approach to the manipulation panel. We describe an efficient detector for medium‐sized objects in three‐dimensional laser scans and apply it to detect the manipulation panel. An object detection architecture based on deep neural networks is used to find and select the correct tool from grayscale images. Parametrized motion primitives are adapted online to percepts of the tool and valve stem to turn the stem. We report in detail on our winning performance at the challenge and discuss lessons learned.     

A Small and Lightweight Autonomous Laser Mapping System without GPS

Autonomous mapping systems execute multiple tasks that include navigation, location, and map generation via the collaborative work of multiple sensors. They are the object of a substantial research focus in the fields of robotics and remote sensing. Although the state‐of‐the‐art mobile mapping systems typically found in ready‐made vehicles or robots are reliable, they are rather large and heavy, their cost is high, and they generally use GPS and an inertial measurement unit to position, so their working environments are limited. After reviewing the current state of autonomous mapping systems, we describe the design and development of a small and lightweight autonomous mapping system (ASQ‐6DMapSys) without GPS, which incorporates low‐cost sensors and components. We describe the layout and selection strategy for sensors and other components in detail, and we present the design methodology for each subsystem. The ASQ‐6DMapSys employs a two‐dimensional (2D) lidar, an inclinometer, and two wheel encoders, which constitute a pose subsystem that uses extended Kalman filtering and simultaneous localization and mapping techniques to compute the pose of the vehicle body. A low‐cost 3D lidar that we developed is also installed on the vehicle body, and the resultant data are aligned with the corresponding pose data of the vehicle body to build a 3D point cloud that describes the global geometry of the environment. We designed and developed every subsystem of the ASQ‐6DMapSys, including the robot vehicle, so it will be easy to expand its functions in the future. The ASQ‐6DMapSys performs well in indoor, outdoor, and tunnel environments, and the experiments in different environments show that the ASQ‐6DMapSys is an effective, small, and lightweight autonomous mapping system with a high performance/price ratio.     

Autonomous Landing of MAVs on an Arbitrarily Textured Landing Site Using Onboard Monocular Vision

This paper presents a novel solution for micro aerial vehicles (MAVs) to autonomously search for and land on an arbitrary landing site using real-time monocular vision. The autonomous MAV is provided with only one single reference image of the landing site with an unknown size before initiating this task. We extend a well-known monocular visual SLAM algorithm that enables autonomous navigation of the MAV in unknown environments, in order to search for such landing sites. Furthermore, a multi-scale ORB feature based method is implemented and integrated into the SLAM framework for landing site detection. We use a RANSAC-based method to locate the landing site within the map of the SLAM system, taking advantage of those map points associated with the detected landing site. We demonstrate the efficiency of the presented vision system in autonomous flights, both indoor and in challenging outdoor environment.     

A lightweight autonomous MAV for indoor search and rescue

Micro Aerial Vehicles (MAVs) have great potentials to be applied for indoor search and rescue missions. In this paper, we propose a modular lightweight design of an autonomous MAV with integrated hardware and software. The MAV is equipped with the 2D laser scanner, camera, mission computer and flight controller, running all the computation onboard in real time. The onboard perception system includes a laser‐based SLAM module and a custom‐designed visual detection module. A dual Kalman filter design provides robust state estimation by multiple sensor fusion. Specifically, the fusion module provides robust altitude measurement in the circumstance of surface changing. In addition, indoor‐outdoor transition is explicitly handled by the fusion module. In order to efficiently navigate through obstacles and adapt to multiple tasks, a task tree‐based mission planning method is seamlessly integrated with path planning and control modules. The MAV is capable of searching and rescuing victims from unknown indoor environments effectively. It was validated by our award‐winning performance at the 2017 International Micro Air Vehicle Competition (IMAV 2017), held in Toulouse, France. The performance video is available on https://youtu.be/8H19ppS_VXM .     

Nonlinear control with integral sliding properties for circular aerial robot trajectory tracking: Real‐time validation

A nonlinear control algorithm for tracking dynamic trajectories using an aerial vehicle is developed in this work. The control structure is designed using a sliding mode methodology, which contains integral sliding properties. The stability analysis of the closed‐loop system is proved using the Lyapunov formalism, ensuring convergence in a desired finite time and robustness toward unknown and external perturbations from the first time instant, even for high frequency disturbances. In addition, a dynamic trajectory is constructed with the translational dynamics of an aerial robot for autonomous take‐off, surveillance missions, and landing. This trajectory respects the constraints imposed by the vehicle characteristics, allowing free initial trajectory conditions. Simulation results demonstrate the good performance of the controller in closed‐loop system when a quadrotor follows the designed trajectory. In addition, flight tests are developed to validate the trajectory and the controller behavior in real time.