Robotic technologies for solar‐powered UAVs: Fully autonomous updraft‐aware aerial sensing for multiday search‐and‐rescue missions

Large‐scale aerial sensing missions can greatly benefit from the perpetual endurance capability provided by high‐performance low‐altitude solar‐powered unmanned aerial vehicles (UAVs). However, today these UAVs suffer from small payload capacity, low energetic margins, and high operational complexity. To tackle these problems, this paper presents four individual technical contributions and integrates them into an existing solar‐powered UAV system: First, a lightweight and power‐efficient day/night‐capable sensing system is discussed. Second, means to optimize the UAV platform to the specific payload and to thereby achieve sufficient energetic margins for day/night flight with payload are presented. Third, existing autonomous launch and landing functionality is extended for solar‐powered UAVs. Fourth, as a main contribution an extended Kalman filter (EKF)‐based autonomous thermal updraft tracking framework is developed. Its novelty is that it allows the end‐to‐end integration of the thermal‐induced roll moment into the estimation process. It is assessed against unscented Kalman filter and particle filter methods in simulation and implemented on the aircraft's low‐power autopilot. The complete system is verified during a 26 h search‐and‐rescue aerial sensing mock‐up mission that represents the first‐ever fully autonomous perpetual endurance flight of a small solar‐powered UAV with a day/night‐capable sensing payload. It also represents the first time that solar‐electric propulsion and autonomous thermal updraft tracking are combined in flight. In contrast to previous work that has focused on the energetic feasibility of perpetual flight, the individual technical contributions of this paper are considered core functionality to guarantee ease‐of‐use, effectivity, and reliability in future multiday aerial sensing operations with small solar‐powered UAVs.     

Autonomous Navigation for Micro Aerial Vehicles in Complex GNSS-denied Environments

Micro aerial vehicles, such as multirotors, are particular well suited for the autonomous monitoring, inspection, and surveillance of buildings, e.g., for maintenance in industrial plants. Key prerequisites for the fully autonomous operation of micro aerial vehicles in restricted environments are 3D mapping, real-time pose tracking, obstacle detection, and planning of collision-free trajectories. In this article, we propose a complete navigation system with a multimodal sensor setup for omnidirectional environment perception. Measurements of a 3D laser scanner are aggregated in egocentric local multiresolution grid maps. Local maps are registered and merged to allocentric maps in which the MAV localizes. For autonomous navigation, we generate trajectories in a multi-layered approach: from mission planning over global and local trajectory planning to reactive obstacle avoidance. We evaluate our approach in a GNSS-denied indoor environment where multiple collision hazards require reliable omnidirectional perception and quick navigation reactions.     

An open‐source system for vision‐based micro‐aerial vehicle mapping,planning, and flight in cluttered environments

We present an open‐source system for Micro‐Aerial Vehicle (MAV) autonomous navigation from vision‐based sensing. Our system focuses on dense mapping, safe local planning, and global trajectory generation, especially when using narrow field‐of‐view sensors in very cluttered environments. In addition, details about other necessary parts of the system and special considerations for applications in real‐world scenarios are presented. We focus our experiments on evaluating global planning, path smoothing, and local planning methods on real maps made on MAVs in realistic search‐and‐rescue and industrial inspection scenarios. We also perform thousands of simulations in cluttered synthetic environments, and finally validate the complete system in real‐world experiments.     

Cooperative digital magnetic‐elevation maps by small autonomous aerial robots

One of the steps to provide fundamental data for planning a mining effort is the magnetic surveying of a target area, which is typically carried out by conventional aircraft campaigns. However, besides the high cost, fixed‐wing aerial vehicles present shortcomings especially for drape flights on mountainous regions, where steep slopes are often present. Traditional human‐crewed flights have to perform tedious and dangerous trajectories, under strict velocity and attitude constraints. In this paper, we deal with the problem of accomplishing digital magnetic‐elevation maps using autonomous and cooperative aerial robots. The proposed approach for autonomous mapping utilizes a custom‐built fluxgate sensor and off the shelf cameras adapted for small airborne platforms. We also propose an innovative approach for generating a digital magnetic‐elevation model from the gathered data. Our method was evaluated and validated in field tests in an industrial scenario to detect scrap metals in ore piles. Results show that the proposed method could reliably detect magnetic anomalies while generating accurate three‐dimensional magnetic maps.     

A distributed architecture for a robotic platform with aerial sensor transportation and self‐deployment capabilities

This paper presents the architecture developed in the framework of the AWARE project for the autonomous distributed cooperation between unmanned aerial vehicles (UAVs), wireless sensor/actuator networks, and ground camera networks. One of the main goals was the demonstration of useful actuation capabilities involving multiple ground and aerial robots in the context of civil applications. A novel characteristic is the demonstration in field experiments of the transportation and deployment of the same load with single/multiple autonomous aerial vehicles. The architecture is endowed with different modules that solve the usual problems that arise during the execution of multipurpose missions, such as task allocation, conflict resolution, task decomposition, and sensor data fusion. The approach had to satisfy two main requirements: robustness for operation in disaster management scenarios and easy integration of different autonomous vehicles. The former specification led to a distributed design, and the latter was tackled by imposing several requirements on the execution capabilities of the vehicles to be integrated in the platform. The full approach was validated in field experiments with different autonomous helicopters equipped with heterogeneous devices onboard, such as visual/infrared cameras and instruments to transport loads and to deploy sensors. Four different missions are presented in this paper: sensor deployment and fire confirmation with UAVs, surveillance with multiple UAVs, tracking of firemen with ground and aerial sensors/cameras, and load transportation with multiple UAVs. © 2011 Wiley Periodicals, Inc.     

Multi-target indoor localization and tracking on video monitoring system in a wireless sensor network

The development of wireless sensor networks (WSNs) has greatly encouraged the use of sensors for multi-target tracking. The high efficiency detection and location monitoring are critical requirements for multi-target tracking in a WSN. In this paper, we present an indoor tracking model using IEEE 802.15.4 compliant radio frequency and video monitoring system to monitor targets in a special way. Our motivation is to manipulate the erratic or unstable received signal strength indicator (RSSI) signals to deliver the stable and precise position information in the indoor environment. We propose a localization algorithm based on statistical uncorrelated vectors and develop a smoothing algorithm to minimize the noise in RSSI values. We also present a solution combining the WSN with the Ethernet technology to decrease the RSSI interference by buildings. The developed system can realize the functions of multi-target detection and tracking, and specific target inquiries, alarms and monitoring. The system architecture, hardware and software organization, as well as the solutions for multiple targets tracking, RSSI interference and localization accuracy have been introduced in details.     

Autonomous science during large‐scale robotic survey

Today's planetary exploration robots rarely travel beyond the yesterday imagery. However, advances in autonomous mobility will soon permit single‐command site surveys of multiple kilometers. Here scientists cannot see the terrain in advance, and explorer robots must navigate and collect data autonomously. Onboard science data understanding can improve these surveys with image analysis, pattern recognition, learned classification, and information‐theoretic planning. We report on field experiments near Amboy Crater, California, that demonstrate fundamental capabilities for autonomous surficial mapping of geologic phenomena with a visible near‐infrared spectrometer. We develop an approach to “science on the fly'' that adapts the robot's exploration using collected instrument data. We demonstrate feature detection and visual servoing to acquire spectra from dozens of targets without human intervention. The rover interprets spectra onboard, learning spatial models of science phenomena that guide it toward informative areas. It discovers spatial structure (correlations between neighboring regions) and cross‐sensor structure (correlations between different scales). The rover uses surface observations to reinterpret satellite imagery and improve exploration efficiency. © 2011 Wiley Periodicals, Inc.     

Communication‐Aware Information‐Gathering Experiments with an Unmanned Aircraft System

This paper presents single and multiaircraft flight experiments to assess an information‐theoretic path planning algorithm that incorporates sensing and communication to guide an unmanned aircraft. The communication is modeled with packet erasure channels for each link in a multihop mesh network. The planning objective is to maximize the mutual information between the target state and measurements received at a single base station from all aircraft. A novel unmanned aircraft system was developed to facilitate experiments with multiple unmanned aircraft utilizing multihop mesh networking. The value of communication‐aware planning was assessed through flight experiments with a single aircraft localizing a radio emitter. Additional experiments with two aircraft demonstrated and assessed the performance of the approach, showing that the improvement in sensing can be appreciable when utilizing multihop communication.     

Cooperative autonomous search,grasping, and delivering in a treasure hunt scenario by a team of unmanned aerial vehicles

This paper addresses the problem of autonomous cooperative localization, grasping and delivering of colored ferrous objects by a team of unmanned aerial vehicles (UAVs). In the proposed scenario, a team of UAVs is required to maximize the reward by collecting colored objects and delivering them to a predefined location. This task consists of several subtasks such as cooperative coverage path planning, object detection and state estimation, UAV self‐localization, precise motion control, trajectory tracking, aerial grasping and dropping, and decentralized team coordination. The failure recovery and synchronization job manager is used to integrate all the presented subtasks together and also to decrease the vulnerability to individual subtask failures in real‐world conditions. The whole system was developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017, where it achieved the highest score and won Challenge No. 3—Treasure Hunt. This paper does not only contain results from the MBZIRC 2017 competition but it also evaluates the system performance in simulations and field tests that were conducted throughout the year‐long development and preparations for the competition.     

Al‐Robotics team: A cooperative multi‐unmanned aerial vehicle approach for the Mohamed Bin Zayed International Robotic Challenge

The Al‐Robotics team was selected as one of the 25 finalist teams out of 143 applications received to participate in the first edition of the Mohamed Bin Zayed International Robotic Challenge (MBZIRC), held in 2017. In particular, one of the competition Challenges offered us the opportunity to develop a cooperative approach with multiple unmanned aerial vehicles (UAVs) searching, picking up, and dropping static and moving objects. This paper presents the approach that our team Al‐Robotics followed to address that Challenge 3 of the MBZIRC. First, we overview the overall architecture of the system, with the different modules involved. Second, we describe the procedure that we followed to design the aerial platforms, as well as all their onboard components. Then, we explain the techniques that we used to develop the software functionalities of the system. Finally, we discuss our experimental results and the lessons that we learned before and during the competition. The cooperative approach was validated with fully autonomous missions in experiments previous to the actual competition. We also analyze the results that we obtained during the competition trials.     

Adaptive Image‐Space Sampling for Gaze‐Contingent Real‐time Rendering

With ever‐increasing display resolution for wide field‐of‐view displays—such as head‐mounted displays or 8k projectors—shading has become the major computational cost in rasterization. To reduce computational effort, we propose an algorithm that only shades visible features of the image while cost‐effectively interpolating the remaining features without affecting perceived quality. In contrast to previous approaches we do not only simulate acuity falloff but also introduce a sampling scheme that incorporates multiple aspects of the human visual system: acuity, eye motion, contrast (stemming from geometry, material or lighting properties), and brightness adaptation. Our sampling scheme is incorporated into a deferred shading pipeline to shade the image's perceptually relevant fragments while a pull‐push algorithm interpolates the radiance for the rest of the image. Our approach does not impose any restrictions on the performed shading. We conduct a number of psycho‐visual experiments to validate scene‐ and task‐independence of our approach. The number of fragments that need to be shaded is reduced by 50 % to 80 %. Our algorithm scales favorably with increasing resolution and field‐of‐view, rendering it well‐suited for head‐mounted displays and wide‐field‐of‐view projection.     

Adaptive continuous‐space informative path planning for online environmental monitoring

Autonomous mobile robots are increasingly employed to take measurements for environmental monitoring, but planning informative, measurement‐rich paths through large three‐dimensional environments is still challenging. Designing such paths, known as the informative path planning (IPP) problem, has been shown to be NP‐hard. Existing algorithms focus on providing guarantees on suboptimal solutions, but do not scale well to large problems. In this paper, we introduce a novel IPP algorithm that uses an evolutionary strategy to optimize a parameterized path in continuous space, which is subject to various constraints regarding path budgets and motion capabilities of an autonomous mobile robot. Moreover, we introduce a replanning scheme to adapt the planned paths according to the measurements taken in situ during data collection. When compared to two state‐of‐the‐art solutions, our method provides competitive results at significantly lower computation times and memory requirements. The proposed replanning scheme enables to build models with up to 25% lower uncertainty within an initially unknown area of interest. Besides presenting theoretical results, we tailored the proposed algorithms for data collection using an autonomous surface vessel for an ecological study, during which the method was validated through three field deployments on Lake Zurich, Switzerland. Spatiotemporal variations are shown over a period of three months and in an area of 350 m × 350 m × 13 m. Whereas our theoretical solution can be applied to multiple applications, our field results specifically highlight the effectiveness of our planner for monitoring toxic microorganisms in a pre‐alpine lake, and for identifying hot‐spots within their distribution.     

A method for protocol‐based collision avoidance between autonomous marine surface craft

This paper is concerned with the in‐field autonomous operation of unmanned marine vehicles in accordance with convention for safe and proper collision avoidance as prescribed by the Coast Guard Collision Regulations (COLREGS). These rules are written to train and guide safe human operation of marine vehicles and are heavily dependent on human common sense in determining rule applicability as well as rule execution, especially when multiple rules apply simultaneously. To capture, the flexibility exploited by humans, this work applies a novel method of multiobjective optimization, interval programming, in a behavior‐based control framework for representing the navigation rules, as well as task behaviors, in a way that achieves simultaneous optimal satisfaction. We present experimental validation of this approach using multiple autonomous surface craft. This work represents the first in‐field demonstration of multiobjective optimization applied to autonomous COLREGS‐based marine vehicle navigation. © 2006 Wiley Periodicals, Inc.     

A Visualization‐Based Analysis System for Urban Search & Rescue Mission Planning Support

We propose a visualization system for incident commanders (ICs) in urban search and rescue scenarios that supports path planning in post‐disaster structures. Utilizing point cloud data acquired from unmanned robots, we provide methods for the assessment of automatically generated paths. As data uncertainty and a priori unknown information make fully automated systems impractical, we present the IC with a set of viable access paths, based on varying risk factors, in a 3D environment combined with visual analysis tools enabling informed decision making and trade‐offs. Based on these decisions, a responder is guided along the path by the IC, who can interactively annotate and reevaluate the acquired point cloud and generated paths to react to the dynamics of the situation. We describe visualization design considerations for our system and decision support systems in general, technical realizations of the visualization components, and discuss the results of two qualitative expert evaluation; one online study with nine search and rescue experts and an eye‐tracking study in which four experts used the system on an application case.     

Decomposition‐based mission planning for fixed‐wing UAVs surveying in wind

This paper presents a new method for planning fixed‐wing aerial survey paths that ensures efficient image coverage of a large complex agricultural field in the presence of wind. By decomposing any complex polygonal field into multiple convex polygons, the traditional back‐and‐forth boustrophedon paths can be used to ensure coverage of these decomposed regions. To decompose a complex field in an efficient and fast manner, a top‐down recursive greedy approach is used to traverse the search space to minimize the flight time of the survey. This optimization can be computed fast enough for use in the field. As wind can severely affect flight time, it is included in the flight time calculation in a systematic way using a verified cost function that offers greatly reduced survey times in the wind. Other improved cost functions have been developed to take into account real‐world problems, for example, No‐Fly Zones, in addition to flight time. A number of real surveys are performed to show the flight time in wind model is accurate, to make further comparisons to previous techniques and to show that the proposed method works in real‐world conditions providing total image coverage. A number of missions are generated and flown for real complex agricultural fields. In addition to this, the wind field around a survey area is measured from a multirotor carrying an ultrasonic wind speed sensor. This shows that the assumption of steady uniform wind holds true for the small areas and time scales of an unmanned aerial vehicle aerial survey.     

Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. This study covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined “Treasure Hunt,” which required mission planning and multirobot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge‐specific extensions, cover the camera‐based perception, explain control and trajectory‐planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real‐robot experiments during the competition in Abu Dhabi. With our system, we—as part of the larger team NimbRo—won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.     

Autonomous UAV path planning and estimation

Unmanned aerial vehicles (UAVs) have shown promise in recent years for autonomous sensing. UAVs systems have been proposed for a wide range of applications such as mapping, surveillance, search, and tracking operations. The recent availability of low-cost UAVs suggests the use of teams of vehicles to perform sensing tasks. To leverage the capabilities of a team of vehicles, efficient methods of decentralized sensing and cooperative path planning are necessary. The goal of this work is to examine practical control strategies for a team of fixed-wing vehicles performing cooperative sensing. We seek to develop decentralized, autonomous control strategies that can account for a wide variety of sensing missions. Sensing goals are posed from an information theoretic standpoint to design strategies that explicitly minimize uncertainty. This work proposes a tightly coupled approach, in which sensor models and estimation objectives are used online for path planning.     

Continuous Airborne Communication Relay Approach Using Unmanned Aerial Vehicles

As a result of unmanned aerial vehicles being widely used in different areas, studies about increasing the autonomous capabilities of unmanned aerial vehicles are gaining momentum. Today, unmanned aerial vehicle platforms are especially used in reconnaissance, surveillance and communications areas. In this study, in order to achieve continuous long-range communication relay infrastructure, artificial potential field based path planning of Unmanned Aerial Vehicles is discussed. A novel dynamic approach to relay-chain concept is proposed to maintain the communication between vehicles. Besides dynamically keeping vehicles in range and appropriate position to maintain communication relay, artificial potential field based path planning also provides collision avoidance system. The performance of the proposed system is measured by applying a simulation under the Matlab Simulink and Network Simulator environment. Artificial potential field based flight patterns are generated in Matlab, and performance of the communication between vehicles is measured in Network Simulation environment. Finally the simulation results show that an airborne communication relay can be established autonomously by using artificial potential filed based autonomous path planning approach. Continues state communication is provided by obtaining a resistant communication relay which depends on artificial potential field based positioning algorithm.     

Robust distributed active power control technique for IEEE 802.15.4 wireless sensor networks — A quantitative feedback theory approach

A novel, practically implementable robust Distributed Active Power Control (DAPC) technique is presented for IEEE 802.15.4 wireless sensor networks by using Quantitative Feedback Theory (QFT). The proposed DAPC framework is based on tracking a target Received Signal Strength Indicator (RSSI) where the effects of radio channel uncertainty and interference between sensor nodes are considered as an unknown output disturbance. The key features of this technique are summarized as follows: (1) quantifiable improvements are achieved in terms of outage probability and power consumption, (2) exact information in relation to the network operating conditions, e.g., radio channels gains and interference between users, is not required, and (3) the proposed graphical design environment simplifies the trade-off between system performance metrics. Experimental results are provided that highlight the effectiveness of the proposed approach when compared with some existing DAPC techniques.