Cooperative autonomous search,grasping, and delivering in a treasure hunt scenario by a team of unmanned aerial vehicles

This paper addresses the problem of autonomous cooperative localization, grasping and delivering of colored ferrous objects by a team of unmanned aerial vehicles (UAVs). In the proposed scenario, a team of UAVs is required to maximize the reward by collecting colored objects and delivering them to a predefined location. This task consists of several subtasks such as cooperative coverage path planning, object detection and state estimation, UAV self‐localization, precise motion control, trajectory tracking, aerial grasping and dropping, and decentralized team coordination. The failure recovery and synchronization job manager is used to integrate all the presented subtasks together and also to decrease the vulnerability to individual subtask failures in real‐world conditions. The whole system was developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017, where it achieved the highest score and won Challenge No. 3—Treasure Hunt. This paper does not only contain results from the MBZIRC 2017 competition but it also evaluates the system performance in simulations and field tests that were conducted throughout the year‐long development and preparations for the competition.     

A mission management system for complex aerial logistics by multiple unmanned aerial vehicles in MBZIRC 2017

In this study, we present a system that manages multiple unmanned aerial vehicles (UAVs) for a search, pickup, and drop mission in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Three UAVs picked up and dropped 23 circular and rectangular targets into a designated drop box. To control the operation of three UAVs flying over an arena of 90 × 60 m, we designed and integrated a set of technologies into our system: airspace allocation, communication framework among UAVs, anticollision based on geofencing, and a token‐based prioritization for coordination. The proposed UAV system uses a single GPS and its error of a few meters is solved by means of the following component technologies: (a) flight path generator based on one reference point, (b) vision‐based redefinition of a reference point for GPS correction, and (c) calibration of flight path to update the reference point. The pickup‐and‐drop mission is conducted via color‐ and shape‐based vision processing and a magnetic gripper to pickup and drop‐off the targets. Our proposed system is able to successfully manage three UAVs, recognize targets on the ground, and drop the targets into a drop box in the drop zone. Finally, we achieved fourth place among 18 teams in Challenge 3.     

Al‐Robotics team: A cooperative multi‐unmanned aerial vehicle approach for the Mohamed Bin Zayed International Robotic Challenge

The Al‐Robotics team was selected as one of the 25 finalist teams out of 143 applications received to participate in the first edition of the Mohamed Bin Zayed International Robotic Challenge (MBZIRC), held in 2017. In particular, one of the competition Challenges offered us the opportunity to develop a cooperative approach with multiple unmanned aerial vehicles (UAVs) searching, picking up, and dropping static and moving objects. This paper presents the approach that our team Al‐Robotics followed to address that Challenge 3 of the MBZIRC. First, we overview the overall architecture of the system, with the different modules involved. Second, we describe the procedure that we followed to design the aerial platforms, as well as all their onboard components. Then, we explain the techniques that we used to develop the software functionalities of the system. Finally, we discuss our experimental results and the lessons that we learned before and during the competition. The cooperative approach was validated with fully autonomous missions in experiments previous to the actual competition. We also analyze the results that we obtained during the competition trials.     

Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. This study covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined “Treasure Hunt,” which required mission planning and multirobot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge‐specific extensions, cover the camera‐based perception, explain control and trajectory‐planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real‐robot experiments during the competition in Abu Dhabi. With our system, we—as part of the larger team NimbRo—won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.     

Autonomous landing on a moving vehicle with an unmanned aerial vehicle

This paper addresses the perception, control, and trajectory planning for an aerial platform to identify and land on a moving car at 15 km/hr. The hexacopter unmanned aerial vehicle (UAV), equipped with onboard sensors and a computer, detects the car using a monocular camera and predicts the car future movement using a nonlinear motion model. While following the car, the UAV lands on its roof, and it attaches itself using magnetic legs. The proposed system is fully autonomous from takeoff to landing. Numerous field tests were conducted throughout the year‐long development and preparations for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 competition, for which the system was designed. We propose a novel control system in which a model predictive controller is used in real time to generate a reference trajectory for the UAV, which are then tracked by the nonlinear feedback controller. This combination allows to track predictions of the car motion with minimal position error. The evaluation presents three successful autonomous landings during the MBZIRC 2017, where our system achieved the fastest landing among all competing teams.     

Unsupervised learning‐based flexible framework for surveillance planning with aerial vehicles

The herein studied problem is motivated by practical needs of our participation in the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 in which a team of unmanned aerial vehicles (UAVs) is requested to collect objects in the given area as quickly as possible and score according to the rewards associated with the objects. The mission time is limited, and the most time‐consuming operation is the collection of the objects themselves. Therefore, we address the problem to quickly identify the most valuable objects as surveillance planning with curvature‐constrained trajectories. The problem is formulated as a multivehicle variant of the Dubins traveling salesman problem with neighborhoods (DTSPN). Based on the evaluation of existing approaches to the DTSPN, we propose to use unsupervised learning to find satisfiable solutions with low computational requirements. Moreover, the flexibility of unsupervised learning allows considering trajectory parametrization that better fits the motion constraints of the utilized hexacopters that are not limited by the minimal turning radius as the Dubins vehicle. We propose to use Bézier curves to exploit the maximal vehicle velocity and acceleration limits. Besides, we further generalize the proposed approach to 3D surveillance planning. We report on evaluation results of the developed algorithms and experimental verification of the planned trajectories using the real UAVs utilized in our participation in MBZIRC 2017.     

Experiments in unmanned aerial vehicle/unmanned ground vehicle radiation search

This paper discusses the results of a field experiment conducted at Savannah River National Laboratory to test the performance of several algorithms for the localization of radioactive materials. In this multirobot system, both an unmanned aerial vehicle, a custom hexacopter, and an unmanned ground vehicle (UGV), the ClearPath Jackal, equipped with γ‐ray spectrometers, were used to collect data from two radioactive source configurations. Both the Fourier scattering transform and the Laplacian eigenmap algorithms for source detection were tested on the collected data sets. These algorithms transform raw spectral measurements into alternate spaces to allow clustering to detect trends within the data which indicate the presence of radioactive sources. This study also presents a point source model and accompanying information‐theoretic active exploration algorithm. Field testing validated the ability of this model to fuse aerial and ground collected radiation measurements, and the exploration algorithm’s ability to select informative actions to reduce model uncertainty, allowing the UGV to locate radioactive material online.     

一种双无人机协同跟踪地面移动目标方法

针对通信延时情况下双无人机协同跟踪地面移动目标问题进行研究, 构建了基于分布式遗传算法和滚动时域优化结合的目标跟踪航迹规划算法模型。考虑到通信延时会增加目标状态信息数据融合时的误差, 导致无人机跟踪任务效果变差, 结合递推最小二乘滤波和加权最小二乘估计设计了融合方法, 来融合处理目标状态信息; 考虑到无人机对目标的观测效果与未来时刻的目标状态信息密切相关, 采用递推最小二乘滤波预测目标的状态信息, 结合分布式遗传算法和滚动时域优化设计了双无人机目标跟踪航迹规划算法。适应度函数考虑了无人机和目标之间的距离、无人机之间的通信距离、无人机之间的通信角度。仿真结果表明：该协同跟踪方法能够较好地完成跟踪任务; 与一架无人机跟踪相比误差明显减小, 并且可以减小通信延时带来的跟踪误差。     

Detection of bodies in maritime rescue operations using unmanned aerial vehicles with multispectral cameras

In this study, we use unmanned aerial vehicles equipped with multispectral cameras to search for bodies in maritime rescue operations. A series of flights were performed in open‐water scenarios in the northwest of Spain, using a certified aquatic rescue dummy in dangerous areas and real people when the weather conditions allowed it. The multispectral images were aligned and used to train a convolutional neural network for body detection. An exhaustive evaluation was performed to assess the best combination of spectral channels for this task. Three approaches based on a MobileNet topology were evaluated, using (a) the full image, (b) a sliding window, and (c) a precise localization method. The first method classifies an input image as containing a body or not, the second uses a sliding window to yield a class for each subimage, and the third uses transposed convolutions returning a binary output in which the body pixels are marked. In all cases, the MobileNet architecture was modified by adding custom layers and preprocessing the input to align the multispectral camera channels. Evaluation shows that the proposed methods yield reliable results, obtaining the best classification performance when combining green, red‐edge, and near‐infrared channels. We conclude that the precise localization approach is the most suitable method, obtaining a similar accuracy as the sliding window but achieving a spatial localization close to 1 m. The presented system is about to be implemented for real maritime rescue operations carried out by Babcock Mission Critical Services Spain.     

多无人机模糊多目标分布式地面目标协同追踪

针对城市环境中多约束条件下多无人机协同追踪地面目标问题,综合考虑具有不同重要性等级的多个优化目标,提出了一种基于分布式预测控制的模糊多目标航迹规划方法.首先,考虑城市环境中建筑物对无人机视线遮挡、无人机和传感器能量消耗等因素,分别采用目标覆盖度、控制输入代价和开关量形式传感器能耗等为目标函数,将多无人机协同追踪航迹规划转化为多目标优化问题;然后,基于分布式预测控制框架,利用每架无人机未来有限时域内的预测状态,构建多无人机之间的避碰约束,并结合最小转弯半径等约束,形成分布式协同航迹规划模型;最后,针对多个优化目标的不同重要性等级要求,利用模糊满意优化思想将目标模糊化,并根据更重要目标具有更重要满意度的原则,将优先等级表示为松弛满意度序,通过在线求解得到有限时域内每架无人机的局部航迹;与传统多目标加权算法仿真结果对比,验证了所提方法的有效性,充分说明了该方法能够获得同时满足目标优化和重要性等级要求的最优航迹.     

复杂环境下多无人机协作式地面移动目标跟踪

针对多无人机(UAV)协同地面移动目标跟踪问题展开研究.提出一种基于主动感知的问题求解框架,建立多UAV协同目标跟踪问题模型;在此基础上,采用分布式无色信息滤波实现目标状态融合估计与预测;然后,基于预测目标状态,结合滚动时域控制与遗传算法设计一种多UAV在线协同航迹规划算法.仿真结果表明:结合预测目标状态在线优化UAV...     

Simultaneous sensor selection and routing of unmanned aerial vehicles for complex mission plans

Military reconnaissance missions often employ a set of unmanned aerial vehicles (UAVs) equipped with sensors to gather intelligence information from a set of known targets. UAVs are limited by the number of sensors they can hold; also attaching a sensor adds weight to the aircraft which in turn reduces the flight time available during a mission. The task of optimally assigning sensors to UAVs and routing them through a target field to maximize intelligence gain is a generalization of the team orienteering problem studied in the vehicle routing literature. This work presents a mathematical programming model for simultaneous sensor selection and routing of UAVs, which solves optimally using CPLEX for simple missions. Larger missions required the development of three heuristics, which were augmented by Column Generation. Results from a performance study indicated that the heuristics quickly found good solutions. Column Generation improved the solution in many instances, with minimal impact on overall solution time. The rapid nature of the overall solution approach allows it to be used in other mission planning tasks. A fleet sizing application is discussed as an example of its flexible usage.     

Localization of emergency acoustic sources by micro aerial vehicles

For micro aerial vehicles (MAVs) involved in search and rescue missions, the ability to locate the source of a distress sound signal is significantly important and allows fast localization of victims and rescuers during nighttime, through foliage and in dust, fog, and smoke. Most emergency sound sources, such as safety whistles and personal alarms, generate a narrowband signal that is difficult to localize by human listeners or with the common localization methods suitable for broadband sounds. In this paper, we present three methods for MAV‐based emergency sound localization system. The first method involves designing a new emergency source for immediate localization by the MAV using a common localization method. The other two novel methods allow localizing the currently available emergency sources, or other narrowband sounds in general, that are difficult to localize due to the periodicity in the sequence of sound samples. The second method exploits the Doppler shift in the sound frequency, caused due to the motion of the MAV and the dynamics of the MAV to assist with the localization. The third method involves active control of the robot's attitude and fusing acoustic and attitude measurements for achieving accurate and robust estimates. We evaluate our methods in real‐world experiments with real flying robots.     

Autonomous recovery of a fixed‐wing UAV using a net suspended by two multirotor UAVs

This article presents a novel recovery method for fixed‐wing unmanned aerial vehicles (UAVs), aimed at enabling operations from marine vessels. Instead of using the conventional method of using a fixed net on the ship deck, we propose to suspend a net under two cooperative multirotor UAVs. While keeping their relative formation, the multirotor UAVs are able to intercept the incoming fixed‐wing UAV along a virtual runway over the sea and transport it back to the ship. In addition to discussing the concept and design a control system, this paper also presents experimental validation of the proposed concept for a small‐scale UAV platform.     

Cooperative digital magnetic‐elevation maps by small autonomous aerial robots

One of the steps to provide fundamental data for planning a mining effort is the magnetic surveying of a target area, which is typically carried out by conventional aircraft campaigns. However, besides the high cost, fixed‐wing aerial vehicles present shortcomings especially for drape flights on mountainous regions, where steep slopes are often present. Traditional human‐crewed flights have to perform tedious and dangerous trajectories, under strict velocity and attitude constraints. In this paper, we deal with the problem of accomplishing digital magnetic‐elevation maps using autonomous and cooperative aerial robots. The proposed approach for autonomous mapping utilizes a custom‐built fluxgate sensor and off the shelf cameras adapted for small airborne platforms. We also propose an innovative approach for generating a digital magnetic‐elevation model from the gathered data. Our method was evaluated and validated in field tests in an industrial scenario to detect scrap metals in ore piles. Results show that the proposed method could reliably detect magnetic anomalies while generating accurate three‐dimensional magnetic maps.     

Antenna array synthesis with clusters of unmanned aerial vehicles

In this paper, we investigate the use of formations of Unmanned Aerial Vehicles (UAVs) as phased antenna arrays. This will help to improve communications with clusters of small unmanned aerial vehicles which are currently constrained by on-board power limitations. The problem of maximizing the power output from the array in the direction of the receiver is posed as an optimization problem which happens to be non-convex; a relaxation of this problem is then solved as a computationally tractable (convex) Second-Order Conic Program (SOCP). The performance obtained by the simplified approach is then tested against rigorous numerical bounds obtained using Semidefinite Programming (SDP) duality theory; these bounds are of independent interest in antenna theory. In order to maintain the objective value close to the optimal when the vehicles deviate from their positions (due to wind gusts, for example), a simple linear control law is proposed. Simulation results are given to show the effectiveness of the proposed approaches.     

未知复杂环境中的无人机平滑飞行路径规划

针对无人机实时路径规划问题,提出了一种基于双层决策的平滑路径规划方法,以弥补现有方法在复杂飞行环境中对路径平滑性优化的不足,增强路径的易跟踪性.本文首先给出路径平滑性度量,然后建模上、下层决策目标、威胁规避与无人机性能约束并引入变长规划时间,进而设计基于双层决策的路径规划模型.规划过程中通过嵌入启发式优化策略来进一步改善路径的全局与局部平滑度,并提高路径搜索效率.大量复杂场景中的仿真及与现有经典方法的对比结果表明:该方法能够实时避开复杂危险区域,规划适合飞行的、较短的平滑路径.     

无人机非确定性等价自适应容错飞行控制

针对存在执行器复合故障的固定翼无人机跟踪控制问题,本文提出一种基于非确定性等价原理的自适应容错飞行控制策略.该策略能够有效地估计无人机纵向动态中执行器的失效及漂移故障,保证故障发生后闭环系统的最优性能指标.在自适应容错飞行控制设计中,通过引入辅助系统并动态调节因子,构造非确定性等价原理中偏微分方程的近似解,以简化自适应律设计复杂度.此外,借助Lyapunov稳定性分析方法,证明了在所设计的自适应容错控制器作用下闭环系统的稳定性.最后,仿真验证表明所设计的控制方法能够保证故障无人机的闭环系统性能.     

Vision techniques for on‐board detection,following, and mapping of moving targets

This study presents computer vision modules of a multi‐unmanned aerial vehicle (UAV) system, which scored gold, silver, and bronze medals at the Mohamed Bin Zayed International Robotics Challenge 2017. This autonomous system, which was running completely on board and in real time, had to address two complex tasks in challenging outdoor conditions. In the first task, an autonomous UAV had to find, track, and land on a human‐driven car moving at 15 km/hr on a figure‐eight‐shaped track. During the second task, a group of three UAVs had to find small colored objects in a wide area, pick them up, and deliver them into a specified drop‐off zone. The computer vision modules presented here achieved computationally efficient detection, accurate localization, robust velocity estimation, and reliable future position prediction of both the colored objects and the car. These properties had to be achieved in adverse outdoor environments with changing light conditions. Lighting varied from intense direct sunlight with sharp shadows cast over the objects by the UAV itself, to reduced visibility caused by overcast to dust and sand in the air. The results presented in this paper demonstrate good performance of the modules both during testing, which took place in the harsh desert environment of the central area of United Arab Emirates, as well as during the contest, which took place at a racing complex in the urban, near‐sea location of Abu Dhabi. The stability and reliability of these modules contributed to the overall result of the contest, where our multi‐UAV system outperformed teams from world’s leading robotic laboratories in two challenging scenarios.     

Nonlinear path-following method for fixed-wing unmanned aerial vehicles

A path-following method for fixed-wing unmanned aerial vehicles(UAVs) is presented in this paper.This method consists of an outer guidance loop and an inner control loop.The guidance law relies on the idea of tracking a virtual target.The motion of the virtual target is explicitly specified.The main advantage of this guidance law is that it considers the maneuvering ability of the aircraft.The aircraft can asymptotically approach the defined path with smooth movements.Meanwhile,the aircraft can anticipate the upcoming transition of the flight path.Moreover,the inner adaptive flight control loop based on attractive manifolds can follow the command generated by the outer guidance loop.This adaptive control law introduces a first-order filter to avoid solving the partial differential equation in the immersion and invariance adaptive control.The performance of the proposed path-following method is validated by the numerical simulation.