Al‐Robotics team: A cooperative multi‐unmanned aerial vehicle approach for the Mohamed Bin Zayed International Robotic Challenge

The Al‐Robotics team was selected as one of the 25 finalist teams out of 143 applications received to participate in the first edition of the Mohamed Bin Zayed International Robotic Challenge (MBZIRC), held in 2017. In particular, one of the competition Challenges offered us the opportunity to develop a cooperative approach with multiple unmanned aerial vehicles (UAVs) searching, picking up, and dropping static and moving objects. This paper presents the approach that our team Al‐Robotics followed to address that Challenge 3 of the MBZIRC. First, we overview the overall architecture of the system, with the different modules involved. Second, we describe the procedure that we followed to design the aerial platforms, as well as all their onboard components. Then, we explain the techniques that we used to develop the software functionalities of the system. Finally, we discuss our experimental results and the lessons that we learned before and during the competition. The cooperative approach was validated with fully autonomous missions in experiments previous to the actual competition. We also analyze the results that we obtained during the competition trials.     

Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. This study covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined “Treasure Hunt,” which required mission planning and multirobot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge‐specific extensions, cover the camera‐based perception, explain control and trajectory‐planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real‐robot experiments during the competition in Abu Dhabi. With our system, we—as part of the larger team NimbRo—won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.     

Unsupervised learning‐based flexible framework for surveillance planning with aerial vehicles

The herein studied problem is motivated by practical needs of our participation in the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 in which a team of unmanned aerial vehicles (UAVs) is requested to collect objects in the given area as quickly as possible and score according to the rewards associated with the objects. The mission time is limited, and the most time‐consuming operation is the collection of the objects themselves. Therefore, we address the problem to quickly identify the most valuable objects as surveillance planning with curvature‐constrained trajectories. The problem is formulated as a multivehicle variant of the Dubins traveling salesman problem with neighborhoods (DTSPN). Based on the evaluation of existing approaches to the DTSPN, we propose to use unsupervised learning to find satisfiable solutions with low computational requirements. Moreover, the flexibility of unsupervised learning allows considering trajectory parametrization that better fits the motion constraints of the utilized hexacopters that are not limited by the minimal turning radius as the Dubins vehicle. We propose to use Bézier curves to exploit the maximal vehicle velocity and acceleration limits. Besides, we further generalize the proposed approach to 3D surveillance planning. We report on evaluation results of the developed algorithms and experimental verification of the planned trajectories using the real UAVs utilized in our participation in MBZIRC 2017.     

Vision techniques for on‐board detection,following, and mapping of moving targets

This study presents computer vision modules of a multi‐unmanned aerial vehicle (UAV) system, which scored gold, silver, and bronze medals at the Mohamed Bin Zayed International Robotics Challenge 2017. This autonomous system, which was running completely on board and in real time, had to address two complex tasks in challenging outdoor conditions. In the first task, an autonomous UAV had to find, track, and land on a human‐driven car moving at 15 km/hr on a figure‐eight‐shaped track. During the second task, a group of three UAVs had to find small colored objects in a wide area, pick them up, and deliver them into a specified drop‐off zone. The computer vision modules presented here achieved computationally efficient detection, accurate localization, robust velocity estimation, and reliable future position prediction of both the colored objects and the car. These properties had to be achieved in adverse outdoor environments with changing light conditions. Lighting varied from intense direct sunlight with sharp shadows cast over the objects by the UAV itself, to reduced visibility caused by overcast to dust and sand in the air. The results presented in this paper demonstrate good performance of the modules both during testing, which took place in the harsh desert environment of the central area of United Arab Emirates, as well as during the contest, which took place at a racing complex in the urban, near‐sea location of Abu Dhabi. The stability and reliability of these modules contributed to the overall result of the contest, where our multi‐UAV system outperformed teams from world’s leading robotic laboratories in two challenging scenarios.     

Autonomous landing on a moving vehicle with an unmanned aerial vehicle

This paper addresses the perception, control, and trajectory planning for an aerial platform to identify and land on a moving car at 15 km/hr. The hexacopter unmanned aerial vehicle (UAV), equipped with onboard sensors and a computer, detects the car using a monocular camera and predicts the car future movement using a nonlinear motion model. While following the car, the UAV lands on its roof, and it attaches itself using magnetic legs. The proposed system is fully autonomous from takeoff to landing. Numerous field tests were conducted throughout the year‐long development and preparations for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 competition, for which the system was designed. We propose a novel control system in which a model predictive controller is used in real time to generate a reference trajectory for the UAV, which are then tracked by the nonlinear feedback controller. This combination allows to track predictions of the car motion with minimal position error. The evaluation presents three successful autonomous landings during the MBZIRC 2017, where our system achieved the fastest landing among all competing teams.     

A mission management system for complex aerial logistics by multiple unmanned aerial vehicles in MBZIRC 2017

In this study, we present a system that manages multiple unmanned aerial vehicles (UAVs) for a search, pickup, and drop mission in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Three UAVs picked up and dropped 23 circular and rectangular targets into a designated drop box. To control the operation of three UAVs flying over an arena of 90 × 60 m, we designed and integrated a set of technologies into our system: airspace allocation, communication framework among UAVs, anticollision based on geofencing, and a token‐based prioritization for coordination. The proposed UAV system uses a single GPS and its error of a few meters is solved by means of the following component technologies: (a) flight path generator based on one reference point, (b) vision‐based redefinition of a reference point for GPS correction, and (c) calibration of flight path to update the reference point. The pickup‐and‐drop mission is conducted via color‐ and shape‐based vision processing and a magnetic gripper to pickup and drop‐off the targets. Our proposed system is able to successfully manage three UAVs, recognize targets on the ground, and drop the targets into a drop box in the drop zone. Finally, we achieved fourth place among 18 teams in Challenge 3.     

A general‐purpose task execution framework for manipulation mission of the 2017 Mohamed Bin Zayed International Robotics Challenge

This paper presents the hardware and software of our team's EurecarBot for Challenge 2 in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Fully automating our robots actions in a real environment required many component technologies for manipulation and vision processing. To perform the complex robotic missions, we developed a task execution framework, which provides a high‐level interface to specify the given tasks. In this study, we focus on the valve operation problem, which was the hardest part of the competition. We also discuss how we overcame the various problems caused by differences between the experimental and the actual competition environments. EurecarBot completed the valve operation mission perfectly in the MBZIRC Grand Challenge and ranked fourth in Challenge 2 and fifth in the Grand Challenge.     

Fully autonomous micro air vehicle flight and landing on a moving target using visual–inertial estimation and model‐predictive control

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) held in spring 2017 was a very successful competition well attended by teams from all over the world. One of the challenges (Challenge 1) required an aerial robot to detect, follow, and land on a moving target in a fully autonomous fashion. In this paper, we present the hardware components of the micro air vehicle (MAV) we built with off the self components alongside the designed algorithms that were developed for the purposes of the competition. We tackle the challenge of landing on a moving target by adopting a generic approach, rather than following one that is tailored to the MBZIRC Challenge 1 setup, enabling easy adaptation to a wider range of applications and targets, even indoors, since we do not rely on availability of global positioning system. We evaluate our system in an uncontrolled outdoor environment where our MAV successfully and consistently lands on a target moving at a speed of up to 5.0 m/s.     

Autonomous recovery of a fixed‐wing UAV using a net suspended by two multirotor UAVs

This article presents a novel recovery method for fixed‐wing unmanned aerial vehicles (UAVs), aimed at enabling operations from marine vessels. Instead of using the conventional method of using a fixed net on the ship deck, we propose to suspend a net under two cooperative multirotor UAVs. While keeping their relative formation, the multirotor UAVs are able to intercept the incoming fixed‐wing UAV along a virtual runway over the sea and transport it back to the ship. In addition to discussing the concept and design a control system, this paper also presents experimental validation of the proposed concept for a small‐scale UAV platform.     

Team NimbRo at MBZIRC 2017: Autonomous valve stem turning using a wrench

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. In this study, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, grasped, and finally used to turn a valve stem. Mario’s omnidirectional base allows both fast locomotion and precise close approach to the manipulation panel. We describe an efficient detector for medium‐sized objects in three‐dimensional laser scans and apply it to detect the manipulation panel. An object detection architecture based on deep neural networks is used to find and select the correct tool from grayscale images. Parametrized motion primitives are adapted online to percepts of the tool and valve stem to turn the stem. We report in detail on our winning performance at the challenge and discuss lessons learned.     

The ETH‐MAV Team in the MBZ International Robotics Challenge

This study describes the hardware and software systems of the Micro Aerial Vehicle (MAV) platforms used by the ETH Zurich team in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). The aim was to develop robust outdoor platforms with the autonomous capabilities required for the competition, by applying and integrating knowledge from various fields, including computer vision, sensor fusion, optimal control, and probabilistic robotics. This paper presents the major components and structures of the system architectures and reports on experimental findings for the MAV‐based challenges in the competition. Main highlights include securing the second place both in the individual search, pick, and place the task of Challenge 3 and the Grand Challenge, with autonomous landing executed in less than 1 min and a visual servoing success rate of over for object pickups.     

Perception,navigation, and manipulation in the team KAUST approach to the MBZIRC ground robotics challenge

The ground robotics challenge in the Mohammed Bin Zayed International Robotics Challenge required a ground vehicle equipped with a robotic arm to autonomously locate a panel, select a proper size wrench among several options mounted on the panel, and use the wrench to rotate a valve. Autonomy was the critical factor in this challenge, which required the teams to devise algorithms that can operate successfully in a semistructured environment without human supervision. This paper presents the approaches taken by team KAUST to meet this challenge, ranging from in‐house hardware designs to algorithm integration and customization. We separated the whole objective into three interconnected tasks: Navigation, perception, and manipulation. For the navigation task, we developed a basic robotic exploration scheme to find the panel front side where the wrenches were present. For the perception task, we integrated common object detection algorithms with neural networks to identify the proper size wrench precisely. For successful manipulation, we designed and built a custom gripper, which was inspired by the common grasping behavior of a human hand under tight clearance conditions. The modular structure of the proposed approach allowed the team to progress in several subtasks simultaneously. However, the interconnection between the subtasks necessitated a reliable integration framework between these modules for effective implementation. We tuned our algorithms in extensive experimental studies and eventually obtained 10 consecutive successful navigation runs, 96% true wrench detection rate, and high success rate in wrench grasping. Furthermore, successful complete tests proved the reliability and repeatability of our system.     

SOFT‐SLAM: Computationally efficient stereo visual simultaneous localization and mapping for autonomous unmanned aerial vehicles

Autonomous navigation of unmanned aerial vehicles (UAVs) in GPS‐denied environments is a challenging problem, especially for small‐scale UAVs characterized by a small payload and limited battery autonomy. A possible solution to the aforementioned problem is vision‐based simultaneous localization and mapping (SLAM), since cameras, due to their dimensions, low weight, availability, and large information bandwidth, circumvent all the constraints of UAVs. In this paper, we propose a stereo vision SLAM yielding very accurate localization and a dense map of the environment developed with the aim to compete in the European Robotics Challenges (EuRoC) targeting airborne inspection of industrial facilities with small‐scale UAVs. The proposed approach consists of a novel stereo odometry algorithm relying on feature tracking (SOFT), which currently ranks first among all stereo methods on the KITTI dataset. Relying on SOFT for pose estimation, we build a feature‐based pose graph SLAM solution, which we dub SOFT‐SLAM. SOFT‐SLAM has a completely separate odometry and mapping threads supporting large loop‐closing and global consistency. It also achieves a constant‐time execution rate of 20 Hz with deterministic results using only two threads of an onboard computer used in the challenge. The UAV running our SLAM algorithm obtained the highest localization score in the EuRoC Challenge 3, Stage IIa–Benchmarking, Task 2. Furthermore, we also present an exhaustive evaluation of SOFT‐SLAM on two popular public datasets, and we compare it to other state‐of‐the‐art approaches, namely ORB‐SLAM2 and LSD‐SLAM. The results show that SOFT‐SLAM obtains better localization accuracy on the majority of datasets sequences, while also having a lower runtime.     

Pick-and-verify: verification-based highly reliable picking system for various target objects in clutter

The reliability of picking task for various objects in clutter, as measured on the Amazon Picking Challenge, is far from the expectations of automation companies. Even if the best-performed team, who run object detection before picking the object, had picked a wrong object in the competition. In this paper, we propose a practical method to compose a highly reliable picking system with verification-based approach to reduce the rate of wrong picking and raise the reliability of picking ordered objects. In our approach, which we call pick-and-verify, the robot recognizes object twice: in clutter scene to detect the target and in hand after picking an object with less time loss and rise of reliability of picking the target. For grasping the detected object we do not assume its pose and it is actually the target object, instead, we adopt vision-based grasp planning for vacuum gripper with sensed 3-D point cloud. With the presented approach, the reliability of picking target objects raised 50%, and the score in the APC2015 competition has been improved to be close to the best-performed team by picking 9 out of 12 objects in 10 min with the same hardware in our previous system.     

UAV Localization in Row Crops

High‐flying unmanned aerial vehicles (UAVs) are transforming industrial and research agriculture by delivering high spatiotemporal resolution data on a field environment. While current UAVs fly high above fields collecting aerial imagery, future low‐flying aircraft will directly interact with the environment and will utilize a wider variety of sensors. Safely and reliably operating close to unstructured environments requires improving UAVs' sensing, localization, and control algorithms. To this end, we investigate localizing a micro‐UAV in corn phenotyping trials using a laser scanner and IMU to control the altitude and position of the vehicle relative to the plant rows. In this process, the laser scanner is not only a means of localization, but also a scientific instrument for measuring plant properties. Experimental evaluations demonstrate that the is capable of safely and reliably operating in real‐world phenotyping trials. We experimentally validate the system in both low and high wind conditions in fully mature corn fields. Using test data from 18 test flights, we show that the UAV is capable of localizing its position to within one field row of the true position.     

Mapping and Tracking

This article presents the implementation of decentralized data fusion (DDF) and cooperative control algorithms on an unmanned aerial system (UAS).We conduct a number of demonstrations with a pair of unmanned aerial vehicles (UAVs) performing an information-gathering mission, and we show that significant benefits can be achieved by enabling cooperation through the sharing of information between members of the team. The objective is to utilize the UAV team to estimate the position and velocity of a number of ground-based features. The UAVs are given some prior knowledge of the feature states and are required to gather further information above a predefined threshold. This situation models a scenario where initial information is made available from an external source (e.g., a high-flying UAV or satellite imagery), which then prompts the start of the feature-localizationmission.     

Balancing search and target response in cooperative unmanned aerial vehicle (UAV) teams.

This paper considers a heterogeneous team of cooperating unmanned aerial vehicles (UAVs) drawn from several distinct classes and engaged in a search and action mission over a spatially extended battlefield with targets of several types. During the mission, the UAVs seek to confirm and verifiably destroy suspected targets and discover, confirm, and verifiably destroy unknown targets. The locations of some (or all) targets are unknown a priori, requiring them to be located using cooperative search. In addition, the tasks to be performed at each target location by the team of cooperative UAVs need to be coordinated. The tasks must, therefore, be allocated to UAVs in real time as they arise, while ensuring that appropriate vehicles are assigned to each task. Each class of UAVs has its own sensing and attack capabilities, so the need for appropriate assignment is paramount. In this paper, an extensive dynamic model that captures the stochastic nature of the cooperative search and task assignment problems is developed, and algorithms for achieving a high level of performance are designed. The paper focuses on investigating the value of predictive task assignment as a function of the number of unknown targets and number of UAVs. In particular, it is shown that there is a tradeoff between search and task response in the context of prediction. Based on the results, a hybrid algorithm for switching the use of prediction is proposed, which balances the search and task response. The performance of the proposed algorithms is evaluated through Monte Carlo simulations.     

Novel Conflict Resolution Model for Multi‐Uav Based on Cpn and 4d Trajectories

This paper presents a conflict resolution (CR) model in colored Petri net (CPN) formalism for 4D (three‐dimensional + time) planned trajectories of cooperating unmanned aerial vehicles (UAVs). The CR model is integrated with the conflict detection algorithm using spatial data structure to detect time‐based separation infringements among UAVs and to generate an intuitionistic representation of 4D conflict information. The main contribution of this paper is the proposed CR model, which is based on the logic of cause‐and‐effect analysis. This model not only chooses a preferred trajectory according to the priority for solving the current conflict but also considers the follow‐up influence (domino effect) to update segments and conflicts. The novel causal model exploits the state space to achieve the solution using CPN. The model is validated with the experimental results of a scenario involving multiple UAVs (composed of clusters) cruising in a bounded region and exhibits the main advantages of scalability, efficiency, and short execution time.     

Autonomous UAV path planning and estimation

Unmanned aerial vehicles (UAVs) have shown promise in recent years for autonomous sensing. UAVs systems have been proposed for a wide range of applications such as mapping, surveillance, search, and tracking operations. The recent availability of low-cost UAVs suggests the use of teams of vehicles to perform sensing tasks. To leverage the capabilities of a team of vehicles, efficient methods of decentralized sensing and cooperative path planning are necessary. The goal of this work is to examine practical control strategies for a team of fixed-wing vehicles performing cooperative sensing. We seek to develop decentralized, autonomous control strategies that can account for a wide variety of sensing missions. Sensing goals are posed from an information theoretic standpoint to design strategies that explicitly minimize uncertainty. This work proposes a tightly coupled approach, in which sensor models and estimation objectives are used online for path planning.     

Cooperative control of multiple unmanned aerial systems for heavy duty carrying

The idea of cooperative carrying using a team of unmanned aerial vehicles (UAVs) has been viewed as a widely applicable and deployable use of unmanned multi-agent systems, yet there are very few working systems that can successfully perform such tasks due to the complexities involved. One of the primary challenges is the limited control authority of the cooperative team due to the scale of the individual agents being necessarily much smaller than the load. Here, we propose a general method that can be used to optimise the control authority of a UAV team in the case where the agents are rigidly mounted to the payload. Firstly, a tilt angle for each UAV relative to the payload is introduced to improve the yaw control of the system, as a large wide payload will invariably have a large moment of inertia in the yaw axis. The positional placement of the UAVs and the value of the tilt angle that maximises control authority was then found using an evolutionary algorithm. The optimised solution based on a case study task involves using four UAVs placed at the corner of a square payload with an inward tilt, which can be effectively controlled as a single large quadrotor. Testing results show that the full system carrying the payload can execute given trajectories autonomously with high accuracy and precision, effectively performing the task of cooperative carrying. The proposed system was successfully demonstrated at the 2017 International Micro Air Vehicles Competition.