Cooperative autonomous search,grasping, and delivering in a treasure hunt scenario by a team of unmanned aerial vehicles

This paper addresses the problem of autonomous cooperative localization, grasping and delivering of colored ferrous objects by a team of unmanned aerial vehicles (UAVs). In the proposed scenario, a team of UAVs is required to maximize the reward by collecting colored objects and delivering them to a predefined location. This task consists of several subtasks such as cooperative coverage path planning, object detection and state estimation, UAV self‐localization, precise motion control, trajectory tracking, aerial grasping and dropping, and decentralized team coordination. The failure recovery and synchronization job manager is used to integrate all the presented subtasks together and also to decrease the vulnerability to individual subtask failures in real‐world conditions. The whole system was developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017, where it achieved the highest score and won Challenge No. 3—Treasure Hunt. This paper does not only contain results from the MBZIRC 2017 competition but it also evaluates the system performance in simulations and field tests that were conducted throughout the year‐long development and preparations for the competition.     

Team NimbRo at MBZIRC 2017: Autonomous valve stem turning using a wrench

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. In this study, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, grasped, and finally used to turn a valve stem. Mario’s omnidirectional base allows both fast locomotion and precise close approach to the manipulation panel. We describe an efficient detector for medium‐sized objects in three‐dimensional laser scans and apply it to detect the manipulation panel. An object detection architecture based on deep neural networks is used to find and select the correct tool from grayscale images. Parametrized motion primitives are adapted online to percepts of the tool and valve stem to turn the stem. We report in detail on our winning performance at the challenge and discuss lessons learned.     

A general‐purpose task execution framework for manipulation mission of the 2017 Mohamed Bin Zayed International Robotics Challenge

This paper presents the hardware and software of our team's EurecarBot for Challenge 2 in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Fully automating our robots actions in a real environment required many component technologies for manipulation and vision processing. To perform the complex robotic missions, we developed a task execution framework, which provides a high‐level interface to specify the given tasks. In this study, we focus on the valve operation problem, which was the hardest part of the competition. We also discuss how we overcame the various problems caused by differences between the experimental and the actual competition environments. EurecarBot completed the valve operation mission perfectly in the MBZIRC Grand Challenge and ranked fourth in Challenge 2 and fifth in the Grand Challenge.     

Unsupervised learning‐based flexible framework for surveillance planning with aerial vehicles

The herein studied problem is motivated by practical needs of our participation in the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 in which a team of unmanned aerial vehicles (UAVs) is requested to collect objects in the given area as quickly as possible and score according to the rewards associated with the objects. The mission time is limited, and the most time‐consuming operation is the collection of the objects themselves. Therefore, we address the problem to quickly identify the most valuable objects as surveillance planning with curvature‐constrained trajectories. The problem is formulated as a multivehicle variant of the Dubins traveling salesman problem with neighborhoods (DTSPN). Based on the evaluation of existing approaches to the DTSPN, we propose to use unsupervised learning to find satisfiable solutions with low computational requirements. Moreover, the flexibility of unsupervised learning allows considering trajectory parametrization that better fits the motion constraints of the utilized hexacopters that are not limited by the minimal turning radius as the Dubins vehicle. We propose to use Bézier curves to exploit the maximal vehicle velocity and acceleration limits. Besides, we further generalize the proposed approach to 3D surveillance planning. We report on evaluation results of the developed algorithms and experimental verification of the planned trajectories using the real UAVs utilized in our participation in MBZIRC 2017.     

Autonomous landing on a moving vehicle with an unmanned aerial vehicle

This paper addresses the perception, control, and trajectory planning for an aerial platform to identify and land on a moving car at 15 km/hr. The hexacopter unmanned aerial vehicle (UAV), equipped with onboard sensors and a computer, detects the car using a monocular camera and predicts the car future movement using a nonlinear motion model. While following the car, the UAV lands on its roof, and it attaches itself using magnetic legs. The proposed system is fully autonomous from takeoff to landing. Numerous field tests were conducted throughout the year‐long development and preparations for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 competition, for which the system was designed. We propose a novel control system in which a model predictive controller is used in real time to generate a reference trajectory for the UAV, which are then tracked by the nonlinear feedback controller. This combination allows to track predictions of the car motion with minimal position error. The evaluation presents three successful autonomous landings during the MBZIRC 2017, where our system achieved the fastest landing among all competing teams.     

Fully autonomous micro air vehicle flight and landing on a moving target using visual–inertial estimation and model‐predictive control

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) held in spring 2017 was a very successful competition well attended by teams from all over the world. One of the challenges (Challenge 1) required an aerial robot to detect, follow, and land on a moving target in a fully autonomous fashion. In this paper, we present the hardware components of the micro air vehicle (MAV) we built with off the self components alongside the designed algorithms that were developed for the purposes of the competition. We tackle the challenge of landing on a moving target by adopting a generic approach, rather than following one that is tailored to the MBZIRC Challenge 1 setup, enabling easy adaptation to a wider range of applications and targets, even indoors, since we do not rely on availability of global positioning system. We evaluate our system in an uncontrolled outdoor environment where our MAV successfully and consistently lands on a target moving at a speed of up to 5.0 m/s.     

The ETH‐MAV Team in the MBZ International Robotics Challenge

This study describes the hardware and software systems of the Micro Aerial Vehicle (MAV) platforms used by the ETH Zurich team in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). The aim was to develop robust outdoor platforms with the autonomous capabilities required for the competition, by applying and integrating knowledge from various fields, including computer vision, sensor fusion, optimal control, and probabilistic robotics. This paper presents the major components and structures of the system architectures and reports on experimental findings for the MAV‐based challenges in the competition. Main highlights include securing the second place both in the individual search, pick, and place the task of Challenge 3 and the Grand Challenge, with autonomous landing executed in less than 1 min and a visual servoing success rate of over for object pickups.     

A mission management system for complex aerial logistics by multiple unmanned aerial vehicles in MBZIRC 2017

In this study, we present a system that manages multiple unmanned aerial vehicles (UAVs) for a search, pickup, and drop mission in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Three UAVs picked up and dropped 23 circular and rectangular targets into a designated drop box. To control the operation of three UAVs flying over an arena of 90 × 60 m, we designed and integrated a set of technologies into our system: airspace allocation, communication framework among UAVs, anticollision based on geofencing, and a token‐based prioritization for coordination. The proposed UAV system uses a single GPS and its error of a few meters is solved by means of the following component technologies: (a) flight path generator based on one reference point, (b) vision‐based redefinition of a reference point for GPS correction, and (c) calibration of flight path to update the reference point. The pickup‐and‐drop mission is conducted via color‐ and shape‐based vision processing and a magnetic gripper to pickup and drop‐off the targets. Our proposed system is able to successfully manage three UAVs, recognize targets on the ground, and drop the targets into a drop box in the drop zone. Finally, we achieved fourth place among 18 teams in Challenge 3.     

Al‐Robotics team: A cooperative multi‐unmanned aerial vehicle approach for the Mohamed Bin Zayed International Robotic Challenge

The Al‐Robotics team was selected as one of the 25 finalist teams out of 143 applications received to participate in the first edition of the Mohamed Bin Zayed International Robotic Challenge (MBZIRC), held in 2017. In particular, one of the competition Challenges offered us the opportunity to develop a cooperative approach with multiple unmanned aerial vehicles (UAVs) searching, picking up, and dropping static and moving objects. This paper presents the approach that our team Al‐Robotics followed to address that Challenge 3 of the MBZIRC. First, we overview the overall architecture of the system, with the different modules involved. Second, we describe the procedure that we followed to design the aerial platforms, as well as all their onboard components. Then, we explain the techniques that we used to develop the software functionalities of the system. Finally, we discuss our experimental results and the lessons that we learned before and during the competition. The cooperative approach was validated with fully autonomous missions in experiments previous to the actual competition. We also analyze the results that we obtained during the competition trials.     

A lightweight autonomous MAV for indoor search and rescue

Micro Aerial Vehicles (MAVs) have great potentials to be applied for indoor search and rescue missions. In this paper, we propose a modular lightweight design of an autonomous MAV with integrated hardware and software. The MAV is equipped with the 2D laser scanner, camera, mission computer and flight controller, running all the computation onboard in real time. The onboard perception system includes a laser‐based SLAM module and a custom‐designed visual detection module. A dual Kalman filter design provides robust state estimation by multiple sensor fusion. Specifically, the fusion module provides robust altitude measurement in the circumstance of surface changing. In addition, indoor‐outdoor transition is explicitly handled by the fusion module. In order to efficiently navigate through obstacles and adapt to multiple tasks, a task tree‐based mission planning method is seamlessly integrated with path planning and control modules. The MAV is capable of searching and rescuing victims from unknown indoor environments effectively. It was validated by our award‐winning performance at the 2017 International Micro Air Vehicle Competition (IMAV 2017), held in Toulouse, France. The performance video is available on https://youtu.be/8H19ppS_VXM .     

Autonomous Navigation for Micro Aerial Vehicles in Complex GNSS-denied Environments

Micro aerial vehicles, such as multirotors, are particular well suited for the autonomous monitoring, inspection, and surveillance of buildings, e.g., for maintenance in industrial plants. Key prerequisites for the fully autonomous operation of micro aerial vehicles in restricted environments are 3D mapping, real-time pose tracking, obstacle detection, and planning of collision-free trajectories. In this article, we propose a complete navigation system with a multimodal sensor setup for omnidirectional environment perception. Measurements of a 3D laser scanner are aggregated in egocentric local multiresolution grid maps. Local maps are registered and merged to allocentric maps in which the MAV localizes. For autonomous navigation, we generate trajectories in a multi-layered approach: from mission planning over global and local trajectory planning to reactive obstacle avoidance. We evaluate our approach in a GNSS-denied indoor environment where multiple collision hazards require reliable omnidirectional perception and quick navigation reactions.     

Robot Teams for Intervention in Confined and Structured Environments

Safety, security, and rescue robotics can be extremely useful in emergency scenarios such as mining accidents or tunnel collapses where robot teams can be used to carry out cooperative exploration, intervention, or logistic missions. Deploying a multirobot team in such confined environments poses multiple challenges that involve task planning, motion planning, localization and mapping, safe navigation, coordination, and communications among all the robots. To complete their mission, robots have to be able to move in the environment with full autonomy while at the same time maintaining communication among themselves and with their human operators to accomplish team collaboration. Guaranteeing connectivity enables robots to explicitly exchange information needed in the execution of collaborative tasks and allows operators to monitor and teleoperate the robots and receive information about the environment. In this work, we present a system that integrates several research aspects to achieve a real exploration exercise in a tunnel using a robot team. These aspects are as follows: deployment planning, semantic feature recognition, multirobot navigation, localization, map building, and real‐time communications. Two experimental scenarios have been used for the assessment of the system. The first is the Spanish Santa Marta mine, a large mazelike environment selected for its complexity for all the tasks involved. The second is the Spanish‐French Somport tunnel, an old railway between Spain and France through the Central Pyrenees, used to carry out the real‐world experiments. The latter is a simpler scenario, but it serves to highlight the real communication issues.     

Air‐ground Matching: Appearance‐based GPS‐denied Urban Localization of Micro Aerial Vehicles

In this paper, we address the problem of globally localizing and tracking the pose of a camera‐equipped micro aerial vehicle (MAV) flying in urban streets at low altitudes without GPS. An image‐based global positioning system is introduced to localize the MAV with respect to the surrounding buildings. We propose a novel air‐ground image‐matching algorithm to search the airborne image of the MAV within a ground‐level, geotagged image database. Based on the detected matching image features, we infer the global position of the MAV by back‐projecting the corresponding image points onto a cadastral three‐dimensional city model. Furthermore, we describe an algorithm to track the position of the flying vehicle over several frames and to correct the accumulated drift of the visual odometry whenever a good match is detected between the airborne and the ground‐level images. The proposed approach is tested on a 2 km trajectory with a small quadrocopter flying in the streets of Zurich. Our vision‐based global localization can robustly handle extreme changes in viewpoint, illumination, perceptual aliasing, and over‐season variations, thus outperforming conventional visual place‐recognition approaches. The dataset is made publicly available to the research community. To the best of our knowledge, this is the first work that studies and demonstrates global localization and position tracking of a drone in urban streets with a single onboard camera.     

Monocular Vision for Long‐term Micro Aerial Vehicle State Estimation: A Compendium

The recent technological advances in Micro Aerial Vehicles (MAVs) have triggered great interest in the robotics community, as their deployability in missions of surveillance and reconnaissance has now become a realistic prospect. The state of the art, however, still lacks solutions that can work for a long duration in large, unknown, and GPS‐denied environments. Here, we present our visual pipeline and MAV state‐estimation framework, which uses feeds from a monocular camera and an Inertial Measurement Unit (IMU) to achieve real‐time and onboard autonomous flight in general and realistic scenarios. The challenge lies in dealing with the power and weight restrictions onboard a MAV while providing the robustness necessary in real and long‐term missions. This article provides a concise summary of our work on achieving the first onboard vision‐based power‐on‐and‐go system for autonomous MAV flights. We discuss our insights on the lessons learned throughout the different stages of this research, from the conception of the idea to the thorough theoretical analysis of the proposed framework and, finally, the real‐world implementation and deployment. Looking into the onboard estimation of monocular visual odometry, the sensor fusion strategy, the state estimation and self‐calibration of the system, and finally some implementation issues, the reader is guided through the different modules comprising our framework. The validity and power of this framework are illustrated via a comprehensive set of experiments in a large outdoor mission, demonstrating successful operation over flights of more than 360 m trajectory and 70 m altitude change. ¹      

Localization of emergency acoustic sources by micro aerial vehicles

For micro aerial vehicles (MAVs) involved in search and rescue missions, the ability to locate the source of a distress sound signal is significantly important and allows fast localization of victims and rescuers during nighttime, through foliage and in dust, fog, and smoke. Most emergency sound sources, such as safety whistles and personal alarms, generate a narrowband signal that is difficult to localize by human listeners or with the common localization methods suitable for broadband sounds. In this paper, we present three methods for MAV‐based emergency sound localization system. The first method involves designing a new emergency source for immediate localization by the MAV using a common localization method. The other two novel methods allow localizing the currently available emergency sources, or other narrowband sounds in general, that are difficult to localize due to the periodicity in the sequence of sound samples. The second method exploits the Doppler shift in the sound frequency, caused due to the motion of the MAV and the dynamics of the MAV to assist with the localization. The third method involves active control of the robot's attitude and fusing acoustic and attitude measurements for achieving accurate and robust estimates. We evaluate our methods in real‐world experiments with real flying robots.     

RANGE–Robust autonomous navigation in GPS‐denied environments

This paper addresses the problem of autonomous navigation of a micro air vehicle (MAV) in GPS‐denied environments. We present experimental validation and analysis for our system that enables a quadrotor helicopter, equipped with a laser range finder sensor, to autonomously explore and map unstructured and unknown environments. The key challenge for enabling GPS‐denied flight of a MAV is that the system must be able to estimate its position and velocity by sensing unknown environmental structure with sufficient accuracy and low enough latency to stably control the vehicle. Our solution overcomes this challenge in the face of MAV payload limitations imposed on sensing, computational, and communication resources. We first analyze the requirements to achieve fully autonomous quadrotor helicopter flight in GPS‐denied areas, highlighting the differences between ground and air robots that make it difficult to use algorithms developed for ground robots. We report on experiments that validate our solutions to key challenges, namely a multilevel sensing and control hierarchy that incorporates a high‐speed laser scan‐matching algorithm, data fusion filter, high‐level simultaneous localization and mapping, and a goal‐directed exploration module. These experiments illustrate the quadrotor helicopter's ability to accurately and autonomously navigate in a number of large‐scale unknown environments, both indoors and in the urban canyon. The system was further validated in the field by our winning entry in the 2009 International Aerial Robotics Competition, which required the quadrotor to autonomously enter a hazardous unknown environment through a window, explore the indoor structure without GPS, and search for a visual target. © 2011 Wiley Periodicals, Inc.     

Optimal surveillance coverage for teams of micro aerial vehicles in GPS-denied environments using onboard vision

This paper deals with the problem of deploying a team of flying robots to perform surveillance-coverage missions over a terrain of arbitrary morphology. In such missions, a key factor for the successful completion of the mission is the knowledge of the terrain’s morphology. The focus of this paper is on the implementation of a two-step procedure that allows us to optimally align a team of flying vehicles for the aforementioned task. Initially, a single robot constructs a map of the area using a novel monocular-vision-based approach. A?state-of-the-art visual-SLAM algorithm tracks the pose of the camera while, simultaneously, autonomously, building an incremental map of the environment. The map generated is processed and serves as an input to an optimization procedure using the cognitive, adaptive methodology initially introduced in Renzaglia et?al. (Proceedings of the IEEE international conference on robotics and intelligent system (IROS), Taipei, Taiwan, pp.?3314–3320, 2010). The output of this procedure is the optimal arrangement of the robots team, which maximizes the monitored area. The efficiency of our approach is demonstrated using real data collected from aerial robots in different outdoor areas.     

Motion planning for cooperative unicycle-type mobile robots with limited sensing ranges: A distributed receding horizon approach

This paper presents a decentralized motion planner for a team of nonholonomic mobile robots subject to constraints imposed by sensors and the communication network. The motion planning scheme consists of decentralized receding horizon planners that reside on each vehicle to achieve coordination among flocking agents. The advantage of the proposed algorithm is that each vehicle only requires local knowledge of its neighboring vehicles. The main requirement for designing an optimal conflict-free trajectory in a decentralized way is that each robot does not deviate too far from its presumed trajectory designed without taking the coupling constraints into account. A comparative study between the proposed algorithm and other existing algorithms is provided in order to show the advantages, especially in terms of computing time. Finally, experiments are performed on a team of three mobile robots to demonstrate the validity of the proposed approach.     

多机器人路径规划的安全性验证

近些年来,伴随着人工智能领域的浪潮,机器人越来越多的出现在我们的日常生活中,例如足球机器人、无人机、无人车等.如何保证这些自治机器人尤其是多个机器人在移动过程中的安全成了人们一直很关心的问题.混成通信顺序进程（Hybrid Communicating Sequential Process,HCSP）是一个针对混成系统的形式化建模语言,在通信顺序进程（Communicating Sequential Process,CSP）的基础上引入了微分方程以描述混成系统中的连续行为和控制逻辑,可以方便高效地对大型控制系统尤其是在有通信事件发生时的情形进行形式化建模.本文就是用HCSP建模多机器人的路径控制算法,并用定理证明工具HProver进行形式化验证.结果证明了在满足一定初始条件下,机器人团队在整个运行途中不会发生碰撞.     

The call of duty: Self-organised task allocation in a population of up to twelve mobile robots

Teams with up to 12 real robots were given the mission to maintain the energy stocked in their nest by collecting food-items. To achieve this mission efficiently, we implemented a simple and decentralised task allocation mechanism based on individual activation-thresholds, i.e. the energy level of the nest under which a given robot decides to go collect food-items. The experiments show that such a mechanism — already studied among social insects — results in an efficient dynamical task allocation even under the noisy conditions prevailing in real experiments. Experiments with different team sizes were carried out to investigate the effect of team size on performance and risk of mission failure.