Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. This study covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined “Treasure Hunt,” which required mission planning and multirobot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge‐specific extensions, cover the camera‐based perception, explain control and trajectory‐planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real‐robot experiments during the competition in Abu Dhabi. With our system, we—as part of the larger team NimbRo—won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.     

Cooperative autonomous search,grasping, and delivering in a treasure hunt scenario by a team of unmanned aerial vehicles

This paper addresses the problem of autonomous cooperative localization, grasping and delivering of colored ferrous objects by a team of unmanned aerial vehicles (UAVs). In the proposed scenario, a team of UAVs is required to maximize the reward by collecting colored objects and delivering them to a predefined location. This task consists of several subtasks such as cooperative coverage path planning, object detection and state estimation, UAV self‐localization, precise motion control, trajectory tracking, aerial grasping and dropping, and decentralized team coordination. The failure recovery and synchronization job manager is used to integrate all the presented subtasks together and also to decrease the vulnerability to individual subtask failures in real‐world conditions. The whole system was developed for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017, where it achieved the highest score and won Challenge No. 3—Treasure Hunt. This paper does not only contain results from the MBZIRC 2017 competition but it also evaluates the system performance in simulations and field tests that were conducted throughout the year‐long development and preparations for the competition.     

Fully autonomous micro air vehicle flight and landing on a moving target using visual–inertial estimation and model‐predictive control

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) held in spring 2017 was a very successful competition well attended by teams from all over the world. One of the challenges (Challenge 1) required an aerial robot to detect, follow, and land on a moving target in a fully autonomous fashion. In this paper, we present the hardware components of the micro air vehicle (MAV) we built with off the self components alongside the designed algorithms that were developed for the purposes of the competition. We tackle the challenge of landing on a moving target by adopting a generic approach, rather than following one that is tailored to the MBZIRC Challenge 1 setup, enabling easy adaptation to a wider range of applications and targets, even indoors, since we do not rely on availability of global positioning system. We evaluate our system in an uncontrolled outdoor environment where our MAV successfully and consistently lands on a target moving at a speed of up to 5.0 m/s.     

A general‐purpose task execution framework for manipulation mission of the 2017 Mohamed Bin Zayed International Robotics Challenge

This paper presents the hardware and software of our team's EurecarBot for Challenge 2 in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Fully automating our robots actions in a real environment required many component technologies for manipulation and vision processing. To perform the complex robotic missions, we developed a task execution framework, which provides a high‐level interface to specify the given tasks. In this study, we focus on the valve operation problem, which was the hardest part of the competition. We also discuss how we overcame the various problems caused by differences between the experimental and the actual competition environments. EurecarBot completed the valve operation mission perfectly in the MBZIRC Grand Challenge and ranked fourth in Challenge 2 and fifth in the Grand Challenge.     

Unsupervised learning‐based flexible framework for surveillance planning with aerial vehicles

The herein studied problem is motivated by practical needs of our participation in the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 in which a team of unmanned aerial vehicles (UAVs) is requested to collect objects in the given area as quickly as possible and score according to the rewards associated with the objects. The mission time is limited, and the most time‐consuming operation is the collection of the objects themselves. Therefore, we address the problem to quickly identify the most valuable objects as surveillance planning with curvature‐constrained trajectories. The problem is formulated as a multivehicle variant of the Dubins traveling salesman problem with neighborhoods (DTSPN). Based on the evaluation of existing approaches to the DTSPN, we propose to use unsupervised learning to find satisfiable solutions with low computational requirements. Moreover, the flexibility of unsupervised learning allows considering trajectory parametrization that better fits the motion constraints of the utilized hexacopters that are not limited by the minimal turning radius as the Dubins vehicle. We propose to use Bézier curves to exploit the maximal vehicle velocity and acceleration limits. Besides, we further generalize the proposed approach to 3D surveillance planning. We report on evaluation results of the developed algorithms and experimental verification of the planned trajectories using the real UAVs utilized in our participation in MBZIRC 2017.     

The ETH‐MAV Team in the MBZ International Robotics Challenge

This study describes the hardware and software systems of the Micro Aerial Vehicle (MAV) platforms used by the ETH Zurich team in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). The aim was to develop robust outdoor platforms with the autonomous capabilities required for the competition, by applying and integrating knowledge from various fields, including computer vision, sensor fusion, optimal control, and probabilistic robotics. This paper presents the major components and structures of the system architectures and reports on experimental findings for the MAV‐based challenges in the competition. Main highlights include securing the second place both in the individual search, pick, and place the task of Challenge 3 and the Grand Challenge, with autonomous landing executed in less than 1 min and a visual servoing success rate of over for object pickups.     

Autonomous landing on a moving vehicle with an unmanned aerial vehicle

This paper addresses the perception, control, and trajectory planning for an aerial platform to identify and land on a moving car at 15 km/hr. The hexacopter unmanned aerial vehicle (UAV), equipped with onboard sensors and a computer, detects the car using a monocular camera and predicts the car future movement using a nonlinear motion model. While following the car, the UAV lands on its roof, and it attaches itself using magnetic legs. The proposed system is fully autonomous from takeoff to landing. Numerous field tests were conducted throughout the year‐long development and preparations for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 competition, for which the system was designed. We propose a novel control system in which a model predictive controller is used in real time to generate a reference trajectory for the UAV, which are then tracked by the nonlinear feedback controller. This combination allows to track predictions of the car motion with minimal position error. The evaluation presents three successful autonomous landings during the MBZIRC 2017, where our system achieved the fastest landing among all competing teams.     

Team NimbRo at MBZIRC 2017: Autonomous valve stem turning using a wrench

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. In this study, we describe our winning entry to MBZIRC Challenge 2: the mobile manipulation robot Mario. It is capable of autonomously solving a valve manipulation task using a wrench tool detected, grasped, and finally used to turn a valve stem. Mario’s omnidirectional base allows both fast locomotion and precise close approach to the manipulation panel. We describe an efficient detector for medium‐sized objects in three‐dimensional laser scans and apply it to detect the manipulation panel. An object detection architecture based on deep neural networks is used to find and select the correct tool from grayscale images. Parametrized motion primitives are adapted online to percepts of the tool and valve stem to turn the stem. We report in detail on our winning performance at the challenge and discuss lessons learned.     

A mission management system for complex aerial logistics by multiple unmanned aerial vehicles in MBZIRC 2017

In this study, we present a system that manages multiple unmanned aerial vehicles (UAVs) for a search, pickup, and drop mission in the 2017 Mohamed Bin Zayed International Robotics Challenge (MBZIRC). Three UAVs picked up and dropped 23 circular and rectangular targets into a designated drop box. To control the operation of three UAVs flying over an arena of 90 × 60 m, we designed and integrated a set of technologies into our system: airspace allocation, communication framework among UAVs, anticollision based on geofencing, and a token‐based prioritization for coordination. The proposed UAV system uses a single GPS and its error of a few meters is solved by means of the following component technologies: (a) flight path generator based on one reference point, (b) vision‐based redefinition of a reference point for GPS correction, and (c) calibration of flight path to update the reference point. The pickup‐and‐drop mission is conducted via color‐ and shape‐based vision processing and a magnetic gripper to pickup and drop‐off the targets. Our proposed system is able to successfully manage three UAVs, recognize targets on the ground, and drop the targets into a drop box in the drop zone. Finally, we achieved fourth place among 18 teams in Challenge 3.     

Force‐based perception and manipulation,the DTU team competing in MBZIRC 2017

This paper presents how the team from the Technical University of Denmark (DTU) implemented and solved the second challenge of the Mohamed Bin Zayed International Robotics Challenge. The competition was imitating a disaster scene where a robotic platform had to operate autonomously in a partly known environment to localize and manipulate a valve on a panel. To solve the given problem, the robot needs to be able to perceive the environment reliably. This is often accomplished using vision based solutions, however these might not always be feasible. Thus we show how force feedback can successfully be used as an alternative way of perception. To accomplish this the team equipped a robot arm with a force torque sensor, allowing the robot to perceive its environment through direct contact. This approach resulted in a robust solution of the task, independent of several external factors, such as lighting, which might affect a more traditional approach. First the theory and thoughts behind the implementation is presented, followed by an evaluation of the results from physical experiments and the competition itself, ultimately resulting in a robust solution which performed without errors in the competition.     

Multimodal sensing and active continuous closed‐loop feedback for achieving reliable manipulation in the outdoor physical world

The use of field robots can greatly decrease the amount of time, effort, and associated risk compared to if human workers were to carryout certain tasks such as disaster response. However, transportability and reliability remain two main issues for most current robot systems. To address the issue of transportability, we have developed a lightweight modularizable platform named AeroArm. To address the issue of reliability, we utilize a multimodal sensing approach, combining the use of multiple sensors and sensor types, and the use of different detection algorithms, as well as active continuous closed‐loop feedback to accurately estimate the state of the robot with respect to the environment. We used Challenge 2 of the 2017 Mohammed Bin Zayed International Robotics Competition as an example outdoor manipulation task, demonstrating the capabilities of our robot system and approach in achieving reliable performance in the fields, and ranked fifth place internationally in the competition.     

Zeus: A system description of the two‐time winner of the collegiate SAE autodrive competition

The SAE AutoDrive Challenge is a 3‐year collegiate competition to develop a self‐driving car by 2020. The second year of the competition was held in June 2019 at MCity, a mock town built for self‐driving car testing at the University of Michigan. Teams were required to autonomously navigate a series of intersections while handling pedestrians, traffic lights, and traffic signs. Zeus is aUToronto's winning entry in the AutoDrive Challenge. This article describes the system design and development of Zeus as well as many of the lessons learned along the way. This includes details on the team's organizational structure, sensor suite, software components, and performance at the Year 2 competition. With a team of mostly undergraduates and minimal resources, aUToronto has made progress toward a functioning self‐driving vehicle, in just 2 years. This article may prove valuable to researchers looking to develop their own self‐driving platform.     

Team O2AS at the world robot summit 2018: an approach to robotic kitting and assembly tasks using general purpose grippers and tools

ABSTRACT

In this article, we propose a versatile robotic system for kitting and assembly tasks which uses no jigs or commercial tool changers. Instead of specialized end effectors, it uses its two-finger grippers to grasp and hold tools to perform subtasks such as screwing and suctioning. A third gripper is used as a precision picking and centering tool, and uses in-built passive compliance to compensate for small position errors and uncertainty. A novel grasp point detection for bin picking is described for the kitting task, using a single depth map. Using the proposed system we competed in the Assembly Challenge of the Industrial Robotics Category of the World Robot Challenge at the World Robot Summit 2018, obtaining 4th place and the SICE award for lean design and versatile tool use. We show the effectiveness of our approach through experiments performed during the competition.     

Technical Overview of Team DRC‐Hubo@UNLV's Approach to the 2015 DARPA Robotics Challenge Finals

This paper presents a technical overview of Team DRC‐Hubo@UNLV's approach to the 2015 DARPA Robotics Challenge Finals (DRC‐Finals). The Finals required a robotic platform that was robust and reliable in both hardware and software to complete tasks in 60 min under degraded communication. With this point of view, Team DRC‐Hubo@UNLV integrated methods and algorithms previously verified, validated, and widely used in the robotics community. For the communication aspect, a common shared memory approach that the team adopted to enable efficient data communication under the DARPA controlled network is described. A new perception head design (optimized for the tasks of the Finals) and its data processing are then presented. In the motion planning and control aspect, various techniques, such as wheel‐driven navigation, zero‐moment‐point (ZMP) ‐based locomotion, and position‐based manipulation and controls, are described in this paper. By introducing strategically critical elements and key lessons learned from DRC‐Trials 2013 and the testbed of Charleston, we also illustrate how DRC‐Hubo has evolved successfully toward the DRC‐Finals.     

From Autonomy to Cooperative Traded Control of Humanoid Manipulation Tasks with Unreliable Communication

In this paper, we present our system design, operational procedure, testing process, field results, and lessons learned for the valve-turning task of the DARPA Robotics Challenge (DRC). We present a software framework for cooperative traded control that enables a team of operators to control a remote humanoid robot over an unreliable communication link. Our system, composed of software modules running on-board the robot and on a remote workstation, allows the operators to specify the manipulation task in a straightforward manner. In addition, we have defined an operational procedure for the operators to manage the teleoperation task, designed to improve situation awareness and expedite task completion. Our testing process, consisting of hands-on intensive testing, remote testing, and remote practice runs , demonstrates that our framework is able to perform reliably and is resilient to unreliable network conditions. We analyze our approach, field tests, and experience at the DRC Trials and discuss lessons learned which may be useful for others when designing similar systems.     

Robot System of DRC‐HUBO+ and Control Strategy of Team KAIST in DARPA Robotics Challenge Finals

This paper summarizes how Team KAIST prepared for the DARPA Robotics Challenge (DRC) Finals, especially in terms of the robot system and control strategy. To imitate the Fukushima nuclear disaster situation, the DRC performed eight tasks and degraded communication conditions. This competition demanded various robotic technologies, such as manipulation, mobility, telemetry, autonomy, and localization. Their systematic integration and the overall system robustness were also important issues in completing the challenge. In this sense, this paper presents a hardware and software system for the DRC‐HUBO+, a humanoid robot that was used for the DRC; it also presents control methods, such as inverse kinematics, compliance control, a walking algorithm, and a vision algorithm, all of which were implemented to accomplish the tasks. The strategies and operations for each task are briefly explained with vision algorithms. This paper summarizes what we learned from the DRC before the conclusion. In the competition, 25 international teams participated with their various robot platforms. We competed in this challenge using the DRC‐HUBO+ and won first place in the competition.     

Vision techniques for on‐board detection,following, and mapping of moving targets

This study presents computer vision modules of a multi‐unmanned aerial vehicle (UAV) system, which scored gold, silver, and bronze medals at the Mohamed Bin Zayed International Robotics Challenge 2017. This autonomous system, which was running completely on board and in real time, had to address two complex tasks in challenging outdoor conditions. In the first task, an autonomous UAV had to find, track, and land on a human‐driven car moving at 15 km/hr on a figure‐eight‐shaped track. During the second task, a group of three UAVs had to find small colored objects in a wide area, pick them up, and deliver them into a specified drop‐off zone. The computer vision modules presented here achieved computationally efficient detection, accurate localization, robust velocity estimation, and reliable future position prediction of both the colored objects and the car. These properties had to be achieved in adverse outdoor environments with changing light conditions. Lighting varied from intense direct sunlight with sharp shadows cast over the objects by the UAV itself, to reduced visibility caused by overcast to dust and sand in the air. The results presented in this paper demonstrate good performance of the modules both during testing, which took place in the harsh desert environment of the central area of United Arab Emirates, as well as during the contest, which took place at a racing complex in the urban, near‐sea location of Abu Dhabi. The stability and reliability of these modules contributed to the overall result of the contest, where our multi‐UAV system outperformed teams from world’s leading robotic laboratories in two challenging scenarios.     

SOFT‐SLAM: Computationally efficient stereo visual simultaneous localization and mapping for autonomous unmanned aerial vehicles

Autonomous navigation of unmanned aerial vehicles (UAVs) in GPS‐denied environments is a challenging problem, especially for small‐scale UAVs characterized by a small payload and limited battery autonomy. A possible solution to the aforementioned problem is vision‐based simultaneous localization and mapping (SLAM), since cameras, due to their dimensions, low weight, availability, and large information bandwidth, circumvent all the constraints of UAVs. In this paper, we propose a stereo vision SLAM yielding very accurate localization and a dense map of the environment developed with the aim to compete in the European Robotics Challenges (EuRoC) targeting airborne inspection of industrial facilities with small‐scale UAVs. The proposed approach consists of a novel stereo odometry algorithm relying on feature tracking (SOFT), which currently ranks first among all stereo methods on the KITTI dataset. Relying on SOFT for pose estimation, we build a feature‐based pose graph SLAM solution, which we dub SOFT‐SLAM. SOFT‐SLAM has a completely separate odometry and mapping threads supporting large loop‐closing and global consistency. It also achieves a constant‐time execution rate of 20 Hz with deterministic results using only two threads of an onboard computer used in the challenge. The UAV running our SLAM algorithm obtained the highest localization score in the EuRoC Challenge 3, Stage IIa–Benchmarking, Task 2. Furthermore, we also present an exhaustive evaluation of SOFT‐SLAM on two popular public datasets, and we compare it to other state‐of‐the‐art approaches, namely ORB‐SLAM2 and LSD‐SLAM. The results show that SOFT‐SLAM obtains better localization accuracy on the majority of datasets sequences, while also having a lower runtime.     

Plant inspection by using a ground vehicle and an aerial robot: lessons learned from plant disaster prevention challenge in world robot summit 2018

Abstract

In recent years, to prevent accidents and disaster are desired by implementing maintenance and management of facilities, such as conducting periodic inspections with appropriate frequency at plants. However, because the dangerous materials such as flammable gas and explosives is used in a plant, and there are many dangerous places in a plant such as high-temperature environment and high places and narrow spaces, it is desirable to use a remote-controlled robot for safety work and short inspections. Against this background, the Disaster Robotics Category-Plant Disaster Prevention Challenge was held in Japan at the World Robot Summit 2018. Our team was ranked 3rd in this competition, because our strategy of ‘inspection and investigation in cooperation with UGV and UAV’ was effective. In this paper, the competition contents of World Robot Summit 2018 and the robot inspection system that we are studying are explained. And what kind of strategy was challenged and result for these given competition tasks by using our robot system are introduced. And the lessons learned such as advantages and issues in UGV and UAV collaboration work at this competition are described for evaluate a robot investigation system for disaster response and inspection work at plants.