Vision Based Output Feedback Control of Micro Aerial Vehicles in Indoor Environments

We present a new image based visual servoing (IBVS) approach for control of micro aerial vehicles (MAVs) in indoor environments. Specifically, we show how a MAV can be stabilized and guided using only corridor lines viewed on a front facing camera and angular velocity measurements. Since the suggested controller does not include explicit attitude feedback it does not require the use of accelerometers which are susceptible to vibrations, nor complex attitude estimation algorithms. The controller also does not require direct velocity measurements which are difficult to obtain in indoor environments. The paper presents the new method, stability analysis, simulations and experiments.     

视觉感知的端到端自动驾驶运动规划综述

视觉感知模块能够利用摄像机等视觉传感器获取丰富的图像和视频信息,进而检测自动驾驶汽车视野中的车辆、行人与交通标识等信息,是自动驾驶最有效、成本最低的感知方式之一。运动规划为自主车辆提供从车辆初始状态到目标状态的一系列运动参数和驾驶动作,而端到端的模型能够直接从感知的数据获取车辆的运动参数,因而受到广泛的关注。为了全面反映视觉感知的端到端自动驾驶运动规划方法的研究进展,本文对国内外公开发表的具有代表性和前沿的论文进行了概述。首先分析端到端方法的应用,以及视觉感知和运动规划在端到端自动驾驶中的作用,然后以自主车辆的学习方式作为分类依据,将视觉感知的端到端自动驾驶运动规划的实现方法分为模仿学习和强化学习两大类,并对各类方法的不同算法进行了归纳和分析;考虑到现阶段端到端模型的研究面临着虚拟到现实的任务,故对基于迁移学习的方法进行了梳理。最后列举与自动驾驶相关的数据集和仿真平台,总结存在的问题和挑战,对未来的发展趋势进行思考和展望。视觉感知的端到端自动驾驶运动规划模型的普适性强且结构简单,这类方法具有广阔的应用前景和研究价值,但是存在不可解释和难以保证绝对安全的问题,未来需要更多的研究改善端到端模型存在的局限性。     

Inertially Aided Visual Odometry for Miniature Air Vehicles in GPS-denied Environments

Unmanned miniature air vehicles (MAVs) have recently become a focus of much research, due to their potential utility in a number of information gathering applications. MAVs currently carry inertial sensor packages that allow them to perform basic flight maneuvers reliably in a completely autonomous manner. However, MAV navigation requires knowledge of location that is currently available only through GPS sensors, which depend on an external infrastructure and are thus prone to reliability issues. Vision-based methods such as Visual Odometry (VO) have been developed that are capable of estimating MAV pose purely from vision, and thus have the potential to provide an autonomous alternative to GPS for MAV navigation. Because VO estimates pose by combining relative pose estimates, constraining relative pose error is the key element of any Visual Odometry system. In this paper, we present a system that fuses measurements from an MAV inertial navigation system (INS) with a novel VO framework based on direct image registration. We use the inertial sensors in the measurement step of the Extended Kalman Filter to determine the direction of gravity, and hence provide error-bounded measurements of certain portions of the aircraft pose. Because of the relative nature of VO measurements, we use VO in the EKF prediction step. To allow VO to be used as a prediction, we develop a novel linear approximation to the direct image registration procedure that allows us to propagate the covariance matrix at each time step. We present offline results obtained from our pose estimation system using actual MAV flight data. We show that fusion of VO and INS measurements greatly improves the accuracy of pose estimation and reduces the drift compared to unaided VO during medium-length (tens of seconds) periods of GPS dropout.     

Improved eye-vergence visual servoing system in longitudinal direction with RM-GA

Visual servoing is a control method to manipulate the motion of the robot using visual information, which aims to realize “working while watching.” However, the visual servoing towards moving target with hand–eye cameras fixed at hand is inevitably affected by hand dynamical oscillation. To overcome this defect of the hand–eye fixed camera system, an eye-vergence system has been put forward, where the pose of the cameras could be rotated to observe the target object. The visual servoing controllers of hand and eye-vergence are installed independently, so that it can observe the target object at the center of camera images through eye-vergence function. In this research, genetic algorithm (GA) is used as a pose tracking method, which is called “Real-Time Multi-step GA(RM-GA),” solves on-line optimization problems for 3D visual servoing. The performances of real-time object tracking using eye-vergence system and “RM-GA” method have been examined, and also the pose tracking accuracy has been verified.     

Image Based and Hybrid Visual Servo Control of an Unmanned Aerial Vehicle

The use of unmanned aerial vehicles (UAVs) for military, scientific, and civilian sectors are increasing drastically in recent years. This study presents algorithms for the visual-servo control of an UAV, in which a quadrotor helicopter has been stabilized with visual information through the control loop. Unlike previous study that use pose estimation approach which is time consuming and subject to various errors, the visual-servo control is more reliable and fast. The method requires a camera on-board the vehicle, which is already available on various UAV systems. The UAV with a camera behaves like an eye-in-hand visual servoing system. In this study the controller was designed by using two different approaches; image based visual servo control method and hybrid visual servo control method. Various simulations are developed on Matlab, in which the quadrotor aerial vehicle has been visual-servo controlled. In order to show the effectiveness of the algorithms, experiments were performed on a model quadrotor UAV, which suggest successful performance.     

基于单目视觉的车辆检测算法研究

本文在全面总结现有的国内外前方行驶车辆的视觉检测算法的基础上,利用由于光照因素使得前方车辆在图像内形成的底部阴影这一先验知识,建立了基于车辆阴影区域检测的前方障碍物视觉检测算法。     

Vision-based integrated system for object inspection and handling

Image-based effector servoing is a process of perception–action cycles for handling a robot effector under continual visual feedback. This paper applies visual servoing mechanisms not only for handling objects, but also for camera calibration and object inspection. A 6-DOF manipulator and a stereo camera head are mounted on separate platforms and are steered independently. In a first phase (calibration phase), camera features are determined like the optical axes and the fields of sharp view. In the second phase (inspection phase), the robot hand carries an object into the field of view of one camera, then approaches the object along the optical axis to the camera, rotates the object for reaching an optimal view, and finally the object shape is inspected in detail. In the third phase (assembly phase), the system localizes a board containing holes of different shapes, determines the hole which fits most appropriate to the object shape, then approaches and arranges the object appropriately. The final object insertion is based on haptic sensors, but is not treated in the paper. At present, the robot system has the competence to handle cylindrical and cuboid pegs. For handling other object categories the system can be extended with more sophisticated strategies of the inspection and/or assembly phase.     

Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System,Algorithms, and Experiments

There are two main trends in the development of unmanned aerial vehicle(UAV)technologies:miniaturization and intellectualization,in which realizing object tracking capabilities for a nano-scale UAV is one of the most challenging problems.In this paper,we present a visual object tracking and servoing control system utilizing a tailor-made 38 g nano-scale quadrotor.A lightweight visual module is integrated to enable object tracking capabilities,and a micro positioning deck is mounted to provide accurate pose estimation.In order to be robust against object appearance variations,a novel object tracking algorithm,denoted by RMCTer,is proposed,which integrates a powerful short-term tracking module and an efficient long-term processing module.In particular,the long-term processing module can provide additional object information and modify the short-term tracking model in a timely manner.Furthermore,a positionbased visual servoing control method is proposed for the quadrotor,where an adaptive tracking controller is designed by leveraging backstepping and adaptive techniques.Stable and accurate object tracking is achieved even under disturbances.Experimental results are presented to demonstrate the high accuracy and stability of the whole tracking system.     

Team NimbRo at MBZIRC 2017: Fast landing on a moving target and treasure hunting with a team of micro aerial vehicles

The Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2017 has defined ambitious new benchmarks to advance the state‐of‐the‐art in autonomous operation of ground‐based and flying robots. This study covers our approaches to solve the two challenges that involved micro aerial vehicles (MAV). Challenge 1 required reliable target perception, fast trajectory planning, and stable control of an MAV to land on a moving vehicle. Challenge 3 demanded a team of MAVs to perform a search and transportation task, coined “Treasure Hunt,” which required mission planning and multirobot coordination as well as adaptive control to account for the additional object weight. We describe our base MAV setup and the challenge‐specific extensions, cover the camera‐based perception, explain control and trajectory‐planning in detail, and elaborate on mission planning and team coordination. We evaluated our systems in simulation as well as with real‐robot experiments during the competition in Abu Dhabi. With our system, we—as part of the larger team NimbRo—won the MBZIRC Grand Challenge and achieved a third place in both subchallenges involving flying robots.     

自主移动机器人足球比赛视觉定位方法综述

综述了RoboCup足球赛中全自主移动机器人基于视觉的定位技术,包括机器人自定位和多机器人协作物体定位.介绍了定位技术的发展情况与分类.从机器人环境构建形式的不同以及先验位姿和概率方法的应用与否等方面,系统地分析和比较了各种自定位方法.对于多机器人协作物体定位,阐述了静态方法和动态跟踪方法.总结了定位过程中需要重点研究的传感器模型构建、图像处理、特征匹配以及协作过程涉及的相关问题.最后就视觉定位存在的问题和技术发展趋势进行了讨论.     

Autonomous Navigation for Micro Aerial Vehicles in Complex GNSS-denied Environments

Micro aerial vehicles, such as multirotors, are particular well suited for the autonomous monitoring, inspection, and surveillance of buildings, e.g., for maintenance in industrial plants. Key prerequisites for the fully autonomous operation of micro aerial vehicles in restricted environments are 3D mapping, real-time pose tracking, obstacle detection, and planning of collision-free trajectories. In this article, we propose a complete navigation system with a multimodal sensor setup for omnidirectional environment perception. Measurements of a 3D laser scanner are aggregated in egocentric local multiresolution grid maps. Local maps are registered and merged to allocentric maps in which the MAV localizes. For autonomous navigation, we generate trajectories in a multi-layered approach: from mission planning over global and local trajectory planning to reactive obstacle avoidance. We evaluate our approach in a GNSS-denied indoor environment where multiple collision hazards require reliable omnidirectional perception and quick navigation reactions.     

Autonomous Landing of MAVs on an Arbitrarily Textured Landing Site Using Onboard Monocular Vision

This paper presents a novel solution for micro aerial vehicles (MAVs) to autonomously search for and land on an arbitrary landing site using real-time monocular vision. The autonomous MAV is provided with only one single reference image of the landing site with an unknown size before initiating this task. We extend a well-known monocular visual SLAM algorithm that enables autonomous navigation of the MAV in unknown environments, in order to search for such landing sites. Furthermore, a multi-scale ORB feature based method is implemented and integrated into the SLAM framework for landing site detection. We use a RANSAC-based method to locate the landing site within the map of the SLAM system, taking advantage of those map points associated with the detected landing site. We demonstrate the efficiency of the presented vision system in autonomous flights, both indoor and in challenging outdoor environment.     

Ensuring visibility in calibration-free path planning for image-based visual servoing

We consider the problem of planning a path for image-based eye-in-hand visual servoing applications, when using an uncalibrated camera. Recent developments in this field lead to analytical interpolation between two views of an unknown object, in the projective space. Literature also provides extensions of these approaches that account for the target visibility constraint. Unfortunately, these extensions require an (even weakly) calibrated camera, and involve iterative algorithms that are not proven to converge. In this paper, we propose a modification of an existing projective interpolation algorithm that guarantees the visibility of the observed target, while avoiding the use of any knowledge on the camera calibration parameters, nor on the observed object.     

On intensity-based 3D visual servoing

This article investigates the problem of pose-based visual servoing whose equilibrium state is defined via a reference image. Differently from most solutions, this work directly exploits the pixel intensities without any feature extraction or matching. Intensity-based methods provide for higher accuracy and versatility. Another central idea of this work concerns the exploitation of the observability issue associated to monocular systems, which always occurs around the equilibrium. This overall framework allows for developing a family of new 3D visual servoing techniques with varying degrees of computational complexity and of prior knowledge. Importantly, they are arranged hierarchically from the simplest to the optimal ones, all in a unified scheme. Three new methods are then presented, and their closed-loop performances are experimentally assessed using a six degree-of-freedom robotic arm. As an additional contribution, these results refute the common belief that correct camera calibration and pose recovery are crucial to the accuracy of 3D visual servoing techniques.     

Virtual Visual Servoing: a framework for real-time augmented reality

This paper presents a framework to achieve real‐time augmented reality applications. We propose a framework based on the visual servoing approach well known in robotics. We consider pose or viewpoint computation as a similar problem to visual servoing. It allows one to take advantage of all the research that has been carried out in this domain in the past. The proposed method features simplicity, accuracy, efficiency, and scalability wrt. to the camera model as well as wrt. the features extracted from the image. We illustrate the efficiency of our approach on augmented reality applications with various real image sequences.     

利用平面单应分解实现服务机器人视觉伺服

智能空间中家庭服务机器人所需完成的主要任务是协助人完成物品的搜寻、定位与传递。而视觉伺服则是完成上述任务的有效手段。搭建了由移动机器人、机械臂、摄像头组成的家庭服务机器人视觉伺服系统,建立了此系统的运动学模型并对安装在机械臂末端执行器上的视觉系统进行了内外参数标定,通过分解世界平面的单应来获取目标物品的位姿参数,利用所获取的位姿参数设计了基于位置的视觉伺服控制律。实验结果表明,使用平面单应分解方法来设计控制律可简单有效地完成家庭物品的视觉伺服任务。     

2 1/2 D Visual Servoing with Respect to Unknown Objects Through a New Estimation Scheme of Camera Displacement

Classical visual servoing techniques need a strong a priori knowledge of the shape and the dimensions of the observed objects. In this paper, we present how the 2 1/2 D visual servoing scheme we have recently developed, can be used with unknown objects characterized by a set of points. Our scheme is based on the estimation of the camera displacement from two views, given by the current and desired images. Since vision-based robotics tasks generally necessitate to be performed at video rate, we focus only on linear algorithms. Classical linear methods are based on the computation of the essential matrix. In this paper, we propose a different method, based on the estimation of the homography matrix related to a virtual plane attached to the object. We show that our method provides a more stable estimation when the epipolar geometry degenerates. This is particularly important in visual servoing to obtain a stable control law, especially near the convergence of the system. Finally, experimental results confirm the improvement in the stability, robustness, and behaviour of our scheme with respect to classical methods.     

Measurement errors in visual servoing

This paper addresses the issue of measurement errors in visual servoing. The error characteristics of the vision based state estimation and the associated uncertainty of the control are investigated. The major contribution is the analysis of the propagation of image error through pose estimation and visual servoing control law. Using the analysis, two classical visual servoing methods are evaluated: position-based and 2.5D visual servoing. The evaluation offers a tool to build and analyze hybrid control systems such as switching or partitioning control.     

An open-source navigation system for micro aerial vehicles

This paper presents an open-source indoor navigation system for quadrotor micro aerial vehicles (MAVs), implemented in the ROS framework. The system requires a minimal set of sensors including a planar laser range-finder and an inertial measurement unit. We address the issues of autonomous control, state estimation, path-planning, and teleoperation, and provide interfaces that allow the system to seamlessly integrate with existing ROS navigation tools for 2D SLAM and 3D mapping. All components run in real time onboard the MAV, with state estimation and control operating at 1 kHz. A major focus in our work is modularity and abstraction, allowing the system to be both flexible and hardware-independent. All the software and hardware components which we have developed, as well as documentation and test data, are available online.     

Dynamic visual servoing of a small scale autonomous helicopter in uncalibrated environments

This paper presents a novel adaptive controller for image-based visual servoing of a small autonomous helicopter to cope with uncalibrated camera parameters and unknown 3D geometry of the feature points.The controller is based on the back-stepping technique,but its design has two new features.First,it incorporates the visual feedback into the last step of the backstepping procedure,while existing backstepping-based methods employ the visual feedback at the early steps.Second,the controller maps the image er...