Extrinsic calibration of heterogeneous cameras by line images

The extrinsic calibration refers to determining the relative pose of cameras. Most of the approaches for cameras with non-overlapping fields of view (FOV) are based on mirror reflection, object tracking or rigidity constraint of stereo systems whereas cameras with overlapping FOV can be calibrated using structure from motion solutions. We propose an extrinsic calibration method within structure from motion framework for cameras with overlapping FOV and its extension to cameras with partially non-overlapping FOV. Recently, omnidirectional vision has become a popular topic in computer vision as an omnidirectional camera can cover large FOV in one image. Combining the good resolution of perspective cameras and the wide observation angle of omnidirectional cameras has been an attractive trend in multi-camera system. For this reason, we present an approach which is applicable to heterogeneous types of vision sensors. Moreover, this method utilizes images of lines as these features possess several advantageous characteristics over point features, especially in urban environment. The calibration consists of a linear estimation of orientation and position of cameras and optionally bundle adjustment to refine the extrinsic parameters.     

ARDEA—An MAV with skills for future planetary missions

We introduce a prototype flying platform for planetary exploration: autonomous robot design for extraterrestrial applications (ARDEA). Communication with unmanned missions beyond Earth orbit suffers from time delay, thus a key criterion for robotic exploration is a robot's ability to perform tasks without human intervention. For autonomous operation, all computations should be done on‐board and Global Navigation Satellite System (GNSS) should not be relied on for navigation purposes. Given these objectives ARDEA is equipped with two pairs of wide‐angle stereo cameras and an inertial measurement unit (IMU) for robust visual‐inertial navigation and time‐efficient, omni‐directional 3D mapping. The four cameras cover a $240^{°}$ vertical field of view, enabling the system to operate in confined environments such as caves formed by lava tubes. The captured images are split into several pinhole cameras, which are used for simultaneously running visual odometries. The stereo output is used for simultaneous localization and mapping, 3D map generation and collision‐free motion planning. To operate the vehicle efficiently for a variety of missions, ARDEA's capabilities have been modularized into skills which can be assembled to fulfill a mission's objectives. These skills are defined generically so that they are independent of the robot configuration, making the approach suitable for different heterogeneous robotic teams. The diverse skill set also makes the micro aerial vehicle (MAV) useful for any task where autonomous exploration is needed. For example terrestrial search and rescue missions where visual navigation in GNSS‐denied indoor environments is crucial, such as partially collapsed man‐made structures like buildings or tunnels. We have demonstrated the robustness of our system in indoor and outdoor field tests.     

3D model based tracking for omnidirectional vision: A new spherical approach

The current work addresses the problem of 3D model tracking in the context of monocular and stereo omnidirectional vision in order to estimate the camera pose. To this end, we track 3D objects modeled by line segments because the straight line feature is often used to model the environment. Indeed, we are interested in mobile robot navigation using omnidirectional vision in structured environments. In the case of omnidirectional vision, 3D straight lines are projected as conics in omnidirectional images. Under certain conditions, these conics may have singularities.In this paper, we present two contributions. We, first, propose a new spherical formulation of the pose estimation withdrawing singularities, using an object model composed of lines. The theoretical formulation and the validation on synthetic images thus show that the new formulation clearly outperforms the former image plane one. The second contribution is the extension of the spherical representation to the stereovision case. We consider in the paper a sensor which combines a camera and four mirrors. Results in various situations show the robustness to illumination changes and local mistracking. As a final result, the proposed new stereo spherical formulation allows us to localize online a robot indoor and outdoor whereas the classical formulation fails.     

Towards depth estimation in a single aerial image

ABSTRACT

Nowadays, depth estimation from a single image is a task that has been successfully addressed by Convolutional Neural Network (CNN) architectures. In this regard, several authors have taken advantage of depth datasets publicly available to the scientific community to train their CNN-based methods. From a project of Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago has emerged KITTI (acronym derived from the institutions' names) as one of the most popular public datasets providing depth estimates associated to RGB (Red, Green, Blue) images. Regarding the depth data in KITTI and typically in many other datasets, these include monocular or stereo RGB images associated with depth images obtained via laser, stereo cameras or a combination of both. These images and depth data have been collected by driving around outdoor urban environments with cameras looking forward to the horizon. In contrast, in this work, we are interested in CNN-based depth estimation in a single aerial image for which depth datasets are not available. In addition, popular CNN architectures for depth estimation in a single-image struggle to estimate depth in aerial scenes due to the fact that the camera angle and object appearance in aerial imagery are significantly different. Nevertheless, we propose to harvest the depth information available in KITTI in order to tackle the problem of depth estimation in a single aerial image. To this end, our approach is a two-step methodology based on patch processing that is later used as input for a set of proposed CNN architectures. Our results indicate that this approach is promising, and those datasets such as KITTI may indeed be exploited in other domains, especially where the data acquisition may be expensive or difficult to be carried out such as for aerial scenes.     

Autonomous Navigation for Micro Aerial Vehicles in Complex GNSS-denied Environments

Micro aerial vehicles, such as multirotors, are particular well suited for the autonomous monitoring, inspection, and surveillance of buildings, e.g., for maintenance in industrial plants. Key prerequisites for the fully autonomous operation of micro aerial vehicles in restricted environments are 3D mapping, real-time pose tracking, obstacle detection, and planning of collision-free trajectories. In this article, we propose a complete navigation system with a multimodal sensor setup for omnidirectional environment perception. Measurements of a 3D laser scanner are aggregated in egocentric local multiresolution grid maps. Local maps are registered and merged to allocentric maps in which the MAV localizes. For autonomous navigation, we generate trajectories in a multi-layered approach: from mission planning over global and local trajectory planning to reactive obstacle avoidance. We evaluate our approach in a GNSS-denied indoor environment where multiple collision hazards require reliable omnidirectional perception and quick navigation reactions.     

Vision-aided Estimation of Attitude,Velocity, and Inertial Measurement Bias for UAV Stabilization

This paper studies vision-aided inertial navigation of small-scale unmanned aerial vehicles (UAVs) in GPS-denied environments. The objectives of the navigation system are to firstly online estimate and compensate the unknown inertial measurement biases, secondly provide drift-free velocity and attitude estimates which are crucial for UAV stabilization control, and thirdly give relatively accurate position estimation such that the UAV is able to perform at least a short-term navigation when the GPS signal is not available. For the vision system, we do not presume maps or landmarks of the environment. The vision system should be able to work robustly even given low-resolution images (e.g., 160 ×120 pixels) of near homogeneous visual features. To achieve these objectives, we propose a novel homography-based vision-aided navigation system that adopts four common sensors: a low-cost inertial measurement unit, a downward-looking monocular camera, a barometer, and a compass. The measurements of the sensors are fused by an extended Kalman filter. Based on both analytical and numerical observability analyses of the navigation system, we theoretically verify that the proposed navigation system is able to achieve the navigation objectives. We also show comprehensive simulation and real flight experimental results to verify the effectiveness and robustness of the proposed navigation system.     

Bearing-only visual SLAM for small unmanned aerial vehicles in GPS-denied environments

This paper presents a hierarchical simultaneous localization and mapping(SLAM) system for a small unmanned aerial vehicle(UAV) using the output of an inertial measurement unit(IMU) and the bearing-only observations from an onboard monocular camera.A homography based approach is used to calculate the motion of the vehicle in 6 degrees of freedom by image feature match.This visual measurement is fused with the inertial outputs by an indirect extended Kalman filter(EKF) for attitude and velocity estimation.Then,another EKF is employed to estimate the position of the vehicle and the locations of the features in the map.Both simulations and experiments are carried out to test the performance of the proposed system.The result of the comparison with the referential global positioning system/inertial navigation system(GPS/INS) navigation indicates that the proposed SLAM can provide reliable and stable state estimation for small UAVs in GPS-denied environments.     

Keyframe‐based thermal–inertial odometry

Autonomous navigation of microaerial vehicles in environments that are simultaneously GPS‐denied and visually degraded, and especially in the dark, texture‐less and dust‐ or smoke‐filled settings, is rendered particularly hard. However, a potential solution arises if such aerial robots are equipped with long wave infrared thermal vision systems that are unaffected by darkness and can penetrate many types of obscurants. In response to this fact, this study proposes a keyframe‐based thermal–inertial odometry estimation framework tailored to the exact data and concepts of operation of thermal cameras. The front‐end component of the proposed solution utilizes full radiometric data to establish reliable correspondences between thermal images, as opposed to operating on rescaled data as previous efforts have presented. In parallel, taking advantage of a keyframe‐based optimization back‐end the proposed method is suitable for handling periods of data interruption which are commonly present in thermal cameras, while it also ensures the joint optimization of reprojection errors of 3D landmarks and inertial measurement errors. The developed framework was verified with respect to its resilience, performance, and ability to enable autonomous navigation in an extensive set of experimental studies including multiple field deployments in severely degraded, dark, and obscurants‐filled underground mines.     

Selective rendering for efficient ray traced stereoscopic images

Depth-related visual effects are a key feature of many virtual environments. In stereo-based systems, the depth effect can be produced by delivering frames of disparate image pairs, while in monocular environments, the viewer has to extract this depth information from a single image by examining details such as perspective and shadows. This paper investigates via a number of psychophysical experiments, whether we can reduce computational effort and still achieve perceptually high-quality rendering for stereo imagery. We examined selectively rendering the image pairs by exploiting the fusing capability and depth perception underlying human stereo vision. In ray-tracing-based global illumination systems, a higher image resolution introduces more computation to the rendering process since many more rays need to be traced. We first investigated whether we could utilise the human binocular fusing ability and significantly reduce the resolution of one of the image pairs and yet retain a high perceptual quality under stereo viewing condition. Secondly, we evaluated subjects’ performance on a specific visual task that required accurate depth perception. We found that subjects required far fewer rendered depth cues in the stereo viewing environment to perform the task well. Avoiding rendering these detailed cues saved significant computational time. In fact it was possible to achieve a better task performance in the stereo viewing condition at a combined rendering time for the image pairs less than that required for the single monocular image. The outcome of this study suggests that we can produce more efficient stereo images for depth-related visual tasks by selective rendering and exploiting inherent features of human stereo vision.     

环视鱼眼图像处理深度学习研究进展

视觉环境感知在自动驾驶汽车发展中起着关键作用,在智能后视镜、倒车雷达、360°全景、行车记录仪、碰撞预警、红绿灯识别、车道偏移、并线辅助和自动泊车等领域也有着广泛运用。传统的环境信息获取方式是窄角针孔摄像头,视野有限有盲区,解决这个问题的方法是环境信息感知使用鱼眼镜头,广角视图能够提供整个180°的半球视图,理论上仅需两个摄像头即可覆盖360°,为视觉感知提供更多信息。处理环视图像目前主要有两种途径：一是对图像先纠正,去失真,缺点是图像去失真会损害图像质量,并导致信息丢失;二是直接对形变的鱼眼图像进行建模,但目前还没有效果比较好的建模方法。此外,环视鱼眼图像数据集的缺乏也是制约相关研究的一大难题。针对上述挑战,本文总结了环视鱼眼图像的相关研究,包括环视鱼眼图像的校正处理、环视鱼眼图像中的目标检测、环视鱼眼图像中的语义分割、伪环视鱼眼图像数据集生成方法和其他鱼眼图像建模方法等,结合自动驾驶汽车的环境感知应用背景,分析了这些模型的效率和这些处理方法的优劣,并对目前公开的环视鱼眼图像通用数据集进行了详细介绍,对环视鱼眼图像中待解决的问题与未来研究方向做出预测和展望。     

Analysis of depth estimation error for cylindrical stereo imaging

Conventional stereo imaging uses area CCDs for which depth perception error has been analyzed in our past research (Comput. Vision Image understand. 63 (3) (1996) 447). In this work we analyze the depth perception error for a stereo system that uses two rotating linear CCD cameras to create cylindrical stereo images. Theoretical analysis shows certain advantages to cylindrical stereo imaging over conventional methods. Initial experimental results are presented to validate the theoretical results. Unlike normal area CCDs, cylindrical images captured by rotating linear CCD cameras can produce very high-resolution images thereby substantially reducing the error in 3D estimation.     

Aerial full spherical HDR imaging and display

This paper describes a framework for aerial imaging of high dynamic range (HDR) scenes for use in virtual reality applications, such as immersive panorama applications and photorealistic superimposition of virtual objects using image-based lighting. We propose a complete and practical system to acquire full spherical HDR images from the sky, using two omnidirectional cameras mounted above and below an unmanned aircraft. The HDR images are generated by combining multiple omnidirectional images captured with different exposures controlled automatically. Our system consists of methods for image completion, alignment, and color correction, as well as a novel approach for automatic exposure control, which selects optimal exposure so as to avoid banding artifacts. Experimental results indicated that our system generated better spherical images compared to an ordinary spherical image completion system in terms of naturalness and accuracy. In addition to proposing an imaging method, we have carried out an experiment about display methods for aerial HDR immersive panoramas utilizing spherical images acquired by the proposed system. The experiment demonstrated HDR imaging is beneficial to immersive panorama using an HMD, in addition to ordinary uses of HDR images.     

Weaver: Hexapod robot for autonomous navigation on unstructured terrain

Legged robots are an efficient alternative for navigation in challenging terrain. In this paper we describe Weaver, a six‐legged robot that is designed to perform autonomous navigation in unstructured terrain. It uses stereo vision and proprioceptive sensing based terrain perception for adaptive control while using visual‐inertial odometry for autonomous waypoint‐based navigation. Terrain perception generates a minimal representation of the traversed environment in terms of roughness and step height. This reduces the complexity of the terrain model significantly, enabling the robot to feed back information about the environment into its controller. Furthermore, we combine exteroceptive and proprioceptive sensing to enhance the terrain perception capabilities, especially in situations in which the stereo camera is not able to generate an accurate representation of the environment. The adaptation approach described also exploits the unique properties of legged robots by adapting the virtual stiffness, stride frequency, and stride height. Weaver's unique leg design with five joints per leg improves locomotion on high gradient slopes, and this novel configuration is further analyzed. Using these approaches, we present an experimental evaluation of this fully self‐contained hexapod performing autonomous navigation on a multiterrain testbed and in outdoor terrain.     

Robust and Accurate Monocular Visual Navigation Combining IMU for a Quadrotor

In this paper, we present a multi-sensor fusion based monocular visual navigation system for a quadrotor with limited payload, power and computational resources. Our system is equipped with an inertial measurement unit (IMU), a sonar and a monocular down-looking camera. It is able to work well in GPS-denied and markerless environments. Different from most of the keyframe-based visual navigation systems, our system uses the information from both keyframes and keypoints in each frame. The GPU-based speeded up robust feature (SURF) is employed for feature detection and feature matching. Based on the flight characteristics of quadrotor, we propose a refined preliminary motion estimation algorithm combining IMU data. A multi-level judgment rule is then presented which is beneficial to hovering conditions and reduces the error accumulation effectively. By using the sonar sensor, the metric scale estimation problem has been solved. We also present the novel IMU+3P (IMU with three point correspondences) algorithm for accurate pose estimation. This algorithm transforms the 6-DOF pose estimation problem into a 4-DOF problem and can obtain more accurate results with less computation time. We perform the experiments of monocular visual navigation system in real indoor and outdoor environments. The results demonstrate that the monocular visual navigation system performing in real-time has robust and accurate navigation results of the quadrotor.      

A multicamera setup for generating stereo panoramic video

Traditional visual communication systems convey only two-dimensional (2-D) fixed field-of-view (FOV) video information. The viewer is presented with a series of flat, nonstereoscopic images, which fail to provide a realistic sense of depth. Furthermore, traditional video is restricted to only a small part of the scene, based on the director's discretion and the user is not allowed to "look around" in an environment. The objective of this work is to address both of these issues and develop new techniques for creating stereo panoramic video sequences. A stereo panoramic video sequence should be able to provide the viewer with stereo vision at any direction (complete 360-degree FOV) at video rates. In this paper, we propose a new technique for creating stereo panoramic video using a multicamera approach, thus creating a high-resolution output. We present a setup that is an extension of a previously known approach, developed for the generation of still stereo panoramas, and demonstrate that it is capable of creating high-resolution stereo panoramic video sequences. We further explore the limitations involved in a practical implementation of the setup, namely the limited number of cameras and the nonzero physical size of real cameras. The relevant tradeoffs are identified and studied.     

Analysis of Error in Depth Perception with Vergence and Spatially Varying Sensing

In stereo vision the depth of a 3-D point is estimated based on the position of its projections on the left and right images. The image plane of cameras that produces the images consists of discrete pixels. This discretization of images generates uncertainty in estimation of the depth at each 3-D point. In this paper, we investigate the effect of vergence and spatially varying resolution on the depth estimation error. First, vergence is studied when pairs of stereo images with uniform resolution are used. Then the problem is studied for a stereo system similar to that of humans, in which cameras have high resolution in the center and nonlinearly decreasing resolution toward the periphery. In this paper we are only concerned with error in depth perception, assuming that stereo matching is already done.     

A Multisensor Integration Approach toward Astronaut Navigation for Landed Lunar Missions

As experienced by Apollo lunar astronauts, spatial orientation can be affected significantly by lunar environmental conditions such as the moon's altered gravity, lack of an atmosphere, limited spatial references, and different level of reflectivity. To help overcome these challenges, a lunar astronaut navigation system called LASOIS (Lunar Astronaut Spatial Orientation and Information System) has been developed. It can significantly reduce spatial disorientation and improve real‐time navigation capability for astronauts exploring the lunar surface. LASOIS is capable of integrating satellite imagery and sensors mounted on the astronaut spacesuit (including inertial measurement units, stereo cameras, and pressure sensors) by an extended Kalman filter algorithm. The processed navigation information is presented through a wrist‐mounted display system. The system has been tested at three field experiment sites, including Moses Lake, WA, Black Lava Point, AZ, and Haleakala National Park, HI. It is demonstrated that the system has achieved an error rate (or relative accuracy) of 2.4% for astronaut navigation over a traverse of 6.1 km in a lunarlike environment.     

Evaluation of the Unified Model of the Sphere for Fisheye Cameras in Robotic Applications

Abstract

A wide field of view is required for many robotic vision tasks. Such an aperture may be acquired by a fisheye camera, which provides a full image compared to catadioptric visual sensors, and does not increase the size and the weakness of the imaging system with respect to perspective cameras. While a unified model exists for all central catadioptric systems, many different models, approximating the radial distortions, exist for fisheye cameras. It is shown in this paper that the unified projection model proposed for central catadioptric cameras is also valid for fisheye cameras in the context of robotic applications. This model consists of a projection onto a virtual unitary sphere followed by a perspective projection onto an image plane. This model is shown equivalent to almost all the fisheye models. Calibration with four cameras and partial Euclidean reconstruction are done using this model, and lead to persuasive results. Finally, an application to a mobile robot navigation task is proposed and correctly executed along a 200-m trajectory.     

3D visual perception for self-driving cars using a multi-camera system: Calibration,mapping, localization,and obstacle detection

Cameras are a crucial exteroceptive sensor for self-driving cars as they are low-cost and small, provide appearance information about the environment, and work in various weather conditions. They can be used for multiple purposes such as visual navigation and obstacle detection. We can use a surround multi-camera system to cover the full 360-degree field-of-view around the car. In this way, we avoid blind spots which can otherwise lead to accidents. To minimize the number of cameras needed for surround perception, we utilize fisheye cameras. Consequently, standard vision pipelines for 3D mapping, visual localization, obstacle detection, etc. need to be adapted to take full advantage of the availability of multiple cameras rather than treat each camera individually. In addition, processing of fisheye images has to be supported. In this paper, we describe the camera calibration and subsequent processing pipeline for multi-fisheye-camera systems developed as part of the V-Charge project. This project seeks to enable automated valet parking for self-driving cars. Our pipeline is able to precisely calibrate multi-camera systems, build sparse 3D maps for visual navigation, visually localize the car with respect to these maps, generate accurate dense maps, as well as detect obstacles based on real-time depth map extraction.