Effects of Errors in the Viewing Geometry on Shape Estimation

A sequence of images acquired by a moving sensor contains information about the three-dimensional motion of the sensor and the shape of the imaged scene. Interesting research during the past few years has attempted to characterize the errors that arise in computing 3D motion (egomotion estimation) as well as the errors that result in the estimation of the scene's structure (structure from motion). Previous research is characterized by the use of optic flow or correspondence of features in the analysis as well as by the employment of particular algorithms and models of the scene in recovering expressions for the resulting errors. This paper presents a geometric framework that characterizes the relationship between 3D motion and shape in the presence of errors. We examine how the three-dimensional space recovered by a moving monocular observer, whose 3D motion is estimated with some error, is distorted. We characterize the space of distortions by its level sets, that is, we characterize the systematic distortion via a family of iso-distortion surfaces, which describes the locus over which the depths of points in the scene in view are distorted by the same multiplicative factor. The framework introduced in this way has a number of applications: Since the visible surfaces have positive depth (visibility constraint), by analyzing the geometry of the regions where the distortion factor is negative, that is, where the visibility constraint is violated, we make explicit situations which are likely to give rise to ambiguities in motion estimation, independent of the algorithm used. We provide a uniqueness analysis for 3D motion analysis from normal flow. We study the constraints on egomotion, object motion, and depth for an independently moving object to be detectable by a moving observer, and we offer a quantitative account of the precision needed in an inertial sensor for accurate estimation of 3D motion.     

Tracking and Motion Estimation of the Articulated Object: a Hierarchical Kalman Filter Approach

Real-time motion capture plays a very important role in various applications, such as 3D interface for virtual reality systems, digital puppetry, and real-time character animation. In this paper we challenge the problem of estimating and recognizing the motion of articulated objects using theoptical motion capturetechnique. In addition, we present an effective method to control the articulated human figure in realtime.The heart of this problem is the estimation of 3D motion and posture of an articulated, volumetric object using feature points from a sequence of multiple perspective views. Under some moderate assumptions such as smooth motion and known initial posture, we develop a model-based technique for the recovery of the 3D location and motion of a rigid object using a variation of Kalman filter. The posture of the 3D volumatric model is updated by the 2D image flow of the feature points for all views. Two novel concepts – the hierarchical Kalman filter (KHF) and the adaptive hierarchical structure (AHS) incorporating the kinematic properties of the articulated object – are proposed to extend our formulation for the rigid object to the articulated one. Our formulation also allows us to avoid two classic problems in 3D tracking: the multi-view correspondence problem, and the occlusion problem. By adding more cameras and placing them appropriately, our approach can deal with the motion of the object in a very wide area. Furthermore, multiple objects can be handled by managing multiple AHSs and processing multiple HKFs.We show the validity of our approach using the synthetic data acquired simultaneously from the multiple virtual camera in a virtual environment (VE) and real data derived from a moving light display with walking motion. The results confirm that the model-based algorithm works well on the tracking of multiple rigid objects.     

Invariant surface and motion estimation from sparse range data

In this paper, we present a system for the estimation of the surface structure and the motion parameters of a free-flying object in a tele-robotics experiment. The system consists of two main components: (i) a vision-based invariant-surface and motion estimator and (ii) a Kalman filter state estimator. We present a new algorithm for motion estimation from sparse multi-sensor range data. The motion estimates from the vision-based estimator are input to a Kalman filter state estimator for continuously tracking a free-flying object in space under zero-gravity conditions. The predicted position and orientation parameters are then fed back to the vision module of the system and serve as an initial guess in the search for optimal motion parameters. The task of the vision module is two-fold: (i) estimating a piecewise-smooth surface from a single frame of multi-sensor data and (ii) determining the most likely (in the Bayesian sense) object motion that makes data in subsequent time frames to have been sampled from the same piecewise-smooth surface. With each incoming data frame, the piecewise-smooth surface is incrementally refined. The problem is formulated as an energy minimization and solved numerically resulting in a surface estimate invariant to 3D rigid motion and the vector of motion parameters. Performance of the system is depicted on simulated and real range data.     

单目视觉中基于IEKF,DD1及DD2滤波器的位姿和运动估计

用单摄像机所获取的二维(2D)图像来估计两坐标之间的相对位姿和运动在实际应用中是可取的,其难点是从物体的三维(3D)特征投影到2D图像特征的过程是一个非线性变换,把基于单目视觉的位姿和运动估计系统定义为一个非线性随机模型,分别以迭代扩展卡尔曼滤波器(IEKF)、一阶斯梯林插值滤波器(DD1)和二阶斯梯林插值滤波器(DD2)作非线性状态估计器来估计位姿和运动.为了验证每种估计器的相对优点,用文中所提方法对每种估计器都作了仿真实验,实验结果表明DD1和DD2滤波器的特性要比IEKF好.     

双目视觉基于修正卡尔曼滤波器的运动估计

研究了目标物体的远程运动估计.首先,建立了一种双目视觉系统的基于卡尔曼滤波器的目标物体运动估计的运动学模型,并且证明了双目视觉系统同步的各自连续两帧图像中至少三个对应图像点能完全确定刚性物体的运动参数和空间位置;然后,通过对状态向量中的速度分量进行再估计,提出了一种修正卡尔曼滤波器对目标物体远程运动估计的算法,与直接卡尔曼滤波器的远程运动估计相比,提高了估计的精度.将该方法运用到一种实时预测的实验中,其结果证明了该算法的有效性.     

3-Dimensional pose sensor algorithm for humanoid robot

In this paper, a 3D pose attitude estimation system using inertial sensors was developed to provide feedback motion and attitude information for a humanoid robot. It has a very effective switching structure and composed of three modules, a motion acceleration detector, a pseudo-accelerometer output estimator, and a linear acceleration estimator. The switching structure based on probability enables a tactful feedback loop for the extended Kalman filter inside the sensor system. Specially designed linear-rotation test equipment was built, and the experimental results showed its fast convergence to actual values in addition to its excellent responses. The output of the proposed 3D sensor can be transmitted to a humanoid at a frequency of 200 Hz.     

Motion estimation of 3-D objects using multisensor data fusion

This article presents an approach to estimate the general 3-D motion of a polyhedral object using multiple sensor data some of which may not provide sufficient information for the estimation of object motion. Motion can be estimated continuously from each sensor through the analysis of the instantaneous state of an object. The instantaneous state of an object is specified by the rotation, which is defined by a rotation axis and rotation angle, and the displacement of the center of rotation. We have introduced a method based on Moore-Penrose pseudoinverse theory to estimate the instantaneous state of an object, and a linear feedback estimation algorithm to approach the motion estimation. The motion estimated from each sensor is fused to provide more accurate and reliable information about the motion of an unknown object. The techniques of multisensor data fusion can be categorized into three methods: averaging, decision, and guiding. We present a fusion algorithm which combines averaging and decision. With the assumption that the motion is smooth, our approach can handle the data sequences from multiple sensors with different sampling times. We can also predict the next immediate object position and its motion. The simulation results show our proposed approach is advantageous in terms of accuracy, speed, and versatility.     

Fusion of IMU and Vision for Absolute Scale Estimation in Monocular SLAM

The fusion of inertial and visual data is widely used to improve an object??s pose estimation. However, this type of fusion is rarely used to estimate further unknowns in the visual framework. In this paper we present and compare two different approaches to estimate the unknown scale parameter in a monocular SLAM framework. Directly linked to the scale is the estimation of the object??s absolute velocity and position in 3D. The first approach is a spline fitting task adapted from Jung and Taylor and the second is an extended Kalman filter. Both methods have been simulated offline on arbitrary camera paths to analyze their behavior and the quality of the resulting scale estimation. We then embedded an online multi rate extended Kalman filter in the Parallel Tracking and Mapping (PTAM) algorithm of Klein and Murray together with an inertial sensor. In this inertial/monocular SLAM framework, we show a real time, robust and fast converging scale estimation. Our approach does not depend on known patterns in the vision part nor a complex temporal synchronization between the visual and inertial sensor.     

双视三维重建的高精度运动参数估计方法

从两幅透视图像恢复被摄目标的三维结构是计算机视觉最基本的任务之一,其中,运动估计算法的性能决定了最终的三维重建精度。首先讨论了双视成像的基本数学模型,并介绍了几种现有运动参数估计方法的基本原理和不足。随后,基于投影误差最小判决函数,提出了用于双像运动估计的改进非线性迭代优化方法。数值仿真结果表明,在大平移小旋转角及小平移大旋转角2种运动条件下,采用文中提出的方法,运动估计精度均有所提高。此外,根据运动参数的估计值对真实目标进行三维重建实验,结果表明尺度重建误差小于2%且角度误差在3°以内。     

Reconstruction of non-rigid 3D shapes from stereo-motion

Several non-rigid structure from motion methods have been proposed so far in order to recover both the motion and the non-rigid structure of an object. However, these monocular algorithms fail to give reliable 3D shape estimates when the overall rigid motion of the sequence is small. Aiming to overcome this limitation, in this paper we propose a novel approach for the 3D Euclidean reconstruction of deformable objects observed by an uncalibrated stereo rig. Using a stereo setup drastically improves the 3D model estimation when the observed 3D shape is mostly deforming without undergoing strong rigid motion. Our approach is based on the following steps. Firstly, the stereo system is automatically calibrated and used to compute metric rigid structures from pairs of views. Afterwards, these 3D shapes are aligned to a reference view using a RANSAC method in order to compute the mean shape of the object and to select the subset of points which have remained rigid throughout the sequence. The selected rigid points are then used to compute frame-wise shape registration and to robustly extract the motion parameters from frame to frame. Finally, all this information is used as initial estimates of a non-linear optimization which allows us to refine the initial solution and also to recover the non-rigid 3D model. Exhaustive results on synthetic and real data prove the performance of our proposal estimating motion, non-rigid models and stereo camera parameters even when there is no rigid motion in the original sequence.     

局部双目视差回归的目标距离估计

目的 双目视觉是目标距离估计问题的一个很好的解决方案。现有的双目目标距离估计方法存在估计精度较低或数据准备较繁琐的问题,为此需要一个可以兼顾精度和数据准备便利性的双目目标距离估计算法。方法 提出一个基于R-CNN（region convolutional neural network）结构的网络,该网络可以实现同时进行目标检测与目标距离估计。双目图像输入网络后,通过主干网络提取特征,通过双目候选框提取网络以同时得到左右图像中相同目标的包围框,将成对的目标框内的局部特征输入目标视差估计分支以估计目标的距离。为了同时得到左右图像中相同目标的包围框,使用双目候选框提取网络代替原有的候选框提取网络,并提出了双目包围框分支以同时进行双目包围框的回归;为了提升视差估计的精度,借鉴双目视差图估计网络的结构,提出了一个基于组相关和3维卷积的视差估计分支。结果 在KITTI（Karlsruhe Institute of Technology and Toyota Technological Institute）数据集上进行验证实验,与同类算法比较,本文算法平均相对误差值约为3.2%,远小于基于双目视差图估计算法（11.3%）,与基于3维目标检测的算法接近（约为3.9%）。另外,提出的视差估计分支改进对精度有明显的提升效果,平均相对误差值从5.1%下降到3.2%。通过在另外采集并标注的行人监控数据集上进行类似实验,实验结果平均相对误差值约为4.6%,表明本文方法可以有效应用于监控场景。结论 提出的双目目标距离估计网络结合了目标检测与双目视差估计的优势,具有较高的精度。该网络可以有效运用于车载相机及监控场景,并有希望运用于其他安装有双目相机的场景。     

A hierarchy of cameras for 3D photography

The view-independent visualization of 3D scenes is most often based on rendering accurate 3D models or utilizes image-based rendering techniques. To compute the 3D structure of a scene from a moving vision sensor or to use image-based rendering approaches, we need to be able to estimate the motion of the sensor from the recorded image information with high accuracy, a problem that has been well-studied. In this work, we investigate the relationship between camera design and our ability to perform accurate 3D photography, by examining the influence of camera design on the estimation of the motion and structure of a scene from video data. By relating the differential structure of the time varying plenoptic function to different known and new camera designs, we can establish a hierarchy of cameras based upon the stability and complexity of the computations necessary to estimate structure and motion. At the low end of this hierarchy is the standard planar pinhole camera for which the structure from motion problem is non-linear and ill-posed. At the high end is a camera, which we call the full field of view polydioptric camera, for which the motion estimation problem can be solved independently of the depth of the scene which leads to fast and robust algorithms for 3D Photography. In between are multiple view cameras with a large field of view which we have built, as well as omni-directional sensors.     

Plane-Based Calibration for Linear Cameras

Linear or 1D cameras are used in several areas such as industrial inspection and satellite imagery. Since 1D cameras consist of a linear sensor, a motion (usually perpendicular to the sensor orientation) is performed in order to acquire a full image. In this paper, we present a novel linear method to estimate the intrinsic and extrinsic parameters of a 1D camera using a planar object. As opposed to traditional calibration scheme based on 3D-2D correspondences of landmarks, our method uses homographies induced by the images of a planar object. The proposed algorithm is linear, simple and produces good results as shown by our experiments.     

Incremental model-based estimation using geometric constraints

We present a model-based framework for incremental, adaptive object shape estimation and tracking in monocular image sequences. Parametric structure and motion estimation methods usually assume a fixed class of shape representation (splines, deformable superquadrics, etc.) that is initialized prior to tracking. Since the model shape coverage is fixed a priori, the incremental recovery of structure is decoupled from tracking, thereby limiting both processes in their scope and robustness. In this work, we describe a model-based framework that supports the automatic detection and integration of low-level geometric primitives (lines) incrementally. Such primitives are not explicitly captured in the initial model, but are moving consistently with its image motion. The consistency tests used to identify new structure are based on trinocular constraints between geometric primitives. The method allows not only an increase in the model scope, but also improves tracking accuracy by including the newly recovered features in its state estimation. The formulation is a step toward automatic model building, since it allows both weaker assumptions on the availability of a prior shape representation and on the number of features that would otherwise be necessary for entirely bottom-up reconstruction. We demonstrate the proposed approach on two separate image-based tracking domains, each involving complex 3D object structure and motion.     

协同运动状态估计的多目标跟踪算法

多目标跟踪在视频分析场景中有着广泛的应用,如人机交互、虚拟现实、自动驾驶、视频监控和机器人导航等。多目标跟踪问题可以表示为在已有的检测数据上进行目标轨迹关联,检测算法的准确性对跟踪性能起着关键性的作用。在基于检测的目标跟踪框架中,提出了一种协同运动状态估计的跟踪算法,该算法主要关注相邻帧之间的数据关联,从目标检测、目标运动状态估计和数据关联这3个方面来直接解决多目标跟踪面临的挑战。首先,对于目标检测,采用Multi Scale Convolutional Neural Network(MS-CNN)算法作为检测器,这是因为深度学习在检测的效益上优于传统的机器学习方法;其次,为了更好地预测目标的运动状态和处理目标间的遮挡,针对不同状态的目标采取不同的运动估计方法: 采用核相关滤波来评估处于跟踪状态的目标的运动状态,当目标处于遮挡状态时,采用卡尔曼滤波做运动估计;最后,采用Kuhn-Munkres算法对检测目标和跟踪轨迹做数据关联。通过大量的实验证实了算法的有效性,且实验结果表明算法的准确性很高。     

Genetic programming extension to APF-based monocular human body pose estimation

New method of the human body pose estimation based on a single camera 2D observation is presented, aimed at smart surveillance related video analysis and action recognition. It employs 3D model of the human body, and genetic algorithm combined with annealed particle filter for searching the global optimum of model state, best matching the object’s 2D observation. Additionally, new motion cost metric is employed, considering current pose and history of the body movement, favouring the estimates with the lowest changes of motion speed comparing to previous poses. The “genetic memory” concept is introduced for the genetic processing of both current and past states of 3D model. State-of-the-art in the field of human body tracking is presented and discussed. Details of implemented method are described. Results of experimental evaluation of developed algorithm are included and discussed.     

Particle filters and occlusion handling for rigid 2D–3D pose tracking

In this paper, we address the problem of 2D–3D pose estimation. Specifically, we propose an approach to jointly track a rigid object in a 2D image sequence and to estimate its pose (position and orientation) in 3D space. We revisit a joint 2D segmentation/3D pose estimation technique, and then extend the framework by incorporating a particle filter to robustly track the object in a challenging environment, and by developing an occlusion detection and handling scheme to continuously track the object in the presence of occlusions. In particular, we focus on partial occlusions that prevent the tracker from extracting an exact region properties of the object, which plays a pivotal role for region-based tracking methods in maintaining the track. To this end, a dynamical choice of how to invoke the objective functional is performed online based on the degree of dependencies between predictions and measurements of the system in accordance with the degree of occlusion and the variation of the object’s pose. This scheme provides the robustness to deal with occlusions of an obstacle with different statistical properties from that of the object of interest. Experimental results demonstrate the practical applicability and robustness of the proposed method in several challenging scenarios.     

Motion estimation of elastic articulated objects from points and contours with volume invariable constraint

This paper presents a model of elastic articulated objects based on revolving conic surface and a method of model-based motion estimation. The model includes 3D object skeleton and deformable surfaces that can represent the deformation of human body surfaces. In each limb, surface deformation is represented by adjusting one or two deformation parameters. Then, the 3D deformation parameters are determined by corresponding 2D image points and contours with volume invariable constraint from a sequence of stereo images. The 3D motion parameters are estimated based on the 3D model. The algorithm presented in this paper includes model-based parameter estimation of motion and parameter determination of deformable surfaces.