Estimating the kinematics and structure of a rigid object from asequence of monocular images

The problem considered involves the use of a sequence of noisy monocular images of a three-dimensional moving object to estimate both its structure and kinematics. The object is assumed to be rigid, and its motion is assumed to be smooth. A set of object match points is assumed to be available, consisting of fixed features on the object, the image plane coordinates of which have been extracted from successive images in the sequence. Structure is defined as the 3-D positions of these object feature points, relative to each other. Rotational motion occurs about the origin of an object-centered coordinate system, while translational motion is that of the origin of this coordinate system. In this work, which is a continuation of the research done by the authors and reported previously (ibid., vol.PAMI-8, p.90-9, Jan. 1986), results of an experiment with real imagery are presented, involving estimation of 28 unknown translational, rotational, and structural parameters, based on 12 images with seven feature points     

Augmented reality with automatic illumination control incorporatingellipsoidal models

In applications of augmented reality like virtual studio TV production, multisite video conference applications using a virtual meeting room and synthetic/natural hybrid coding according to the new ISO/MPEG-4 standard, a synthetic scene is mixed into a natural scene to generate a synthetic/natural hybrid image sequence. For realism, the illumination in both scenes should be identical. In this paper, the illumination of the natural scene is estimated automatically and applied to the synthetic scene. The natural scenes are restricted to scenes with nonoccluding, simple, moving, mainly rigid objects. For illumination estimation, these natural objects are automatically segmented in the natural image sequence and three-dimensionally (3-D) modeled using ellipsoid-like models. The 3-D shape, 3-D motion, and the displaced frame difference between two succeeding images are evaluated to estimate three illumination parameters. The parameters describe a distant point light source and ambient light. Using the estimated illumination parameters, the synthetic scene is rendered and mixed to the natural image sequence. Experimental results with a moving virtual object mixed into real video telephone sequences show that the virtual object appears naturally having the same shading and shadows as the real objects. Further, shading and shadow allows the viewer to understand the motion trajectory of the objects much better     

Recursive-Batch Estimation of Motion and Structure from Monocular Image Sequences

This paper addresses the issue of optimal motion and structure estimation from monocular image sequences of a rigid scene. The new method has the following characteristics: (1) the dimension of the search space in the nonlinear optimization is drastically reduced by exploiting the relationship between structure and motion parameters; (2) the degree of reliability of the observations and estimates is effectively taken into account; (3) the proposed formulation allows arbitrary interframe motion; (4) the information about the structure of the scene, acquired from previous images, is systematically integrated into the new estimations; (5) the integration of multiple views using this method gives a large 2.5D visual map, much larger than that covered by any single view. It is shown also that the scale factor associated with any two consecutive images in a monocular sequence is determined by the scale factor of the first two images. Our simulation results and experiments with long image sequences of real world scenes indicate that the optimization method developed in this paper not only greatly reduces the computational complexity but also substantially improves the motion and structure estimates over those produced by the linear algorithms.     

Fusing Visual and Inertial Sensing to Recover Robot Ego‐motion

A method for estimating mobile robot ego‐motion is presented, which relies on tracking contours in real‐time images acquired with a calibrated monocular video system. After fitting an active contour to an object in the image, 3D motion is derived from the affine deformations suffered by the contour in an image sequence. More than one object can be tracked at the same time, yielding some different pose estimations. Then, improvements in pose determination are achieved by fusing all these different estimations. Inertial information is used to obtain better estimates, as it introduces in the tracking algorithm a measure of the real velocity. Inertial information is also used to eliminate some ambiguities arising from the use of a monocular image sequence. As the algorithms developed are intended to be used in real‐time control systems, considerations on computation costs are taken into account. © 2004 Wiley Periodicals, Inc.     

Reconstruction of a Scene with Multiple Linearly Moving Objects

In this paper we describe an algorithm to recover the scene structure, the trajectories of the moving objects and the camera motion simultaneously given a monocular image sequence. The number of the moving objects is automatically detected without prior motion segmentation. Assuming that the objects are moving linearly with constant speeds, we propose a unified geometrical representation of the static scene and the moving objects. This representation enables the embedding of the motion constraints into the scene structure, which leads to a factorization-based algorithm. We also discuss solutions to the degenerate cases which can be automatically detected by the algorithm. Extension of the algorithm to weak perspective projections is presented as well. Experimental results on synthetic and real images show that the algorithm is reliable under noise.     

基于遗传算法的直线光流刚体运动重建

建立一种新的基于直线光流场从单目图像序列恢复刚体运动和结构的模型,推导出直线光流场与刚体的运动参数之间的关系,用2个二阶线性微分方程表达这种关系,并提出一种求解刚体运动参数的遗传算法,只需要获得图像平面的2条直线光流即可求解刚体的旋转参数,并用合成图像测试了该算法的有效性。     

Automatic Reconstruction of Buildings from Stereoscopic Image Sequences

A vision–based 3-D scene analysis system is described that is capable to model complex real–world scenes like streets and buildings automatically from stereoscopic image pairs. Input to the system is a sequence of stereoscopic images taken with two standard CCD Cameras and TV lenses. The relative orientation of both cameras to each other is known by calibration. The camerapair is then moved throughout the scene and a long sequence of closely spaced views is recorded. Each of the stereoscopic image pairs is rectified and a dense map of 3-D suface points is obtained by area correlation, object segmentation, interpolation, and triangulation. 3-D camera motion relative to the scene coordinate system is tracked directly from the image sequence which allows to fuse 3-D surface measurements from different viewpoints into a consistent 3-D model scene. The surface geometry of each scene object is approximated by a triangular surface mesh which stores the suface texture in a texture map. From the textured 3-D models, realistic looking image sequences from arbitrary view points can be synthesized using computer graphics.     

3-D motion estimation by integrating visual cues in 2-D multi-modal opti-acoustic stereo sequences

Object reconstruction and target-based positioning are among critical capabilities in deploying submersible platforms for a range of underwater applications, e.g., search and inspection missions. Optical cameras provide high-resolution and target details, but their utility becomes constrained by the visibility range. In comparison, high-frequency (MHz) 2-D sonar imaging systems introduced to the commercial market in recent years can image targets at distances of tens of meters in highly turbid waters.Where fair visibility permits optical imaging at reasonable quality, the integration with 2-D sonar data can enable better performance compared to deploying either system alone, and thus enabling automated operation in a wider range of conditions.We investigate the estimation of 3-D motion by exploiting the visual cues in optical and sonar video for vision-based navigation and 3-D positioning of submersible platforms. The application of structure from motion paradigm in this multi-modal imaging scenario also enables the 3-D reconstruction of scene features. Our method does not require establishing multi-modal association between corresponding optical and sonar features, but rather the tracking of features in the sonar and optical motion sequences independently. In addition to improving the motion estimation accuracy, another advantage of the proposed method includes overcoming the inherent ambiguities of monocular vision, e.g., the scale-factor ambiguity and dual interpretation of motion relative to planar scenes. We discuss how our solution can also provide an effective strategy to address the complex opti-acoustic stereo matching problem. Experiment with synthetic and real data demonstrate the advantages of our technical contribution.     

基于多约束融合的人手臂三维运动分析

为了在没有任何特殊标志的情况下,实现从单目序列图象中分析、估计人手臂的三维运动,提出了一种多约束融合的方法,该方法是利用棍棒模型来模拟人的手臂,首先通过处理单目图象序列来自动获取图象序列中手臂关节点的对应;然后再利用多约束融合及基于图象序列中关节点的对应,即估计尺度意义下关节点的三维相对运动轨迹;最后利用真实图象来获得相应人手臂的三维运动轨迹,并将其与通过运动捕捉系统获得的人手臂的真实三维运动轨迹进行了比较实验。实验结果表明,该方法用于对人手臂的运动分析非常有效。     

基于二维关节型人体模型和EM算法的人体跟踪

提出一种跟踪单眼图像序列中的行人,并恢复其运动参数的新方法.在跟踪中采用了基于SPM(Sealed Prismat Model)扩展的二维纸板人模型取代三维人体模型,以获取更快的计算速度.作者使用EM算法在概率框架下进行运动估计,同时,算法也考虑了混合的运动模型和运动约束,以减小解的搜索空间.试验结果证明了该方法的有效性.     

Passive ranging using a moving camera

This article addresses the problem of determining the 3-dimensional locations of salient points in the environment of a moving camera based on a monocular image sequence obtained by the camera. The camera's translational and rotational velocities are assumed to be known approximately via inertial sensors. The motion of the camera is represented by a constant velocity model. Salient points in the image sequence are extracted using Gabor wavelets and tracked using labeled graph matching. The 3-D positions of the selected environmental points relative to the camera are then estimated recursively using an extended Kalman filter (EKF), after initialization by two-frame motion stereo. The motion parameters of the camera are also refined simultaneously. Experimental results on real data are given. © 1992 John Wiley & Sons, Inc.     

人体运动分析的实例学习方法

目的 面向实时、准确、鲁棒的人体运动分析应用需求,从运动分析的特征提取和运动建模问题出发,本文人体运动分析的实例学习方法。方法 在构建人体姿态实例库基础上,首先,采用运动检测方法得到视频每帧的人体轮廓;其次,基于形状上下文轮廓匹配方法,从实例库中检索得到每帧视频的候选姿态集;最后,通过统计建模和转移概率建模实现人体运动分析。结果 对步行、跑步、跳跃等测试视频进行实验,基于轮廓的形状上下文特征表示和匹配方法具有良好的表达能力;本文方法运动分析结果,关节夹角平均误差在5°左右,与其他算法相比,有效提高了运动分析的精度。结论 本文人体运动分析的实例学习方法,能有效分析单目视频中的人体运动,并克服了映射的深度歧义,对运动的视角变化鲁棒,具有良好的计算效率和精度。     

Non-Rigid Stereo Factorization

In this paper we address the problem of recovering 3D non-rigid structure from a sequence of images taken with a stereo pair. We have extended existing non-rigid factorization algorithms to the stereo camera case and presented an algorithm to decompose the measurement matrix into the motion of the left and right cameras and the 3D shape, represented as a linear combination of basis-shapes. The added constraints in the stereo camera case are that both cameras are viewing the same structure and that the relative orientation between both cameras is fixed. Our focus in this paper is on the recovery of flexible 3D shape rather than on the correspondence problem. We propose a method to compute reliable 3D models of deformable structure from stereo images. Our experiments with real data show that improved reconstructions can be achieved using this method. The algorithm includes a non-linear optimization step that minimizes image reprojection error and imposes the correct structure to the motion matrix by choosing an appropriate parameterization. We show that 3D shape and motion estimates can be successfully disambiguated after bundle adjustment and demonstrate this on synthetic and real image sequences. While this optimization step is proposed for the stereo camera case, it can be readily applied to the case of non-rigid structure recovery using a monocular video sequence. Electronic supplementary material Electronic supplementary material is available for this article at and accessible for authorised users.     

A Local Algorithm for the Computation of Image Velocity via Constructive Interference of Global Fourier Components

A novel Fourier-based technique for local motion detection from image sequences is proposed. In this method, the instantaneous velocities of local image points are inferred directly from the global 3D Fourier components of the image sequence. This is done by selecting those velocities for which the superposition of the corresponding Fourier gratings leads to constructive interference at the image point. Hence, image velocities can be assigned locally even though position is computed from the phases and amplitudes of global Fourier components (spanning the whole image sequence) that have been filtered based on the motion-constraint equation, reducing certain aperture effects typically arising from windowing in other methods. Regularization is introduced for sequences having smooth flow fields. Aperture effects and their effect on optic-flow regularization are investigated in this context. The algorithm is tested on both synthetic and real image sequences and the results are compared to those of other local methods. Finally, we show that other motion features, i.e. motion direction, can be computed using the same algorithmic framework without requiring an intermediate representation of local velocity, which is an important characteristic of the proposed method.     

3D head tracking under partial occlusion

A new algorithm for 3D head tracking under partial occlusion from 2D monocular image sequences is proposed. The extended superquadric (ESQ) is used to generate a geometric 3D face model in order to reduce the shape ambiguity during tracking. Optical flow is then regularized by this model to estimate the 3D rigid motion. To deal with occlusion, a new motion segmentation algorithm using motion residual error analysis is developed. The occluded areas are successfully detected and discarded as noise. Furthermore, accumulation error is heavily reduced by a new post-regularization process based on edge flow. This makes the algorithm more stable over long image sequences. The algorithm is applied to both synthetic occlusion sequence and real image sequences. Comparisons with the ground truth indicate that our method is effective and is not sensitive to occlusion during head tracking.     

Motion analysis of isolated targets in infrared image sequences

We describe a simple algorithm for estimating the 3-D motion of isolated targets in infrared image sequences. The algorithm is based on analyzing the changes in the shapes and the relative positions of segmented image regions. Experimental results of applying this algorithm to single IR images and to an image sequence are presented.     

基于ORB特征的单目视觉定位算法研究

针对大范围室外场景和具有重复、高频纹理特征(例如水泥地、草坪)的场景,提出了一种鲁棒性强、定位精度高、速度更快的视觉定位算法。采用8级图像金字塔的ORB (Oriented FAST and Rotated BRIEF)特征描述子提取图像特征点,通过K近邻(KNN)匹配相邻图像序列的特征点对,依次解算基础矩阵F和本质矩阵E,采用自适应法利用单应矩阵和本质矩阵进行位姿估计,最后解算两帧图像间相机刚体运动的旋转R和平移t,利用三角测量法则求解出匹配点的三维坐标,重建相机运动轨迹。为了提高算法性能,提出采用最小化基于点特征的非线性重投影误差优化三维点。通过调用OpenCV在C++中实现,对所采集的数据集进行测试,测试结果表明,该方法比传统的3D位姿估计更优,实时可行。由于其基于单目而实现,因此无法得到尺度信息。     

Tracking nonrigid motion and structure from 2D satellite cloudimages without correspondences

Tracking both structure and motion of nonrigid objects from monocular images is an important problem in vision. In this paper, a hierarchical method which integrates local analysis (that recovers small details) and global analysis (that appropriately limits possible nonrigid behaviors) is developed to recover dense depth values and nonrigid motion from a sequence of 2D satellite cloud images without any prior knowledge of point correspondences. This problem is challenging not only due to the absence of correspondence information but also due to the lack of depth cues in the 2D cloud images (scaled orthographic projection). In our method, the cloud images are segmented into several small regions and local analysis is performed for each region. A recursive algorithm is proposed to integrate local analysis with appropriate global fluid model constraints, based on which a structure and motion analysis system, SMAS, is developed. We believe that this is the first reported system in estimating dense structure and nonrigid motion under scaled orthographic views using fluid model constraints. Experiments on cloud image sequences captured by meteorological satellites (GOES-8 and GOES-9) have been performed using our system, along with their validation and analyses. Both structure and 3D motion correspondences are estimated to subpixel accuracy. Our results are very encouraging and have many potential applications in earth and space sciences, especially in cloud models for weather prediction