Estimating camera and object translation in the presence of camera rotation

This paper deals with stereo camera-based estimation of sensor translation in the presence of modest sensor rotation and moving objects. It also deals with the estimation of object translation from a moving sensor. The approach is based on Gabor filters, direct passive navigation, and Kalman filters.Three difficult problems associated with the estimation of motion from an image sequence are solved. (1) The temporal correspondence problem is solved using multi-scale prediction and phase gradients. (2) Segmentation of the image measurements into groups belonging to stationary and moving objects is achieved using the Mahalanobis distance. (3) Compensation for sensor rotation is achieved by internally representing the inter-frame (short-term) rotation in a rigid-body model. These three solutions possess a circular dependency, forming a cycle of perception. A seeding process is developed to correctly initialize the cycle. An additional complication is the translation-rotation ambiguity that sometimes exists when sensor motion is estimated from an image velocity field. Temporal averaging using Kalman filters reduces the effect of motion ambiguities. Experimental results from real image sequences confirm the utility of the approach.Financial support from the Natural Science and Engineering Research Council (NSERC) of Canada is acknowledged.     

Computation of ego motion using the vertical cue

This paper describes the development and implementation of some layers of a line-segment-based module to recover ego motion while building a 3D map of the environment in which the absolute vertical is taken into account. We use a monocular sequence of images and 2D-line segments in this sequence. The proposed method reduces the disparity between two frames in such a way that 3D vision is simplified. In particular, the correspondence problem is simplified. Moreover, a estimation of the 3D rotation is provided.Using the vertical as a basic cue for 3D-orientation tremendously simplifies and improves the structure from motion paradigm, but the usual equations have to be worked out in a different way.An approach which combines ecological hypotheses and general rigid motion equations is presented, and the equations are derived and discussed in the case of small rigid motions. Algorithms, based on the minimization of theMahalanobis distance between two estimates, are given and their implementations discussed.Based on Computation of ego motion and structure from visual and inertial sensors using the vertical cue by T. Viéville and P.E.D.S. Facao and E. Clergue, which appeared in the Fourth International Conference on Computer Vision, Berlin, 1993 May 11–14, pages 591–598, ©1993 IEEE     

Monocular human motion tracking

Human motion tracking from monocular image sequences has been explored widely. However, a framework that addresses the variety of sensing conditions is lacking. In this paper, we present a simple, efficient, and robust method for recovering plausible 3D motion from a video without knowledge of the cameras parameters. Our method transforms the motion capture problem into a convex problem and employs a hierarchical geometrical solver for the minimization. This algorithm was applied to numerous synthetic and real image sequences with very encouraging results. Specifically, our results indicate that it can handle challenges posed by variation of lighting, partial self-occlusion, and rapid motion.Published online: 21 October 2004     

2D/3D image registration using regression learning

In computer vision and image analysis, image registration between 2D projections and a 3D image that achieves high accuracy and near real-time computation is challenging. In this paper, we propose a novel method that can rapidly detect an object’s 3D rigid motion or deformation from a 2D projection image or a small set thereof. The method is called CLARET (Correction via Limited-Angle Residues in External Beam Therapy) and consists of two stages: registration preceded by shape space and regression learning. In the registration stage, linear operators are used to iteratively estimate the motion/deformation parameters based on the current intensity residue between the target projection(s) and the digitally reconstructed radiograph(s) (DRRs) of the estimated 3D image. The method determines the linear operators via a two-step learning process. First, it builds a low-order parametric model of the image region’s motion/deformation shape space from its prior 3D images. Second, using learning-time samples produced from the 3D images, it formulates the relationships between the model parameters and the co-varying 2D projection intensity residues by multi-scale linear regressions. The calculated multi-scale regression matrices yield the coarse-to-fine linear operators used in estimating the model parameters from the 2D projection intensity residues in the registration. The method’s application to Image-guided Radiation Therapy (IGRT) requires only a few seconds and yields good results in localizing a tumor under rigid motion in the head and neck and under respiratory deformation in the lung, using one treatment-time imaging 2D projection or a small set thereof.     

Optimal Structure from Motion: Local Ambiguities and Global Estimates

Structure From Motion (SFM) refers to the problem of estimating spatial properties of a three-dimensional scene from the motion of its projection onto a two-dimensional surface, such as the retina. We present an analysis of SFM which results in algorithms that are provably convergent and provably optimal with respect to a chosen norm.In particular, we cast SFM as the minimization of a high-dimensional quadratic cost function, and show how it is possible to reduce it to the minimization of a two-dimensional function whose stationary points are in one-to-one correspondence with those of the original cost function. As a consequence, we can plot the reduced cost function and characterize the configurations of structure and motion that result in local minima. As an example, we discuss two local minima that are associated with well-known visual illusions. Knowledge of the topology of the residual in the presence of such local minima allows us to formulate minimization algorithms that, in addition to provably converge to stationary points of the original cost function, can switch between different local extrema in order to converge to the global minimum, under suitable conditions. We also offer an experimental study of the distribution of the estimation error in the presence of noise in the measurements, and characterize the sensitivity of the algorithm using the structure of Fisher's Information matrix.     

The Least-Squares Error for Structure from Infinitesimal Motion

We analyze the least-squares error for structure from motion with a single infinitesimal motion (structure from optical flo). We present asymptotic approximations to the noiseless error over two, complementary regions of motion estimates: roughly forward and non-forward translations. Our approximations are powerful tools for understanding the error. Experiments show that they capture its detailed behavior over the entire range of motions. We illustrate the use of our approximations by deriving new properties of the least-squares error. We generalize the earlier results of Jepson/Heeger/Maybank on the bas-relief ambiguity and of Oliensis on the reflected minimum. We explain the error's complexity and its multiple local minima for roughly forward translation estimates (epipoles within the field of view) and identify the factors that make this complexity likely. For planar scenes, we clarify the effects of the two-fold ambiguity, show the existence of a new, double bas-relief ambiguity, and analyze the error's local minima. For nonplanar scenes, we derive simplified error approximations for reasonable assumptions on the image and scene. For example, we show that the error tends to have a simpler form when many points are tracked. We show experimentally that our analysis for zero image noise gives a good model of the error for large noise. We show theoretically and experimentally that the error for projective structure from motion is simpler but flatter than the error for calibrated images.     

Graph theoretic closure properties of the family of boundary NLC graph languages

Summary Node label controlled (NLC) grammars are graph grammars (operating on node labeled undirected graphs) which rewrite single nodes only and establish connections between the embedded graph and the neighbors of the rewritten node on the basis of the labels of the involved nodes only. They define (possibly infinite) languages of undirected node labeled graphs (or, if we just omit the labels, languages of unlabeled graphs). Boundary NLC (BNLC) grammars are NLC grammars with the property that whenever — in a graph already generated — two nodes may be rewritten, then these nodes are not adjacent. The graph languages generated by this type of grammars are called BNLC languages. The present paper continues the investigations of basic properties of BNLC grammars and languages where the central question is the following: If L is a BNLC language and P is a graph theoretic property, is the set of all graphs from L satisfying P again a BNLC language? We demonstrate that the class of BNLC languages is very stable in the sense that for almost all properties we consider the resulting languages are BNLC. In particular, the above question gets an affirmative answer, if the property P is: being k-colorable, being connected, having a subgraph homeomorphic to a given graph, and being nonplanar.This research was carried out during the second author's stay at the Rijksuniversiteit Leiden, The Netherlands     

Multigrid based total variation image registration

We consider the image registration problem to find a reasonable displacement field, such that a transformed template image becomes similar to a so-called reference image. The minimization of the similarity measure (exemplarily based on the gray-value difference) yields a nonlinear ill-posed inverse problem. The necessary regularization is done by replacing the ill-conditioned Hessian by a multidimensional total variation norm. This allows steep gradients and discontinuities in the displacement field in contrast to the common approach by elastic regularization which leads to globally smooth displacement fields. We propose and investigate a multigrid algorithm as inner iteration for registration. As we use Neumann boundary conditions which lead to singular systems, a special treatment before and during the FAS multigrid algorithm is required, e.g. the introduction and solution of an augmented system. We describe the necessary modifications for the multigrid algorithm and present convergence results as well as first registration experiments demonstrating the capabilities of the proposed approach. The work of the first author was supported by the Deutsche Forschungsgemeinschaft; grant HE 3404.     

Fast Local and Global Projection-Based Methods for Affine Motion Estimation

The demand for more effective compression, storage, and transmission of video data is ever increasing. To make the most effective use of bandwidth and memory, motion-compensated methods rely heavily on fast and accurate motion estimation from image sequences to compress not the full complement of frames, but rather a sequence of reference frames, along with differences between these frames which results from estimated frame-to-frame motion. Motivated by the need for fast and accurate motion estimation for compression, storage, and transmission of video as well as other applications of motion estimation, we present algorithms for estimating affine motion from video image sequences. Our methods utilize properties of the Radon transform to estimate image motion in a multiscale framework to achieve very accurate results. We develop statistical and computational models that motivate the use of such methods, and demonstrate that it is possible to improve the computational burden of motion estimation by more than an order of magnitude, while maintaining the degree of accuracy afforded by the more direct, and less efficient, 2-D methods.     

The dynamic retina: Contrast and motion detection for active vision

This paper presents an efficient, biologically-inspired early vision architecture, the dynamic retina, that is well-suited to highly active and responsive vision platforms. The dynamic retina exploits normally undesirable camera motion as a necessary step in detecting image contrast, by using dynamic receptive fields instead of traditional spatial-neighborhood operators. We analyze the continuous miniature noise movements made by active imaging systems, and show that they can be exploited to detect contrast. We then develop an appropriate photoreceptor response function, based on light-adaptation models for vertebrate receptors. Together, the movements and response function over time compute image contrast. The dynamic retina is also useful for motion analysis, since moving objects processed by the system leave a clear signature from which motion parameters can be extracted. Results from a number of experiments with real video sequences demonstrate the effectiveness of the system for both contrast detection and motion analysis.     

联合弯曲能量和标志点对应约束的非刚性医学图像配准

由于缺乏图像几何空间约束,基于互信息的非刚性医学图像配准常常产生不合理的形变。提出一种联合弯曲能量和标志点对应约束的非刚性医学图像配准方法,在互信息配准目标函数中添加弯曲能量惩罚和对应标志点间欧氏距离2个正则项,约束医学图像软组织不合理形变。脑部MRI、头颈部CT、胸部CBCT影像配准实验结果表明,该方法可有效提高配准质量。     

基于相位相关法的全局运动估计算法

提出了一种基于鲁捧统计和相位相关法相结合的全局运动估计算法;由于相位相关法利用图像的功率谱信息,减少了对图像内容的依赖,具有一定的抗噪能力,因此该算法将块匹配法与相位相关相结合来计算图像间的运动矢量场,不仅减少了运算量而且能得到更加准确的矢量场;为了提高模型参数估计精度和运算效率,运用多分辨率鲁棒统计的方法来计算运动估计模型参数;航拍视频图像配准与独立运动检测的仿真结果均验证了算法的有效性。     

基于混合特征的非刚性点阵配准算法

提出一种基于混合特征的非刚性点阵配准算法.该算法包含了对应关系评估与空间变换更新两个相互交替的步骤.首先定义了两个特征描述法用于描述两个点阵之间的全局和局部几何结构特征差异,随后合并这两个特征描述法建立一个基于混合特征的能量优化方程.该能量优化方程可以利用线性分配技术进行求解,同时可以灵活地选择使用最小化全局结构特征差异或最小化局部结构特征差异来评估两个点阵之间的对应关系.为了增强前述两个步骤之间的协调性,我们利用能量权重调节在整个配准过程中控制能量优化从最小化局部结构特征差异逐步转变为最小化全局结构特征差异,同时控制用于空间变换的薄板样条函数（Thin plate spline）的更新从刚性变换逐步转变为非刚性变换.我们在二维轮廓配准、三维轮廓配准、序列图像配准和图像特征点配准下对本文算法进行了各项性能测试,同时也与当前8种流行算法进行了性能比较.本文算法展现了卓越的非刚性配准性能,并在大部分实验中超越了当前的相关算法.     

A step‐by‐step modeling,analysis and annotation of locomotion

Annotating unlabeled motion captures plays an important role in Computer Animation for motion analysis and motion edition purposes. Locomotion is a difficult case study as all the limbs of the human body are involved whereas a low‐dimensional global motion is performed. The oscillatory nature of the locomotion makes difficult the distinction between straight steps and turning ones, especially for subtle orientation changes. In this paper we propose to geometrically model the center of mass trajectory during locomotion as a C‐continuous circular arcs sequence. Our model accurately analyzes the global motion into the velocity‐curvature space. An experimental study demonstrates that an invariant law links curvature and velocity during straight walk. We finally illustrate how the resulting law can be used for annotation purposes: any unlabeled motion captured walk can be transformed into an annotated sequence of straight and turning steps. Several examples demonstrate the robustness of our approach and give comparison with classical threshold‐based techniques. Copyright © 2011 John Wiley & Sons, Ltd.     

Information Theoretic Deformable Registration Using Local Image Information

We present a deformable registration algorithm for multi-modality images based on information theoretic similarity measures at the scale of individual image voxels. We derive analytical expressions for the mutual information, the joint entropy, and the sum of marginal entropies of two images over a small neighborhood in terms of image gradients. Using these expressions, we formulate image registration algorithms maximizing local similarity over the whole image domain in an energy minimization framework. This strategy produces highly elastic image alignment as the registration is driven by voxel similarities between the images, the algorithms are easily implementable using the closed-form expressions for the derivative of the optimization function with respect to the deformation, and avoid estimation of joint and marginal probability densities governing the image intensities essential to conventional information theoretic image registration methods. This work has been supported in part by NIH grants R01-NS42645 and R01-AG14971.     

Camera Motion Estimation Through Planar Deformation Determination

In this paper, we propose a global method for estimating the motion of a camera which films a static scene. Our approach is direct, fast and robust, and deals with adjacent frames of a sequence. It is based on a quadratic approximation of the deformation between two images, in the case of a scene with constant depth in the camera coordinate system. This condition is very restrictive but we show that, provided translation and depth inverse variations are small enough, the error on optical flow involved by the approximation of depths by a constant is small. In this context, we propose a new model of camera motion which allows to separate the image deformation in a similarity and a “purely” projective application, due to change of optical axis direction. This model leads to a quadratic approximation of image deformation that we estimate with an M-estimator; we can immediately deduce camera motion parameters. Electronic Supplementary Material  The online version of this article () contains supplementary material, which is available to authorized users.

Structure from Motion: Beyond the Epipolar Constraint

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images.The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the least varying estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.