Structure from Motion: Beyond the Epipolar Constraint

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images.The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the least varying estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.     

Design and Use of Linear Models for Image Motion Analysis

Linear parameterized models of optical flow, particularly affine models, have become widespread in image motion analysis. The linear model coefficients are straightforward to estimate, and they provide reliable estimates of the optical flow of smooth surfaces. Here we explore the use of parameterized motion models that represent much more varied and complex motions. Our goals are threefold: to construct linear bases for complex motion phenomena; to estimate the coefficients of these linear models; and to recognize or classify image motions from the estimated coefficients. We consider two broad classes of motions: i) generic motion features such as motion discontinuities and moving bars; and ii) non-rigid, object-specific, motions such as the motion of human mouths. For motion features we construct a basis of steerable flow fields that approximate the motion features. For object-specific motions we construct basis flow fields from example motions using principal component analysis. In both cases, the model coefficients can be estimated directly from spatiotemporal image derivatives with a robust, multi-resolution scheme. Finally, we show how these model coefficients can be use to detect and recognize specific motions such as occlusion boundaries and facial expressions.     

Optical Snow

Classical methods for measuring image motion by computer have concentrated on the cases of optical flow in which the motion field is continuous, or layered motion in which the motion field is piecewise continuous. Here we introduce a third natural category which we call optical snow. Optical snow arises in many natural situations such as camera motion in a highly cluttered 3-D scene, or a passive observer watching a snowfall. Optical snow yields dense motion parallax with depth discontinuities occurring near all image points. As such, constraints on smoothness or even smoothness in layers do not apply. In the Fourier domain, optical snow yields a one-parameter family of planes which we call a bowtie. We present a method for measuring the parameters of the direction and range of speeds of the motion for the special case of parallel optical snow. We demonstrate the effectiveness of the method for both synthetic and real image sequences.Supplementary material to this paper is available in electronic form at http://dx.doi.org/10.1023/A:1024440524579     

Accurate optical flow computation under non-uniform brightness variations

In this paper, we present a very accurate algorithm for computing optical flow with non-uniform brightness variations. The proposed algorithm is based on a generalized dynamic image model (GDIM) in conjunction with a regularization framework to cope with the problem of non-uniform brightness variations. To alleviate flow constraint errors due to image aliasing and noise, we employ a reweighted least-squares method to suppress unreliable flow constraints, thus leading to robust estimation of optical flow. In addition, a dynamic smoothness adjustment scheme is proposed to efficiently suppress the smoothness constraint in the vicinity of the motion and brightness variation discontinuities, thereby preserving motion boundaries. We also employ a constraint refinement scheme, which aims at reducing the approximation errors in the first-order differential flow equation, to refine the optical flow estimation especially for large image motions. To efficiently minimize the resulting energy function for optical flow computation, we utilize an incomplete Cholesky preconditioned conjugate gradient algorithm to solve the large linear system. Experimental results on some synthetic and real image sequences show that the proposed algorithm compares favorably to most existing techniques reported in literature in terms of accuracy in optical flow computation with 100% density.     

Exploiting Discontinuities in Optical Flow

Most optical flow estimation techniques have substantial difficulties dealing with flow discontinuities. Methods which simultaneously detect flow boundaries and use the detected boundaries to aid in flow estimation can produce significantly improved results. Current approaches to implementing these methods still have important limitations, however. We demonstrate three such problems: errors due to the mixture of image properties across boundaries, an intrinsic ambiguity in boundary location when only short sequences are considered, and difficulties insuring that the motion of a boundary aids in flow estimation for the surface to which it is attached without corrupting the flow estimates for the occluded surface on the other side. The first problem can be fixed by basing flow estimation only on image changes at edges. The second requires an analysis of longer time intervals. The third can be aided by using a boundary detection mechanism which classifies the sides of boundaries as occluding and occluded at the same time as the boundaries are detected.     

基于直线光流场的三维运动和结构重建

利用直线间运动对应关系，将像素点光流的概念和定义方法应用于直线，提出了直线光流的概念，建立了求解空间物体运动参数的线性方程组，利用三幅图像21条直线的光流场，可以求得物体运动的12个参数以及空间直线坐标．但是在实际应用当中，要找出这21条直线的光流场是很困难的，因此该文提出了运用解非线性方程组的方法，只需要6条直线的光流．就可以分步求出物体的12个运动参数，并根据求得的12个运动参数和一致的图像坐标系中的直线坐标，求得空间直线的坐标，从而实现了三维场景的重建．     

Image flow segmentation and estimation by constraint lineclustering

Image flow is the velocity field in the image plane caused by the motion of the observer, objects in the scene, or apparent motion, and can contain discontinuities due to object occlusion in the scene. An algorithm that can estimate the image flow velocity field when there are discontinuities due to occlusions is described. The constraint line clustering algorithm uses a statistical test to estimate the image flow velocity field in the presence of step discontinuities in the image irradiance or velocity field. Particular emphasis is placed on motion estimation and segmentation in situations such as random dot patterns where motion is the only cue to segmentation. Experimental results on a demanding synthetic test case and a real image are presented. A smoothing algorithm for improving the velocity field estimate is also described. The smoothing algorithm constructs a smooth estimate of the velocity field by approximating a surface between step discontinuities. It is noted that the velocity field estimate can be improved using surface reconstruction between velocity field boundaries     

面向运动分割优化的变分光流计算方法研究

面向运动分割的需求,围绕变分光流计算中的运动边缘保留问题,对反应-扩散式光流计算模型中的扩散张量设计与改进方法进行系统深入研究。 在分析已有设计方法的基础上,提出了一种融合图像和流场信息驱动的扩散张量设计方法,并利用该扩散张量建立光流计算模型,然后在多尺度计算框架下给出了相应的偏微分方程数值计算方法。 理论分析与对比实验结果表明,这种设计方法能有效克服现有方法提取运动边缘不精确的缺点,更准确地刻画出运动边缘,提高光流的计算精度。     

Motion estimation and segmentation

In the general structure-from-motion (SFM) problem involving several moving objects in a scene, the essential first step is to segment moving objects independently. We attempt to deal with the problem of optical flow estimation and motion segmentation over a pair of images. We apply a mean field technique to determine optical flow and motion boundaries and present a deterministic algorithm. Since motion discontinuities represented by line process are embedded in the estimation of the optical flow, our algorithm provides accurate estimates of optical flow especially along motion boundaries and handles occlusion and multiple motions. We show that the proposed algorithm outperforms other well-known algorithms in terms of estimation accuracy and timing.     

Non Uniform Multiresolution Method for Optical Flow and Phase Portrait Models: Environmental Applications

In this paper we define a complete framework for processing large image sequences for a global monitoring of short range oceanographic and atmospheric processes. This framework is based on the use of a non quadratic regularization technique for optical flow computation that preserves flow discontinuities. We also show that using an appropriate tessellation of the image according to an estimate of the motion field can improve optical flow accuracy and yields more reliable flows. This method defines a non uniform multiresolution approach for coarse to fine grid generation. It allows to locally increase the resolution of the grid according to the studied problem. Each added node refines the grid in a region of interest and increases the numerical accuracy of the solution in this region. We make use of such a method for solving the optical flow equation with a non quadratic regularization scheme allowing the computation of optical flow field while preserving its discontinuities. The second part of the paper deals with the interpretation of the obtained displacement field. For this purpose a phase portrait model used along with a new formulation of the approximation of an oriented flow field allowing to consider arbitrary polynomial phase portrait models for characterizing salient flow features. This new framework is used for processing oceanographic and atmospheric image sequences and presents an alternative to complex physical modeling techniques.