A neural model for the integration of stereopsis and motion parallax in structure from motion

We introduce a model for the computation of structure from motion based on the physiology of visual cortical areas MT and MST. The model assumes that the perception of depth from motion is related to the firing of a subset of MT neurons tuned to both velocity and disparity. The model's MT neurons are connected to each other laterally to form modulatory receptive-field surrounds that are gated by feedback connections from area MST. This allows the building up of a depth map from motion in area MT, even in absence of disparity in the input. Depth maps from motion and from stereo are combined by a weighted average at a final stage. The model's predictions for the interaction between motion and stereo cues agree with previous psychophysical data, both when the cues are consistent with each other or when they are contradictory. In particular, the model shows nonlinearities as a result of early interactions between motion and stereo before their depth maps are averaged. The two cues interact in a way that represents an alternative to the “modified weak fusion” model of depth–cue combination.     

A Closed-Form Solution to Non-Rigid Shape and Motion Recovery

Recovery of three dimensional (3D) shape and motion of non-static scenes from a monocular video sequence is important for applications like robot navigation and human computer interaction. If every point in the scene randomly moves, it is impossible to recover the non-rigid shapes. In practice, many non-rigid objects, e.g. the human face under various expressions, deform with certain structures. Their shapes can be regarded as a weighted combination of certain shape bases. Shape and motion recovery under such situations has attracted much interest. Previous work on this problem (Bregler, C., Hertzmann, A., and Biermann, H. 2000. In Proc. Int. Conf. Computer Vision and Pattern Recognition; Brand, M. 2001. In Proc. Int. Conf. Computer Vision and Pattern Recognition; Torresani, L., Yang, D., Alexander, G., and Bregler, C. 2001. In Proc. Int. Conf. Computer Vision and Pattern Recognition) utilized only orthonormality constraints on the camera rotations (rotation constraints). This paper proves that using only the rotation constraints results in ambiguous and invalid solutions. The ambiguity arises from the fact that the shape bases are not unique. An arbitrary linear transformation of the bases produces another set of eligible bases. To eliminate the ambiguity, we propose a set of novel constraints, basis constraints, which uniquely determine the shape bases. We prove that, under the weak-perspective projection model, enforcing both the basis and the rotation constraints leads to a closed-form solution to the problem of non-rigid shape and motion recovery. The accuracy and robustness of our closed-form solution is evaluated quantitatively on synthetic data and qualitatively on real video sequences.     

A semi-direct approach to structure from motion

The problem of structure from motion is often decomposed into two steps: feature correspondence and three-dimensional reconstruction. This separation often causes gross errors when establishing correspondence fails. Therefore, we advocate the necessity to integrate visual information not only in time (i.e. across different views), but also in space, by matching regions – rather than points – using explicit photometric deformation models. We present an algorithm that integrates image-feature tracking and three-dimensional motion estimation into a closed loop, while detecting and rejecting outlier regions that do not fit the model. Due to occlusions and the causal nature of our algorithm, a drift in the estimates accumulates over time. We describe a method to perform global registration of local estimates of motion and structure by matching the appearance of feature regions stored over long time periods. We use image intensities to construct a score function that takes into account changes in brightness and contrast. Our algorithm is recursive and suitable for real-time implementation.     

改进的从运动中恢复目标结构的因子分解法

因子分解法是从图像序列中恢复刚体目标几何结构的重要方法。介绍了传统因子分解法的基本过程,分析了该方法存在的不足,并针对该方法容易失效的缺点,提出一种改进的因子分解法。该方法避开传统方法中求解修正矩阵的复杂过程,利用旋转矩阵的特性,直接修正由传统方法SVD分解得到的每帧图像的旋转矩阵,然后根据观测矩阵和得到旋转矩阵直接利用线性最小二乘法求解目标的结构矩阵。仿真和实测数据的实验结果表明,本文方法能够有效地从序列图像中恢复目标的几何结构,相比传统的因子分解法而言,在稳定性上有较大的提升。     

Shape from Specular Reflection and Optical Flow

Inferring scene geometry from a sequence of camera images is one of the central problems in computer vision. While the overwhelming majority of related research focuses on diffuse surface models, there are cases when this is not a viable assumption: in many industrial applications, one has to deal with metal or coated surfaces exhibiting a strong specular behavior. We propose a novel and generalized constrained gradient descent method to determine the shape of a purely specular object from the reflection of a calibrated scene and additional data required to find a unique solution. This data is exemplarily provided by optical flow measurements obtained by small scale motion of the specular object, with camera and scene remaining stationary. We present a non-approximative general forward model to predict the optical flow of specular surfaces, covering rigid body motion as well as elastic deformation, and allowing for a characterization of problematic points. We demonstrate the applicability of our method by numerical experiments on synthetic and real data.     

Egomotion analysis based on the Frenet-Serret motion model

In this paper we propose a new model,Frenet-Serret motion, for the motion of an observer in a stationary environment. This model relates the motion parameters of the observer to the curvature and torsion of the path along which the observer moves. Screw-motion equations for Frenet-Serret motion are derived and employed for geometrical analysis of the motion. Normal flow is used to derive constraints on the rotational and translational velocity of the observer and to compute egomotion by intersecting these constraints in the manner proposed in (Duri and Aloimonos 1991) The accuracy of egomotion estimation is analyzed for different combinations of observer motion and feature distance. We explain the advantages of controlling feature distance to analyze egomotion and derive the constraints on depth which make either rotation or translation dominant in the perceived normal flow field. The results of experiments on real image sequences are presented.The support of the Air Force Office of Scientific Research under Grant F49620-93-1-0039 is gratefully acknowledged.     

Preemptive RANSAC for live structure and motion estimation

A system capable of performing robust live ego-motion estimation for perspective cameras is presented. The system is powered by random sample consensus with preemptive scoring of the motion hypotheses. A general statement of the problem of efficient preemptive scoring is given. Then a theoretical investigation of preemptive scoring under a simple inlier–outlier model is performed. A practical preemption scheme is proposed and it is shown that the preemption is powerful enough to enable robust live structure and motion estimation.Prepared through collaborative participation in the Robotics Consortium sponsored by the U.S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD19-01-2-0012. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. David Nistér received PhD degree in computer vision, numerical analysis and computing science from the Royal Institute of Technology (KTH), Stockholm, Sweden, with the thesis ‘Automatic Dense Reconstruction from Uncalibrated Video Sequences’. He is currently an assistant professor at the Computer Science Department and the Center for Visualization and Virtual Environments, University of Kentucky, Lexington. Before joining UK, he was a researcher in the Vision Technologies Laboratory, Sarnoff Corporation, Princeton, and Visual Technology, Ericsson Research, Stockholm, Sweden. His research interests include computer vision, computer graphics, structure from motion, multiple view geometry, Bayesian formulations, tracking, recognition, image and video compression. He is a member of the IEEE and American Mensa.     

Determining shape and motion from non-overlapping multi-camera rig: A direct approach using normal flows

In this paper, we explore how a wide field-of-view imaging system that consists of a number of cameras in a network arranged to approximate a spherical eye can reduce the complexity of estimating camera motion. Depth map of the imaged scene can be reconstructed once the camera motion is there. We present a direct method to recover camera motion from video data, which neither requires establishment of feature correspondences nor recovery of optical flow, but from normal flow which is directly observable. With a wide visual field, the inherent ambiguities between translation and rotation disappear. Several subsets of normal flow pairs and triplets can be utilized to constraint the directions of translation and rotation separately. The intersection of solution spaces arising from normal flow pairs or triplets yields the estimate on the direction of motion. In addition, the larger number of normal flow measurements so resulted can be used to combat the local flow extraction error. Rotational magnitude is recovered in a subsequent stage. This article details how motion recovery can be improved with the use of such an approximate spherical imaging system. Experimental results on synthetic and real image data are provided. The results show that the accuracy of motion estimation is comparable to those of the state-of-the-art methods that require to use explicit feature correspondences or full optical flows, and our method has a much faster computational speed.     

Discontinuity pattern detection and orientation measurement for tunnel faces by using structure from motion photogrammetry

Crack is a common condition that affects water-rich loess tunnels, and it negatively influences the reliability and safety of the tunnel. Jointed rock masses are commonly found during underground excavation. Many underground openings have failed during excavation and operation. The evaluation of tunnel faces by using a new measurement system based on structure from motion (SfM) photogrammetry is proposed in this study. The main objective is to determine the discontinuity pattern and orientation of tunnel faces as an input for tunnel face stability evaluation. A set of overlapping images obtained from a tunnel face replica using 3D polystyrene and the SfM photogrammetry approach is utilized to generate a 3D point cloud model. The discontinuity pattern and orientation are determined via facet extraction of the K_D-tree plugin in CloudCompare. The same set of overlapping images is analyzed in four different quality settings (low, medium, high, and ultrahigh) with different sets of point cloud numbers that control the accuracy of the measurement of discontinuity. Results show that the high-quality setting presents a consistent measurement of the discontinuity pattern and orientation in contrast to the real 3D polystyrene tunnel face model. 2D pattern results from CloudCompare are validated using the fractal analysis method in an image analysis software, where the image is converted into a binary-one. Two sets of discontinuity are derived. The means of dip and dip direction from the cluster analysis are 82°/164° and 84°/307°, respectively. The orientations are verified through manual compass measurement. The orientations for manual compass measurement for Sets 1 and 2 are 86°/151° and 80°/303°, respectively. The discontinuities obtained from the high-quality setting of the point cloud and manual measurement of the orientation by using a geological compass highlight the similarity of the discontinuity plane in both discontinuity sets for the tunnel face replica. The proposed method for tunnel face evaluation has good judgment for tunnel support and the prevention of biased analyses by professionals.     

Spherical approximation for multiple cameras in motion estimation: Its applicability and advantages

Estimating motions of a multi-camera system which may not have overlapping fields of view is generally complex and computationally expensive because of the non-zero offset between each camera’s center. It is conceivable that if we can assume that multiple cameras share a single optical center, and thus can be modeled as a spherical imaging system, motion estimation and calibration of this system would become simpler and more efficient.     

Absolute motion and structure from stereo image sequences without stereo correspondence and analysis of degenerate cases

This paper deals with the estimation of motion and structure with an absolute scale factor from stereo image sequences without stereo correspondence. We show that the absolute motion and structure can be determined using only motion correspondences. This property is very useful in two aspects: first, motion correspondence is easier to solve than stereo correspondence because sequences of images can be taken at short time intervals; second, it is not necessary that the rigid scene be included in the intersection of the field of view of the two cameras. It is also shown that the degenerate cases reported in this paper constitute all of the degenerate cases for the scheme and can be easily avoided.     

Combining 3D flow fields with silhouette-based human motion capture for immersive video

In recent years, the convergence of computer vision and computer graphics has put forth a new field of research that focuses on the reconstruction of real-world scenes from video streams. To make immersive 3D video reality, the whole pipeline spanning from scene acquisition over 3D video reconstruction to real-time rendering needs to be researched. In this paper, we describe latest advancements of our system to record, reconstruct and render free-viewpoint videos of human actors. We apply a silhouette-based non-intrusive motion capture algorithm making use of a 3D human body model to estimate the actor’s parameters of motion from multi-view video streams. A renderer plays back the acquired motion sequence in real-time from any arbitrary perspective. Photo-realistic physical appearance of the moving actor is obtained by generating time-varying multi-view textures from video. This work shows how the motion capture sub-system can be enhanced by incorporating texture information from the input video streams into the tracking process. 3D motion fields are reconstructed from optical flow that are used in combination with silhouette matching to estimate pose parameters. We demonstrate that a high visual quality can be achieved with the proposed approach and validate the enhancements caused by the the motion field step.     

Recursive non-rigid structure from motion with online learned shape prior

Most existing approaches in structure from motion for deformable objects focus on non-incremental solutions utilizing batch type algorithms. All data is collected before shape and motion reconstruction take place. This methodology is inherently unsuitable for applications that require real-time learning. Ideally the online system is capable of incrementally learning and building accurate shapes using current measurement data and past reconstructed shapes. Estimation of 3D structure and camera position is done online. To rely only on the measurements up until that moment is still a challenging problem.     

Robust structure from motion with affine camera via low-rank matrix recovery

We present a novel approach to structure from motion that can deal with missing data and outliers with an affine camera. We model the corruptions as sparse error. Therefore the structure from motion problem is reduced to the problem of recovering a low-rank matrix from corrupted observations. We first decompose the matrix of trajectories of features into low-rank and sparse components by nuclear-norm and l1-norm minimization, and then obtain the motion and structure from the low-rank components by the classical factorization method. Unlike pervious methods, which have some drawbacks such as depending on the initial value selection and being sensitive to the large magnitude errors, our method uses a convex optimization technique that is guaranteed to recover the low-rank matrix from highly corrupted and incomplete observations. Experimental results demonstrate that the proposed approach is more efficient and robust to large-scale outliers.     

Structure from Motion Using Bio-Inspired Intelligence Algorithm and Conformal Geometric Algebra

Structure from Motion algorithms offer good advantages, such as extract 3D information in monocular systems and structures estimation as shown in Hartley & Zisserman for numerous applications, for instance; augmented reality, autonomous navigation, motion capture, remote sensing and object recognition among others. Nevertheless, this algorithm suffers some weaknesses in precision. In the present work, we extent the proposal in Arana-Daniel, Villaseñor, López-Franco, & Alanís that presents a new strategy using bio-inspired intelligence algorithm and Conformal Geometric Algebra, based in the object mapping paradigm, to overcome the accuracy problem in two-view Structure form motion algorithms. For this instance, we include two new experiments and the inclusion of the circle entity; the circle carries stronger information about its motion than other geometric entities, as we will show.     

Recursive estimation of motion and a scene model with a two-camera system of divergent view

This paper deals with recursive reconstruction of a scene model from unknown motion of a two-camera system capturing the images of the scene. Single camera systems with a relatively small field of view have limited accuracy because of the inherent confusion between translation and rotation. Estimation results from the stereo camera systems are also compromised due to this confusion if the systems require the fields of view to intersect for stereo correspondence. The cameras constituting the two-camera system considered in this paper are arranged so that there is a small intersection of the fields of view. This configuration of divergent view improves the accuracy of the structure and motion estimation because the ambiguity mentioned above decreases due to a large field of view. In this paper, a recursive algorithm is proposed for fast scene model reconstruction using a two-camera system of divergent view. Using inversely inferred stereo correspondences in the intersection of the fields of view is also proposed to remove degeneracy of scale factor determination and to acquire more accurate results from the information redundancy. The results of the experiments with long term real image sequences are presented to demonstrate the feasibility of the proposed system.     

采用碰撞测试和回归机制的非完整约束机器人快速扩展随机树运动规划

本文提出一种改进的快速扩展随机树(rapidly-exploring random trees,RRT)运动规划方法,用于非完整微分约束下的机器人运动规划.针对类似目标偏好与双向RRT(bi-directional RRT,bi-RRT)等目标区域导向的RRT运动规划所存在的局部极小问题,结合回归检测与碰撞检测机制,设计了一种碰撞检测与回归机制(collision-test and regression mechanism,CR)机制.该方法使得机器人在规划过程中能获取到全局障碍物信息,从而避免对已扩展节点的重复搜索,以及重复对边缘节点的回归测试和避障检测.该机制使得机器人可加快跳出局部极小区域,提高运动规划实的时性.将改进的RRT运动算法在容易产生局部极小值的环境中仿真测试,结果表明该算法在不显著影响其他性能的前提下,可以明显提高规划的实时性.     

RGBD融合明暗恢复形状的全视角三维重建技术研究

为了高效、高精度、低成本地实现对物体的全视角三维重建, 提出一种使用深度相机融合光照约束实现全视角三维重建的方法。该重建方法中,在进行单帧重建时采用RGBD深度图像融合明暗恢复形状(Shape from shading,SFS)的重建方法, 即在原有的深度数据上加上额外的光照约束来优化深度值; 在相邻两帧配准时, 采用快速点特征直方图（Fast point feature histograms, FPFH）特征进行匹配并通过随机采样一致性（Random sample consensus, RANSAC）滤除错误的匹配点对求解粗配准矩阵, 然后通过迭代最近点（Iterative closest point, ICP）算法进行精配准得出两帧间的配准矩阵; 在进行全视角的三维重建时, 采用光束平差法优化相机位姿, 从而消除累积误差使首尾帧完全重合, 最后融合生成一个完整的模型。该方法融入了物体表面的光照信息,因此生成的三维模型更为光顺,也包含了更多物体表面的细节信息,提高了重建精度;同时该方法仅通过单张照片就能在自然光环境下完成对多反射率三维物体的重建,适用范围更广。本文方法的整个实验过程通过手持深度相机就能完成,不需要借助转台,操作更加方便。     

Efficient and robust model fitting with unknown noise scale

This paper addresses the general problem of robust parametric model estimation from data that has both an unknown (and possibly majority) fraction of outliers as well as an unknown scale of measurement noise. We focus on computer vision applications from image correspondences, such as camera resectioning, estimation of the fundamental matrix or relative pose for 3D reconstruction, and estimation of 2D homographies for image registration and motion segmentation, although there are many other applications. In practice, these methods typically rely on a predefined inlier thresholds because automatic scale detection is usually too unreliable or too slow. We propose a new method for robust estimation with automatic scale detection that is faster, more precise and more robust than previous alternatives, and show that it can be practically applied to these problems.     

Measuring and modelling sewer pipes from video

This article presents a system for the automatic measurement and modelling of sewer pipes. The system recovers the interior shape of a sewer pipe from a video sequence which is acquired by a fish-eye lens camera moving inside the pipe. The approach is based on tracking interest points across successive video frames and posing the general structure-from-motion problem. It is shown that the tracked points can be reliably reconstructed despite the forward motion of the camera. This is achieved by utilizing a fish-eye lens with a wide field of view. The standard techniques for robust estimation of the two- and three-view geometry are modified so that they can be used for calibrated fish-eye lens cameras with a field of view less than 180°. The tubular arrangement of the reconstructed points allows pipe shape estimation by surface fitting. Hence, a method for modelling such surfaces with a locally cylindrical model is proposed. The system is demonstrated with a real sewer video and an error analysis for the recovered structure is presented.