Vanishing points and three-dimensional lines from omni-directional video

This paper describes a system for structure from motion using vanishing points and three-dimensional lines extracted from omni-directional video sequences. To track lines, we use a novel dynamic programming approach to improve ambiguity resolution, and we use delayed states to aid in the initialization of landmarks. By reobserving vanishing points we get direct measurements of the robots three-dimensional attitude that are independent of its position. Using vanishing points simplifies the representation since parallel lines share the same direction states. We show the performance of the system in various indoor and outdoor environments and include comparisons with independent two-dimensional reference maps for each experiment .     

Omnidirectional video

Omnidirectional video enables direct surround immersive viewing of a scene by warping the original image into the correct perspective given a viewing direction. However, novel views from viewpoints off the camera path can only be obtained if we solve the three-dimensional motion and calibration problem. In this paper we address the case of a parabolic catadioptric camera – a paraboloidal mirror in front of an orthographic lens – and we introduce a new representation, called the circle space, for points and lines in such images. In this circle space, we formulate an epipolar constraint involving a 4×4 fundamental matrix. We prove that the intrinsic parameters can be inferred in closed form from the two-dimensional subspace of the new fundamental matrix from two views if they are constant or from three views if they vary. Three-dimensional motion and structure can then be estimated from the decomposition of the fundamental matrix.     

A Green's function approach to shape from shading

We present a new algorithm for shape from shading, inspired on the recently introduced disparity-based approach to photometric stereo (DBPS). Assuming that the single input image will be matched to a second image through a uniform disparity field, we construct an irradiance conservation equation and solve it for the matching image, via Green's function. When a linear expansion of the reflectance map is considered, this leads to a closed-form approximate expression for the surface function, whose parameters can be estimated via a structure-from-motion approach already used for DBPS.     

A low-cost calibration method for automated optical mensuration using a video camera

Automated optical mensuration gauges the acquired image of the inspected unit while assessing its actual size and shape. The mensuration requires the following preparations: (1) alignment of the video camera perpendicularly to the inspection table, and (2) calibration of the scale ratios of image acquisition, notably, the stretching ratio caused by signal conversion and the magnification ratio of optical coupling. This paper presents the unique two-stage calibration method. The first stage applies the parallelogram conservation property, a property very sensitive to misorientation, to test against the potential misalignment. Once detected, we adjust the misalignment towards orthogonal alignment using image patterns of the calibration template. Then, the second stage determines the scale ratios. The proposed calibration method is suitable for on-site applications, and its implementation cost is low. Sensitivity analysis and experimental results are reported.     

Plenoptic video geometry

More and more processing of visual information is nowadays done by computers, but the images captured by conventional cameras are still based on the pinhole principle inspired by our own eyes. This principle though is not necessarily the optimal image-formation principle for automated processing of visual information. Each camera samples the space of light rays according to some pattern. If we understand the structure of the space formed by the light rays passing through a volume of space, we can determine the camera, or in other words the sampling pattern of light rays, that is optimal with regard to a given task. In this work we analyze the differential structure of the space of time-varying light rays described by the plenoptic function and use this analysis to relate the rigid motion of an imaging device to the derivatives of the plenoptic function. The results can be used to define a hierarchy of camera models with respect to the structure from motion problem and formulate a linear, scene-independent estimation problem for the rigid motion of the sensor purely in terms of the captured images.     

Keyframe-based recognition and localization during video-rate parallel tracking and mapping

Generating situational awareness by augmenting live imagery with collocated scene information has applications from game-playing to military command and control. We propose a method of object recognition, reconstruction, and localization using triangulation of SIFT features from keyframe camera poses in a 3D map. The map and keyframe poses themselves are recovered at video-rate by bundle adjustment of FAST image features in the parallel tracking and mapping algorithm. Detected objects are automatically labeled on the user's display using predefined annotations. Experimental results are given for laboratory scenes, and in more realistic applications.     

A neural model for the integration of stereopsis and motion parallax in structure from motion

We introduce a model for the computation of structure from motion based on the physiology of visual cortical areas MT and MST. The model assumes that the perception of depth from motion is related to the firing of a subset of MT neurons tuned to both velocity and disparity. The model's MT neurons are connected to each other laterally to form modulatory receptive-field surrounds that are gated by feedback connections from area MST. This allows the building up of a depth map from motion in area MT, even in absence of disparity in the input. Depth maps from motion and from stereo are combined by a weighted average at a final stage. The model's predictions for the interaction between motion and stereo cues agree with previous psychophysical data, both when the cues are consistent with each other or when they are contradictory. In particular, the model shows nonlinearities as a result of early interactions between motion and stereo before their depth maps are averaged. The two cues interact in a way that represents an alternative to the “modified weak fusion” model of depth–cue combination.     

Computation of ego motion using the vertical cue

This paper describes the development and implementation of some layers of a line-segment-based module to recover ego motion while building a 3D map of the environment in which the absolute vertical is taken into account. We use a monocular sequence of images and 2D-line segments in this sequence. The proposed method reduces the disparity between two frames in such a way that 3D vision is simplified. In particular, the correspondence problem is simplified. Moreover, a estimation of the 3D rotation is provided.Using the vertical as a basic cue for 3D-orientation tremendously simplifies and improves the structure from motion paradigm, but the usual equations have to be worked out in a different way.An approach which combines ecological hypotheses and general rigid motion equations is presented, and the equations are derived and discussed in the case of small rigid motions. Algorithms, based on the minimization of theMahalanobis distance between two estimates, are given and their implementations discussed.Based on Computation of ego motion and structure from visual and inertial sensors using the vertical cue by T. Viéville and P.E.D.S. Facao and E. Clergue, which appeared in the Fourth International Conference on Computer Vision, Berlin, 1993 May 11–14, pages 591–598, ©1993 IEEE     

Reconstruction of non-rigid 3D shapes from stereo-motion

Several non-rigid structure from motion methods have been proposed so far in order to recover both the motion and the non-rigid structure of an object. However, these monocular algorithms fail to give reliable 3D shape estimates when the overall rigid motion of the sequence is small. Aiming to overcome this limitation, in this paper we propose a novel approach for the 3D Euclidean reconstruction of deformable objects observed by an uncalibrated stereo rig. Using a stereo setup drastically improves the 3D model estimation when the observed 3D shape is mostly deforming without undergoing strong rigid motion. Our approach is based on the following steps. Firstly, the stereo system is automatically calibrated and used to compute metric rigid structures from pairs of views. Afterwards, these 3D shapes are aligned to a reference view using a RANSAC method in order to compute the mean shape of the object and to select the subset of points which have remained rigid throughout the sequence. The selected rigid points are then used to compute frame-wise shape registration and to robustly extract the motion parameters from frame to frame. Finally, all this information is used as initial estimates of a non-linear optimization which allows us to refine the initial solution and also to recover the non-rigid 3D model. Exhaustive results on synthetic and real data prove the performance of our proposal estimating motion, non-rigid models and stereo camera parameters even when there is no rigid motion in the original sequence.     

基于直升机激光点云的分裂导线重建研究

电力线三维模型是输电线路安全和增容分析的基础,机载/直升机激光雷达技术已经成为电力三维模型重建的重要技术手段,但少有研究涉及分裂导线的高精度建模。通过分析分裂导线点云数据的特点,提出一种基于点云分段、聚类分析和曲线拟合的分裂导线精细三维重建方法。最后利用获取的四分裂导线点云数据,考虑实际作业中可能遇到的点云缺失、密度偏低等情况,对算法进行了测试和分析。结果表明:该算法能够自适应地识别点云中分裂导线的数目,分离不同分裂导线的点云,建模结果可以满足输电线路安全分析的要求。     

Detailed Real-Time Urban 3D Reconstruction from Video

The paper presents a system for automatic, geo-registered, real-time 3D reconstruction from video of urban scenes. The system collects video streams, as well as GPS and inertia measurements in order to place the reconstructed models in geo-registered coordinates. It is designed using current state of the art real-time modules for all processing steps. It employs commodity graphics hardware and standard CPU’s to achieve real-time performance. We present the main considerations in designing the system and the steps of the processing pipeline. Our system extends existing algorithms to meet the robustness and variability necessary to operate out of the lab. To account for the large dynamic range of outdoor videos the processing pipeline estimates global camera gain changes in the feature tracking stage and efficiently compensates for these in stereo estimation without impacting the real-time performance. The required accuracy for many applications is achieved with a two-step stereo reconstruction process exploiting the redundancy across frames. We show results on real video sequences comprising hundreds of thousands of frames.     

Structure from Motion Using Bio-Inspired Intelligence Algorithm and Conformal Geometric Algebra

Structure from Motion algorithms offer good advantages, such as extract 3D information in monocular systems and structures estimation as shown in Hartley & Zisserman for numerous applications, for instance; augmented reality, autonomous navigation, motion capture, remote sensing and object recognition among others. Nevertheless, this algorithm suffers some weaknesses in precision. In the present work, we extent the proposal in Arana-Daniel, Villaseñor, López-Franco, & Alanís that presents a new strategy using bio-inspired intelligence algorithm and Conformal Geometric Algebra, based in the object mapping paradigm, to overcome the accuracy problem in two-view Structure form motion algorithms. For this instance, we include two new experiments and the inclusion of the circle entity; the circle carries stronger information about its motion than other geometric entities, as we will show.     

Absolute motion and structure from stereo image sequences without stereo correspondence and analysis of degenerate cases

This paper deals with the estimation of motion and structure with an absolute scale factor from stereo image sequences without stereo correspondence. We show that the absolute motion and structure can be determined using only motion correspondences. This property is very useful in two aspects: first, motion correspondence is easier to solve than stereo correspondence because sequences of images can be taken at short time intervals; second, it is not necessary that the rigid scene be included in the intersection of the field of view of the two cameras. It is also shown that the degenerate cases reported in this paper constitute all of the degenerate cases for the scheme and can be easily avoided.     

Modeling the World from Internet Photo Collections

There are billions of photographs on the Internet, comprising the largest and most diverse photo collection ever assembled. How can computer vision researchers exploit this imagery? This paper explores this question from the standpoint of 3D scene modeling and visualization. We present structure-from-motion and image-based rendering algorithms that operate on hundreds of images downloaded as a result of keyword-based image search queries like “Notre Dame” or “Trevi Fountain.” This approach, which we call Photo Tourism, has enabled reconstructions of numerous well-known world sites. This paper presents these algorithms and results as a first step towards 3D modeling of the world’s well-photographed sites, cities, and landscapes from Internet imagery, and discusses key open problems and challenges for the research community.     

Measuring elemental time and duty cycle using automated video processing

A marker-less 2D video algorithm measured hand kinematics (location, velocity and acceleration) in a paced repetitive laboratory task for varying hand activity levels (HAL). The decision tree (DT) algorithm identified the trajectory of the hand using spatiotemporal relationships during the exertion and rest states. The feature vector training (FVT) method utilised the k-nearest neighbourhood classifier, trained using a set of samples or the first cycle. The average duty cycle (DC) error using the DT algorithm was 2.7%. The FVT algorithm had an average 3.3% error when trained using the first cycle sample of each repetitive task, and had a 2.8% average error when trained using several representative repetitive cycles. Error for HAL was 0.1 for both algorithms, which was considered negligible. Elemental time, stratified by task and subject, were not statistically different from ground truth (p < 0.05). Both algorithms performed well for automatically measuring elapsed time, DC and HAL.

Practitioner Summary: A completely automated approach for measuring elapsed time and DC was developed using marker-less video tracking and the tracked kinematic record. Such an approach is automatic, repeatable, objective and unobtrusive, and is suitable for evaluating repetitive exertions, muscle fatigue and manual tasks.     

从双视图复原三维形状的新算法

为了能鲁棒精确地从目标的双视图复原其三维视觉信息，基于随机采样最小冗余子集新概念，并利用数据正则化技术，开发了一个根据目标的双视图特征点对集合，能鲁棒精确地复原其三维视觉信息的新算法，由于该算法有如下优点：（1）随机采样大幅度减少了子集的采样次数，并能确保好子集被采样到；（2）被采样到的最小冗余子集中的冗余信息能有效地用于检验该子集的正当性和优劣程度；（3）数据正则化技术又可有效地克服由数据病态带来的计算不稳定性。因此，在强噪声、高出格点率的恶劣条件下，该算法仍能高精度地复原目标的三维视觉信息，实验结果例证了此结论。     

Visual SLAM: Why filter?

While the most accurate solution to off-line structure from motion (SFM) problems is undoubtedly to extract as much correspondence information as possible and perform batch optimisation, sequential methods suitable for live video streams must approximate this to fit within fixed computational bounds. Two quite different approaches to real-time SFM – also called visual SLAM (simultaneous localisation and mapping) – have proven successful, but they sparsify the problem in different ways. Filtering methods marginalise out past poses and summarise the information gained over time with a probability distribution. Keyframe methods retain the optimisation approach of global bundle adjustment, but computationally must select only a small number of past frames to process.     

Preemptive RANSAC for live structure and motion estimation

A system capable of performing robust live ego-motion estimation for perspective cameras is presented. The system is powered by random sample consensus with preemptive scoring of the motion hypotheses. A general statement of the problem of efficient preemptive scoring is given. Then a theoretical investigation of preemptive scoring under a simple inlier–outlier model is performed. A practical preemption scheme is proposed and it is shown that the preemption is powerful enough to enable robust live structure and motion estimation.Prepared through collaborative participation in the Robotics Consortium sponsored by the U.S. Army Research Laboratory under the Collaborative Technology Alliance Program, Cooperative Agreement DAAD19-01-2-0012. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation thereon. David Nistér received PhD degree in computer vision, numerical analysis and computing science from the Royal Institute of Technology (KTH), Stockholm, Sweden, with the thesis ‘Automatic Dense Reconstruction from Uncalibrated Video Sequences’. He is currently an assistant professor at the Computer Science Department and the Center for Visualization and Virtual Environments, University of Kentucky, Lexington. Before joining UK, he was a researcher in the Vision Technologies Laboratory, Sarnoff Corporation, Princeton, and Visual Technology, Ericsson Research, Stockholm, Sweden. His research interests include computer vision, computer graphics, structure from motion, multiple view geometry, Bayesian formulations, tracking, recognition, image and video compression. He is a member of the IEEE and American Mensa.     

从视频中恢复三维人脸的实时方法

三维人脸恢复是视觉交互的一个难点问题,提出了一种从视频中实时恢复三维人脸的新方法.该方法利用主动形状模型进行人脸特征点提取和跟踪,确保了三维形状恢复和特征跟踪的有效性和一致性;采用非刚体形状和运动估计方法构建三维形变基,有效地适应人脸形状变化的多样性;采用非线性优化算法估算人脸姿态和三维形变基参数,实现了三维人脸形状和姿态的实时恢复.实验结果表明,该方法不仅能从视频中实时恢复三维人脸模型,而且可有效跟踪人脸各种姿态的变化.     

Practical Structure and Motion from Stereo When Motion is Unconstrained

This paper describes a system which robustly estimates motion, and the 3D structure of a rigid environment, as a stereo vision platform moves through it. The system can cope with any camera motion, and any scene structure and is successful even in the presence of large jumps in camera position between the capture of successive image pairs, and when point matching is ambiguous. The system was developed to provide robust obstacle avoidance for a partially sighted person.The process described attempts to maximise use of the abundant information present in a stereo sequence. Key features include the use of multiple stereo match hypotheses, efficient motion computation from three images, and the use of this motion to ensure reliable matching, and to eliminate multiple stereo matches. Points are reconstructed in 3D space and tracked in a static coordinate frame with a Kalman Filter.This results in good 3D scene reconstructions. Structure which is impossible to match with certainty is absent, rather than being incorrectly reconstructed. As a result, the system is appropriate for obstacle detection. The results of processing some indoor and outdoor scenes, are given in the paper, and practical issues are highlighted throughout.