A hierarchy of cameras for 3D photography

The view-independent visualization of 3D scenes is most often based on rendering accurate 3D models or utilizes image-based rendering techniques. To compute the 3D structure of a scene from a moving vision sensor or to use image-based rendering approaches, we need to be able to estimate the motion of the sensor from the recorded image information with high accuracy, a problem that has been well-studied. In this work, we investigate the relationship between camera design and our ability to perform accurate 3D photography, by examining the influence of camera design on the estimation of the motion and structure of a scene from video data. By relating the differential structure of the time varying plenoptic function to different known and new camera designs, we can establish a hierarchy of cameras based upon the stability and complexity of the computations necessary to estimate structure and motion. At the low end of this hierarchy is the standard planar pinhole camera for which the structure from motion problem is non-linear and ill-posed. At the high end is a camera, which we call the full field of view polydioptric camera, for which the motion estimation problem can be solved independently of the depth of the scene which leads to fast and robust algorithms for 3D Photography. In between are multiple view cameras with a large field of view which we have built, as well as omni-directional sensors.     

Omnidirectional video

Omnidirectional video enables direct surround immersive viewing of a scene by warping the original image into the correct perspective given a viewing direction. However, novel views from viewpoints off the camera path can only be obtained if we solve the three-dimensional motion and calibration problem. In this paper we address the case of a parabolic catadioptric camera – a paraboloidal mirror in front of an orthographic lens – and we introduce a new representation, called the circle space, for points and lines in such images. In this circle space, we formulate an epipolar constraint involving a 4×4 fundamental matrix. We prove that the intrinsic parameters can be inferred in closed form from the two-dimensional subspace of the new fundamental matrix from two views if they are constant or from three views if they vary. Three-dimensional motion and structure can then be estimated from the decomposition of the fundamental matrix.     

Spherical approximation for multiple cameras in motion estimation: Its applicability and advantages

Estimating motions of a multi-camera system which may not have overlapping fields of view is generally complex and computationally expensive because of the non-zero offset between each camera’s center. It is conceivable that if we can assume that multiple cameras share a single optical center, and thus can be modeled as a spherical imaging system, motion estimation and calibration of this system would become simpler and more efficient.     

Accurate Camera Calibration from Multi-View Stereo and Bundle Adjustment

The advent of high-resolution digital cameras and sophisticated multi-view stereo algorithms offers the promise of unprecedented geometric fidelity in image-based modeling tasks, but it also puts unprecedented demands on camera calibration to fulfill these promises. This paper presents a novel approach to camera calibration where top-down information from rough camera parameter estimates and the output of a multi-view-stereo system on scaled-down input images is used to effectively guide the search for additional image correspondences and significantly improve camera calibration parameters using a standard bundle adjustment algorithm (Lourakis and Argyros 2008). The proposed method has been tested on six real datasets including objects without salient features for which image correspondences cannot be found in a purely bottom-up fashion, and objects with high curvature and thin structures that are lost in visual hull construction even with small errors in camera parameters. Three different methods have been used to qualitatively assess the improvements of the camera parameters. The implementation of the proposed algorithm is publicly available at Furukawa and Ponce (2008b).     

Multi-camera calibration based on an invariant pattern

In this paper we present a method for the calibration of multiple cameras based on the extraction and use of the physical characteristics of a one-dimensional invariant pattern which is defined by four collinear markers. The advantages of this kind of pattern stand out in two key steps of the calibration process. In the initial step of camera calibration methods, related to sample points capture, the proposed method takes advantage of using a new technique for the capture and recognition of a robust sample of projective invariant patterns, which allows to capture simultaneously more than one invariant pattern in the tracking area and recognize each pattern individually as well as each marker that composes them. This process is executed in real time while capturing our sample of calibration points in the cameras of our system. This new feature allows to capture a more numerous and robust set of sample points than other patterns used for multi-camera calibration methods. In the last step of the calibration process, related to camera parameters' optimization, we explore the collinearity feature of the invariant pattern and add this feature in the camera parameters optimization model. This approach obtains better results in the computation of camera parameters. We present the results obtained with the calibration of two multi-camera systems using the proposed method and compare them with other methods from the literature.     

Easy Calibration of a Multi-projector Display System

In this paper, we present a method for the geometric calibration of a multi-projector display system. The method is such that in order to calibrate the system, the user is only required to place the projectors and capture a single image of the images projected from them onto a planar screen using a hand-held camera. The problem to be solved is divided into the image registration for stitching different projector images into a single seamless image and the image rectification for making the image have the correct rectangular shape. The proposed method is characterized by simultaneously solving both of them from only a single image, which makes the calibration procedures easy. The method assumes an uncalibrated camera and partially calibrated projectors in which only focal lengths are unknown among the internal parameters. In the paper, we first prove the uniqueness of solutions to the problem, which was unclear in the previous studies, and then present a stable numerical algorithm for actually finding the solution. We present several experimental results for synthetic data, in which we show the relation between the calibration accuracy and several factors, and also present experimental results for real data, in which we demonstrate that the proposed method can calibrate a real system with sufficient accuracy for a number of layouts of the projectors.     

Measuring and modelling sewer pipes from video

This article presents a system for the automatic measurement and modelling of sewer pipes. The system recovers the interior shape of a sewer pipe from a video sequence which is acquired by a fish-eye lens camera moving inside the pipe. The approach is based on tracking interest points across successive video frames and posing the general structure-from-motion problem. It is shown that the tracked points can be reliably reconstructed despite the forward motion of the camera. This is achieved by utilizing a fish-eye lens with a wide field of view. The standard techniques for robust estimation of the two- and three-view geometry are modified so that they can be used for calibrated fish-eye lens cameras with a field of view less than 180°. The tubular arrangement of the reconstructed points allows pipe shape estimation by surface fitting. Hence, a method for modelling such surfaces with a locally cylindrical model is proposed. The system is demonstrated with a real sewer video and an error analysis for the recovered structure is presented.     

基于图像运动的三维重建及虚拟化应用

提出了一种稳定、快速地获取摄像机视频运动图像的三维重建方法,并对该运动图像做适当的虚拟化处理以展示重建效果。采用基于尺度不变特征点匹配的摄像机标定进行三维重建。尺度不变特征对于视频图像中的特征具有优秀敏锐的匹配能力,极大地放宽了摄像机标定对于设备上的限制,拓宽了实时三维重建的适用范围。通过对系统的一系列优化,不但提升了三维重建的精度,减少了错误匹配对摄像机标定的影响,而且进一步提升了处理速度。通过在三维重建的基础之上进行虚拟化处理,展示了本系统的三维重建效果。实验结果表明,该系统适用范围广,处理速度较快,重建精度高,实现了基于视频运动图像的三维重建。     

Vanishing points and three-dimensional lines from omni-directional video

This paper describes a system for structure from motion using vanishing points and three-dimensional lines extracted from omni-directional video sequences. To track lines, we use a novel dynamic programming approach to improve ambiguity resolution, and we use delayed states to aid in the initialization of landmarks. By reobserving vanishing points we get direct measurements of the robots three-dimensional attitude that are independent of its position. Using vanishing points simplifies the representation since parallel lines share the same direction states. We show the performance of the system in various indoor and outdoor environments and include comparisons with independent two-dimensional reference maps for each experiment .     

三维动态脚型测量方法

基于多视几何原理来恢复和重建随时间变化的三维柔性物体是当前计算机视觉和三维扫描领域的研究热点之一,本文利用主动式标记点方法,提出并实现了在三维模型指导下鲁棒地从多视点视频中恢复和重建三维脚型的方法.它首先基于传统立体视觉方法重建出参考帧中的三维脚模型,然后在此基础上,通过构建三维模型顶点运动投影的速度矢量场来建立相邻帧...     

使用对极几何的多视角分布式视频编码

为了降低多视角分布式视频编码中解码器复杂度,解决视角间边信息的获取精度等问题,将一种新的基于对极几何的视差估计算法用于多视角分布式视频编码系统。提出的系统通过对视角间的冗余信息使用对极几何约束,采用新的视差搜索开始点,限制垂直方向的搜索范围以加快搜索速度。实验结果表明,提出的系统相对于运动JPEG（MJPEG）编码的主要运算绝对差值和（SAD）计算次数减少10倍,相对H.263+减少7倍。同时提出系统相对MJPEG编码有6～7 dB增益,相对H.263+有3 dB增益。     

Multi-view structure-from-motion for hybrid camera scenarios

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely omnidirectional and perspective cameras. For the steps of this pipeline, we propose new approaches or adapt the existing perspective camera methods to make the pipeline effective and automatic. We model our cameras of different types with the sphere camera model. To match feature points, we describe a preprocessing algorithm which significantly increases scale invariant feature transform (SIFT) matching performance for hybrid image pairs. With this approach, automatic point matching between omnidirectional and perspective images is achieved. We robustly estimate the hybrid fundamental matrix with the obtained point correspondences. We introduce the normalization matrices for lifted coordinates so that normalization and denormalization can be performed linearly for omnidirectional images. We evaluate the alternatives of estimating camera poses in hybrid pairs. A weighting strategy is proposed for iterative linear triangulation which improves the structure estimation accuracy. Following the addition of multiple perspective and omnidirectional images to the structure, we perform sparse bundle adjustment on the estimated structure by adapting it to use the sphere camera model. Demonstrations of the end-to-end multi-view SfM pipeline with the real images of mixed camera types are presented.     

Reconstructing camera projection matrices from multiple pairwise overlapping views

In this work we address the problem of projective reconstruction from multiple views with missing data. Factorization based algorithms require point correspondences across all the views. In many applications this is an unrealistic assumption. Current methods that solve the problem of projective reconstruction with missing data require correspondence information across triplets of images. We propose a projective reconstruction method that yields a consistent camera set given the fundamental matrices between pairs of views without directly using the image correspondences. The algorithm is based on breaking the reconstruction problem into small steps. In each step, we eliminate as much uncertainty as possible.     

Automatic reconstruction of 3D human motion pose from uncalibrated monocular video sequences based on markerless human motion tracking

We present a method to reconstruct human motion pose from uncalibrated monocular video sequences based on the morphing appearance model matching. The human pose estimation is made by integrated human joint tracking with pose reconstruction in depth-first order. Firstly, the Euler angles of joint are estimated by inverse kinematics based on human skeleton constrain. Then, the coordinates of pixels in the body segments in the scene are determined by forward kinematics, by projecting these pixels in the scene onto the image plane under the assumption of perspective projection to obtain the region of morphing appearance model in the image. Finally, the human motion pose can be reconstructed by histogram matching. The experimental results show that this method can obtain favorable reconstruction results on a number of complex human motion sequences.     

Inferring 3D structure from three points in rigid motion

We prove the following: Given four (or more) orthographic views of three points then (a) the views almost surely have no rigid interpretation but (b) if they do then they almost surely have at most thirty-two rigid interpretations. Part (a) means that the measure of false targets, viz., the measure of nonrigid motions that project to views having rigid interpretations, is zero. Part (b) means that rigid interpretations, when they exist, are not unique. Uniqueness of interpretation can be obtained if a point is added, but not if views are added. Our proof relies on an upper semicontinuity theorem for proper mappings of complex algebraic varieties. We note some psychophysical motivations of the theory.     

Determining shape and motion from non-overlapping multi-camera rig: A direct approach using normal flows

In this paper, we explore how a wide field-of-view imaging system that consists of a number of cameras in a network arranged to approximate a spherical eye can reduce the complexity of estimating camera motion. Depth map of the imaged scene can be reconstructed once the camera motion is there. We present a direct method to recover camera motion from video data, which neither requires establishment of feature correspondences nor recovery of optical flow, but from normal flow which is directly observable. With a wide visual field, the inherent ambiguities between translation and rotation disappear. Several subsets of normal flow pairs and triplets can be utilized to constraint the directions of translation and rotation separately. The intersection of solution spaces arising from normal flow pairs or triplets yields the estimate on the direction of motion. In addition, the larger number of normal flow measurements so resulted can be used to combat the local flow extraction error. Rotational magnitude is recovered in a subsequent stage. This article details how motion recovery can be improved with the use of such an approximate spherical imaging system. Experimental results on synthetic and real image data are provided. The results show that the accuracy of motion estimation is comparable to those of the state-of-the-art methods that require to use explicit feature correspondences or full optical flows, and our method has a much faster computational speed.     

Non-goal scene analysis for soccer video

The broadcast soccer video is usually recorded by one main camera, which is constantly gazing somewhere of playfield where a highlight event is happening. So the camera parameters and their variety have close relationship with semantic information of soccer video, and much interest has been caught in camera calibration for soccer video. The previous calibration methods either deal with goal scene, or have strict calibration conditions and high complexity. So, it does not properly handle the non-goal scene such as midfield or center-forward scene. In this paper, based on a new soccer field model, a field symbol extraction algorithm is proposed to extract the calibration information. Then a two-stage calibration approach is developed which can calibrate camera not only for goal scene but also for non-goal scene. The preliminary experimental results demonstrate its robustness and accuracy.     

An Iterative Multiresolution Scheme for SFM with Missing Data

Several techniques have been proposed for tackling the Structure from Motion problem through factorization in the case of missing data. However, when the percentage of unknown data is high, most of them may not perform as well as expected. Focussing on this problem, an iterative multiresolution scheme, which aims at recovering missing entries in the originally given input matrix, is proposed. Information recovered following a coarse-to-fine strategy is used for filling in the missing entries. The objective is to recover, as much as possible, missing data in the given matrix. Thus, when a factorization technique is applied to the partially or totally filled in matrix, instead of to the originally given input one, better results will be obtained. An evaluation study about the robustness to missing and noisy data is reported. Experimental results obtained with synthetic and real video sequences are presented to show the viability of the proposed approach.

Concise vector equations for stereopsis

The vector formulae for computing the spatial coordinates of a point from matched images are derived using a simple procedure, yielding compact expressions suitable for a vector co-processor. The relationship between accuracy of depth computation and image-matching algorithm parameters is also discussed.