Generalised Epipolar Constraints

In this paper we will discuss structure and motion problems for curved surfaces. These will be studied using the silhouettes or apparent contours in the images. The problem of determining camera motion from the apparent contours of curved three-dimensional surfaces, is studied. It will be shown how special points, called epipolar tangency points or frontier points, can be used to solve this problem. A generalised epipolar constraint is introduced, which applies to points, curves, as well as to apparent contours of surfaces. The theory is developed for both continuous and discrete motion, known and unknown orientation, calibrated and uncalibrated, perspective, weak perspective and orthographic cameras. Results of an iterative scheme to recover the epipolar line structure from real image sequences using only the outlines of curved surfaces, is presented. A statistical evaluation is performed to estimate the stability of the solution. It is also shown how the motion of the camera from a sequence of images can be obtained from the relative motion between image pairs.     

Motion estimation in image sequences using the deformation ofapparent contours

The problem of determining the camera motion from apparent contours or silhouettes of a priori unknown curved 3D surfaces is considered. In a sequence of images, it is shown how to use the generalized epipolar constraint on apparent contours. One such constraint is obtained for each epipolar tangency point in each image pair. An accurate algorithm for computing the motion is presented based on a maximum likelihood estimate. It is shown how to generate initial estimates on the camera motion using only the tracked contours. It is also shown that in theory the motion can be calculated from the deformation of a single contour. The algorithm has been tested on several real image sequences, for both Euclidean and projective reconstruction. The resulting motion estimate is compared to motion estimates calculated independently using standard feature-based methods. The motion estimate is also used to classify the silhouettes as curves or apparent contours. The statistical evaluation shows that the technique gives accurate and stable results     

Arbitrary viewpoint video synthesis from multiple uncalibrated cameras.

We propose a method for arbitrary view synthesis from uncalibrated multiple camera system, targeting large spaces such as soccer stadiums. In Projective Grid Space (PGS), which is a three-dimensional space defined by epipolar geometry between two basis cameras in the camera system, we reconstruct three-dimensional shape models from silhouette images. Using the three-dimensional shape models reconstructed in the PGS, we obtain a dense map of the point correspondence between reference images. The obtained correspondence can synthesize the image of arbitrary view between the reference images. We also propose a method for merging the synthesized images with the virtual background scene in the PGS. We apply the proposed methods to image sequences taken by a multiple camera system, which installed in a large concert hall. The synthesized image sequences of virtual camera have enough quality to demonstrate effectiveness of the proposed method.     

Self-Calibration of Turntable Sequences from Silhouettes

This paper addresses the problem of recovering both the intrinsic and extrinsic parameters of a camera from the silhouettes of an object in a turntable sequence. Previous silhouette-based approaches have exploited correspondences induced by epipolar tangents to estimate the image invariants under turntable motion and achieved a weak calibration of the cameras. It is known that the fundamental matrix relating any two views in a turntable sequence can be expressed explicitly in terms of the image invariants, the rotation angle, and a fixed scalar. It will be shown that the imaged circular points for the turntable plane can also be formulated in terms of the same image invariants and fixed scalar. This allows the imaged circular points to be recovered directly from the estimated image invariants, and provide constraints for the estimation of the imaged absolute conic. The camera calibration matrix can thus be recovered. A robust method for estimating the fixed scalar from image triplets is introduced, and a method for recovering the rotation angles using the estimated imaged circular points and epipoles is presented. Using the estimated camera intrinsics and extrinsics, a Euclidean reconstruction can be obtained. Experimental results on real data sequences are presented, which demonstrate the high precision achieved by the proposed method.     

基于图像轮廓的三维重建方法

根据平面镜成像原理,提出了一种基于单幅图像的侧影轮廓线可见外壳重建方法。利用成角度平面镜装置模拟实现多相机同时拍摄,对图像中各轮廓线之间极线几何关系的分析得到对应的相机参数,实现目标物体的三维重建。实验结果表明,该方法简单有效,不需要通过实验室条件下的特殊仪器进行标定即可实现目标物体的三维重建,具有一定的实用价值。     

Uncalibrated Motion Capture Exploiting Articulated Structure Constraints

We present an algorithm for 3D reconstruction of dynamic articulated structures, such as humans, from uncalibrated multiple views. The reconstruction exploits constraints associated with a dynamic articulated structure, specifically the conservation over time of length between rotational joints. These constraints admit reconstruction of metric structure from at least two different images in each of two uncalibrated parallel projection cameras. As a by product, the calibration of the cameras can also be computed. The algorithm is based on a stratified approach, starting with affine reconstruction from factorization, followed by rectification to metric structure using the articulated structure constraints. The exploitation of these specific constraints admits reconstruction and self-calibration with fewer feature points and views compared to standard self-calibration. The method is extended to pairs of cameras that are zooming, where calibration of the cameras allows compensation for the changing scale factor in a scaled orthographic camera. Results are presented in the form of stick figures and animated 3D reconstructions using pairs of sequences from broadcast television. The technique shows promise as a means of creating 3D animations of dynamic activities such as sports events.     

Estimation of the fundamental matrix from uncalibrated stereo hand images for 3D hand gesture recognition

All 3D hand models employed for hand gesture recognition so far use kinematic models of the hand. We propose to use computer vision models of the hand, and recover hand gestures using 3D reconstruction techniques. In this paper, we present a new method to estimate the epipolar geometry between two uncalibrated cameras from stereo hand images. We first segmented hand images using the RCE neural network based color segmentation algorithm and extracted edge points of fingers as points of interest, then match them based on the topological features of the hand. The fundamental matrix is estimated using a combination of techniques such as input data normalization, rank-2 constraint, linear criterion, nonlinear criterion as well as M-estimator. This method has been tested with real calibrated and uncalibrated images. The experimental comparison demonstrates the effectiveness and robustness of the method.     

单轴运动下未标定图像序列的度量重构

提出了一种由单轴运动下的未标定运动图像序列进行三维重构的新方法。利用不变量计算出第一幅图像对应的摄像机矩阵,根据转台的旋转角度计算出其余图像对应的摄像机矩阵,再使用SIFT算法查找相邻两幅图像可见的匹配点,并利用线性三角形法对图像进行重构,最后用捆集调整优化重构的结果。真实图像序列的重构实验验证了算法的正确性和有效性。     

Camera Network Calibration and Synchronization from Silhouettes in Archived Video

In this paper we present an automatic method for calibrating a network of cameras that works by analyzing only the motion of silhouettes in the multiple video streams. This is particularly useful for automatic reconstruction of a dynamic event using a camera network in a situation where pre-calibration of the cameras is impractical or even impossible. The key contribution of this work is a RANSAC-based algorithm that simultaneously computes the epipolar geometry and synchronization of a pair of cameras only from the motion of silhouettes in video. Our approach involves first independently computing the fundamental matrix and synchronization for multiple pairs of cameras in the network. In the next stage the calibration and synchronization for the complete network is recovered from the pairwise information. Finally, a visual-hull algorithm is used to reconstruct the shape of the dynamic object from its silhouettes in video. For unsynchronized video streams with sub-frame temporal offsets, we interpolate silhouettes between successive frames to get more accurate visual hulls. We show the effectiveness of our method by remotely calibrating several different indoor camera networks from archived video streams.     

Structure and motion estimation from apparent contours under circular motion

In this paper, we address the problem of recovering structure and motion from the apparent contours of a smooth surface. Fixed image features under circular motion and their relationships with the intrinsic parameters of the camera are exploited to provide a simple parameterization of the fundamental matrix relating any pair of views in the sequence. Such a parameterization allows a trivial initialization of the motion parameters, which all bear physical meaning. It also greatly reduces the dimension of the search space for the optimization problem, which can now be solved using only two epipolar tangents. In contrast to previous methods, the motion estimation algorithm introduced here can cope with incomplete circular motion and more widely spaced images. Existing techniques for model reconstruction from apparent contours are then reviewed and compared. Experiment on real data has been carried out and the 3D model reconstructed from the estimated motion is presented.     

Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.     

1D camera geometry and its application to the self-calibration of circular motion sequences

This paper proposes a novel method for robustly recovering the camera geometry of an uncalibrated image sequence taken under circular motion. Under circular motion, all the camera centers lie on a circle and the mapping from the plane containing this circle to the horizon line observed in the image can be modelled as a 1D projection. A 2x2 homography is introduced in this paper to relate the projections of the camera centers in two 1D views. It is shown that the two imaged circular points of the motion plane and the rotation angle between the two views can be derived directly from such a homography. This way of recovering the imaged circular points and rotation angles is intrinsically a multiple view approach, as all the sequence geometry embedded in the epipoles is exploited in the estimation of the homography for each view pair. This results in a more robust method compared to those computing the rotation angles using adjacent views only. The proposed method has been applied to self-calibrate turntable sequences using either point features or silhouettes, and highly accurate results have been achieved.     

Simultaneous two-view epipolar geometry estimation and motion segmentation by 4D tensor voting

We address the problem of simultaneous two-view epipolar geometry estimation and motion segmentation from nonstatic scenes. Given a set of noisy image pairs containing matches of n objects, we propose an unconventional, efficient, and robust method, 4D tensor voting, for estimating the unknown n epipolar geometries, and segmenting the static and motion matching pairs into n, independent motions. By considering the 4D isotropic and orthogonal joint image space, only two tensor voting passes are needed, and a very high noise to signal ratio (up to five) can be tolerated. Epipolar geometries corresponding to multiple, rigid motions are extracted in succession. Only two uncalibrated frames are needed, and no simplifying assumption (such as affine camera model or homographic model between images) other than the pin-hole camera model is made. Our novel approach consists of propagating a local geometric smoothness constraint in the 4D joint image space, followed by global consistency enforcement for extracting the fundamental matrices corresponding to independent motions. We have performed extensive experiments to compare our method with some representative algorithms to show that better performance on nonstatic scenes are achieved. Results on challenging data sets are presented.     

Animated solid model of the lung constructed from unsynchronized MR sequential images

This work discusses a 4D lung reconstruction method from unsynchronized MR sequential images. The lung, differently from the heart, does not have its own muscles, turning impossible to see its real movements. The visualization of the lung in motion is an actual topic of research in medicine. CT (Computerized Tomography) can obtain spatio-temporal images of the heart by synchronizing with electrocardiographic waves. The FOV of the heart is small when compared to the lung’s FOV. The lung’s movement is not periodic and is susceptible to variations in the degree of respiration. Compared to CT, MR (Magnetic Resonance) imaging involves longer acquisition times and it is not possible to obtain instantaneous 3D images of the lung. For each slice, only one temporal sequence of 2D images can be obtained. However, methods using MR are preferable because they do not involve radiation. In this paper, based on unsynchronized MR images of the lung an animated B-Rep solid model of the lung is created. The 3D animation represents the lung’s motion associated to one selected sequence of MR images. The proposed method can be divided in two parts. First, the lung’s silhouettes moving in time are extracted by detecting the presence of a respiratory pattern on 2D spatio-temporal MR images. This approach enables us to determine the lung’s silhouette for every frame, even on frames with obscure edges. The sequence of extracted lung’s silhouettes are unsynchronized sagittal and coronal silhouettes. Using our algorithm it is possible to reconstruct a 3D lung starting from a silhouette of any type (coronal or sagittal) selected from any instant in time. A wire-frame model of the lung is created by composing coronal and sagittal planar silhouettes representing cross-sections. The silhouette composition is severely underconstrained. Many wire-frame models can be created from the observed sequences of silhouettes in time. Finally, a B-Rep solid model is created using a meshing algorithm. Using the B-Rep solid model the volume in time for the right and left lungs were calculated. It was possible to recognize several characteristics of the 3D real right and left lungs in the shaded model.     

Affine reconstruction of curved surfaces from uncalibrated views ofapparent contours

In this paper, we consider uncalibrated reconstruction of curved surfaces from apparent contours. Since apparent contours are not fixed features (viewpoint independent), we cannot directly apply the recent results of the uncalibrated reconstruction from fixed features. We show that, nonetheless, curved surfaces can be reconstructed up to an affine ambiguity from their apparent contours viewed from uncalibrated cameras with unknown linear translations. Furthermore, we show that, even if the reconstruction is nonmetric (non-Euclidean), we can still extract useful information for many computer vision applications just from the apparent contours. We first show that if the camera motion is linear translation (but arbitrary direction and magnitude), the epipolar geometry can be recovered from the apparent contours without using any optimization process. The extracted epipolar geometry is next used for reconstructing curved surfaces from the deformations of the apparent contours viewed from uncalibrated cameras. The result is applied to distinguishing curved surfaces from fixed features in images. It is also shown that the time-to-contact to the curved surfaces can be computed from simple measurements of the apparent contours     

Shape recovery from turntable sequence using rim reconstruction

This paper makes use of both feature points and silhouettes to deliver fast 3D shape recovery. The algorithm exploits object silhouettes in two views to establish a 3D rim curve, which is defined with respect to the two frontier points arising from two views. The images of this 3D rim curve in the two views are matched using cross-correlation technique. A 3D planar rim curve is then reconstructed using point-based reconstruction method. A set of 3D rim curves enclosing the object can be obtained from an image sequence captured under circular motion. Silhouettes are further utilized to check for mismatched rim points. The proposed method solves the problem of reconstruction of concave object surface, which is usually left unresolved in general silhouette-based reconstruction methods. In addition, the property of the reconstructed 3D rim curves allows fast surface extraction. Experimental results with real data are presented.     

Experimenting with nonintrusive motion capture in a virtual environment

A growing number of promising applications requires recognizing human posture and motion. Conventional techniques require us to attach foreign objects to the body, which in some applications is disturbing or even impossible. New, nonintrusive motion capture approaches are called for. The well-known shape-from-silhouette technique for understanding 3D shapes could also be effective for human bodies. We present a novel technique for model-based motion capture that uses silhouettes extracted from multiple views. A 3D reconstruction of the performer can be computed from a silhouette with a technique known as volume intersection. We can recover the posture by fitting a model of the human body to the reconstructed volume. The purpose of this work is to test the effectiveness of this approach in a virtual environment by investigating the precision of the posture and motion obtained with various numbers and arrangements of stationary cameras. An average 1% position error has been obtained with five cameras.     

基于场景几何约束未标定两视图的三维模型重建

提出了一种从两幅未标定图象重建场景三维模型的方法 .这种方法充分利用了人造结构场景中大量存在的平行性和正交性几何约束 ,即利用每幅视图中三组互相垂直的平行线 ,计算出 3个影灭点 ,从而对每幅视图进行标定 .对两幅未标定图象 ,从基本矩阵只能得到射影重构 ,如果每幅图象都已标定 ,则可将基本矩阵转化为本质矩阵 .三维重构过程有两个步骤 :先是恢复相机的位置和运动 ;后是用三角测量法计算出点的三维坐标 .对多平面组成的场景进行三维重构实验 ,所得三维模型产生新的视点图象 ,与所观察的场景一致 ,重构的两个平面夹角与实际值相近 ,实验结果表明 ,该算法是行之有效的