Estimation of 3D Surface Shape and Smooth Radiance from 2D Images: A Level Set Approach

We cast the problem of shape reconstruction of a scene as the global region segmentation of a collection of calibrated images. We assume that the scene is composed of a number of smooth surfaces and a background, both of which support smooth Lambertian radiance functions. We formulate the problem in a variational framework, where the solution (both the shape and radiance of the scene) is a minimizer of a global cost functional which combines a geometric prior on shape, a smoothness prior on radiance and a data fitness score. We estimate the shape and radiance via an alternating minimization: The radiance is computed as the solutions of partial differential equations defined on the surface and the background. The shape is estimated using a gradient descent flow, which is implemented using the level set method. Our algorithm works for scenes with smooth radiances as well as fine homogeneous textures, which are known challenges to traditional stereo algorithms based on local correspondence.     

3-D Reconstruction of Shaded Objects from Multiple Images Under Unknown Illumination

We propose a variational algorithm to jointly estimate the shape, albedo, and light configuration of a Lambertian scene from a collection of images taken from different vantage points. Our work can be thought of as extending classical multi-view stereo to cases where point correspondence cannot be established, or extending classical shape from shading to the case of multiple views with unknown light sources. We show that a first naive formalization of this problem yields algorithms that are numerically unstable, no matter how close the initialization is to the true geometry. We then propose a computational scheme to overcome this problem, resulting in provably stable algorithms that converge to (local) minima of the cost functional. We develop a new model that explicitly enforces positivity in the light sources with the assumption that the object is Lambertian and its albedo is piecewise constant and show that the new model significantly improves the accuracy and robustness relative to existing approaches.     

Photo Hull regularized stereo

A regularization-based approach to 3D reconstruction from multiple images is proposed. As one of the most widely used multiple-view 3D reconstruction algorithms, Space Carving can produce a Photo Hull of a scene, which is at best a coarse volumetric model. The two-view stereo algorithm, on the other hand, can generate a more accurate reconstruction of the surfaces, provided that a given surface is visible to both views. The proposed method is essentially a data fusion approach to 3D reconstruction, combining the above two algorithms by means of regularization. The process is divided into two steps: (1) computing the Photo Hull from multiple calibrated images and (2) selecting two of the images as input and solving the two-view stereo problem by global optimization, using the Photo Hull as the regularizer. Our dynamic programming implementation of this regularization-based stereo approach potentially provides an efficient and robust way of reconstructing 3D surfaces. The results of an implementation of this theory is presented on real data sets and compared with peer algorithms.     

Omnivergent Stereo

The notion of a virtual camera for optimal 3D reconstruction is introduced. Instead of planar perspective images that collect many rays at a fixed viewpoint, omnivergent cameras collect a small number of rays at many different viewpoints. The resulting 2D manifold of rays is arranged into two multiple-perspective images for stereo reconstruction. We call such images omnivergent images, and the process of reconstructing the scene from such images omnivergent stereo. This procedure is shown to produce 3D scene models with minimal reconstruction error, due to the fact that for any point in the 3D scene, two rays with maximum vergence angle can be found in the omnivergent images. Furthermore, omnivergent images are shown to have horizontal epipolar lines, enabling the application of traditional stereo matching algorithms, without modification. Three types of omnivergent virtual cameras are presented: spherical omnivergent cameras,center-strip cameras and dual-strip cameras.     

Some experiments on the detection of three-dimensional motion of rigid objects in blocks world

The present paper describes some major steps in our experiment of a two-frame based approach analysis. Its application deals with 3-D time-varing scene analysis. The entire analysis process can be divided into several major steps. Here, we are concerned with four main steps: preprocessing, knowledge representation, features matching and motion estimation. The work is directly related to the problem encountered by researchers in machine intelligence area where a vision system is indispensable. For each step, experimental results are given to illustrate the performance of the algorithms of processing. Furthermore, the implemented algorithms provide somewhat versatility and flexibility in the sense that they can be applied to other tasks of scene analysis, such as: stereo vision, object recognition and dynamic scene segmentation since the problem in determining the movement of an object using successive images is similar in many ways to the problem met in optic flow analysis and stereopsis. Finally, it should be pointed out that a vision system can be easily built when combining all of these available algorithms.     

A Roadmap to the Integration of Early Visual Modules

By examining the problem of image correspondence (binocular stereo and optical flow) and its relationship with other modules such as segmentation, shape and depth estimation, occlusion detection, and local signal processing, we argue that early visual modules are entangled in chicken-and-egg relationships, and unraveling these necessitates a compositional approach. In this paper, we present compositional algorithms which can match images containing slanted surfaces and images having different contrast, while simultaneously solving other problems as part of the same process. Ultimately, our goal is to motivate the application of the compositional approach to unify many other early visual modules. Experimental results have been presented on a large variety of stereo and motion images, including images with contrast mismatch and images containing untextured slanted surfaces.     

Mumford-Shah on the Move: Region-Based Segmentation on Deforming Manifolds with Application to 3-D Reconstruction of Shape and Appearance from Multi-View Images

We address the problem of estimating the shape and appearance of a scene made of smooth Lambertian surfaces with piecewise smooth albedo. We allow the scene to have self-occlusions and multiple connected components. This class of surfaces is often used as an approximation of scenes populated by man-made objects. We assume we are given a number of images taken from different vantage points. Mathematically this problem can be posed as an extension of Mumford and Shah’s approach to static image segmentation to the segmentation of a function defined on a deforming surface. We propose an iterative procedure to minimize a global cost functional that combines geometric priors on both the shape of the scene and the boundary between smooth albedo regions. We carry out the numerical implementation in the level set framework.     

Progressively weighted affine adaptive correlation matching for quasi-dense 3D reconstruction

Correlation matching has been widely accepted as a rudimentary similarity measure to obtain dense 3D reconstruction from a stereo pair. In particular, given a large overlapping area between images with minimal scale differences, the correlation results followed by a geometrically constrained global optimisation delivers adequately dense and accurate reconstruction results. In order to achieve greater reliability, however, correlation matching should correctly account for the geometrical distortion introduced by the different viewing angles of the stereo or multi-view sensors. Conventional adaptive least squares correlation (ALSC) matching addresses this by modifying the shape of a matching window iteratively, assuming that the distortion can be approximated by an affine transform. Nevertheless, since an image captured from different viewing angle is often not practically identical due to scene occlusions, the matching confidence normally deteriorates. Subsequently, it affects the density of the reconstruction results from ALSC-based stereo region growing algorithms. To address this, we propose an advanced ALSC matching method that can progressively update matching weight for each pixel in an aggregating window using a relaxation labelling technique. The experimental results show that the proposed method can improve matching performance, which consequently enhances the quality of stereo reconstruction. Also, the results demonstrate its ability to refine a scale invariant conjugate point pair to an affine and scale invariant point pair.     

Bayesian shape estimation: Shape-from-shading and photometric stereo revisited

We present a Bayesian approach to the machine vision processes of shape-from-shading and photometric stereo, also considering the associated question of the detection of shape discontinuities. The shape reconstruction problem is formulated as a maximum a posteriori (MAP) estimation from probability distributions of Gibbs form, and is solved via simulated annealing. In shape-from-shading, our formulation leads to a constrained optimization problem, where the constraints come from the image irradiance equation and from the incorporation of the necessary boundary conditions. In photometric stereo, we are able to estimate shape directly from degraded input images. We also propose an edge-detection algorithm that works cooperatively with the reconstruction process, employing the shape estimates to locate the discontinuities of the reconstructed surface. We show results of the application of our framework both to synthetic and to real imagery.