The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows

We present an algorithm for separating the local gradient information and Lambertian color by using 4-source color photometric stereo in the presence of highlights and shadows. We assume that the surface reflectance can be approximated by the sum of a Lambertian and a specular component. The conventional photometric method is generalized for color images. Shadows and highlights in the input images are detected using either spectral or directional cues and excluded from the recovery process, thus giving more reliable estimates of local surface parameters.     

Photometric stereo with an arbitrary number of illuminants

We present an optimal generalisation of the 4-light photometric stereo technique for an arbitrary number of Q illuminants. We assume that the surface reflectance can be approximated by the Lambertian model plus a specular reflection. The algorithm works in a recursive manner eliminating the pixel intensities affected by shadows or highlights, based on a least squares error technique, retaining only the information coming from illumination directions that can be used for photometric stereo reconstruction of the normal of the corresponding surface patch. We report results for both simulated and real surfaces and compare them with the results of other state of the art photometric stereo algorithms.     

Appearance characterization of linear Lambertian objects, generalized photometric stereo, and illumination-invariant face recognition

Traditional photometric stereo algorithms employ a Lambertian reflectance model with a varying albedo field and involve the appearance of only one object. In this paper, we generalize photometric stereo algorithms to handle all appearances of all objects in a class, in particular the human face class, by making use of the linear Lambertian property. A linear Lambertian object is one which is linearly spanned by a set of basis objects and has a Lambertian surface. The linear property leads to a rank constraint and, consequently, a factorization of an observation matrix that consists of exemplar images of different objects (e.g., faces of different subjects) under different, unknown illuminations. Integrability and symmetry constraints are used to fully recover the subspace bases using a novel linearized algorithm that takes the varying albedo field into account. The effectiveness of the linear Lambertian property is further investigated by using it for the problem of illumination-invariant face recognition using just one image. Attached shadows are incorporated in the model by a careful treatment of the inherent nonlinearity in Lambert's law. This enables us to extend our algorithm to perform face recognition in the presence of multiple illumination sources. Experimental results using standard data sets are presented     

Solving Uncalibrated Photometric Stereo Using Fewer Images by Jointly Optimizing Low-rank Matrix Completion and Integrability

We introduce a new, integrated approach to uncalibrated photometric stereo. We perform 3D reconstruction of Lambertian objects using multiple images produced by unknown, directional light sources. We show how to formulate a single optimization that includes rank and integrability constraints, allowing also for missing data. We then solve this optimization using the Alternating Direction Method of Multipliers (ADMM). We conduct extensive experimental evaluation on real and synthetic data sets. Our integrated approach is particularly valuable when performing photometric stereo using as few as 4–6 images, since the integrability constraint is capable of improving estimation of the linear subspace of possible solutions. We show good improvements over prior work in these cases.     

Optimal Reconstruction of Approximate Planar Surfaces Using Photometric Stereo

Photometric stereo can be used to obtain a fast and noncontact surface reconstruction of Lambertian surfaces. Despite several published works concerning the uncertainties and optimal light configurations of photometric stereo, no solutions for optimal surface reconstruction from noisy real images have been proposed. In this paper, optimal surface reconstruction methods for approximate planar textured surfaces using photometric stereo are derived, given that the statistics of imaging errors are measurable. Simulated and real surfaces are experimentally studied, and the results validate that the proposed approaches improve the surface reconstruction especially for the high-frequency height variations.     

Robust 3D face capture using example-based photometric stereo

We show that using example-based photometric stereo, it is possible to achieve realistic reconstructions of the human face. The method can handle non-Lambertian reflectance and attached shadows after a simple calibration step. We use spherical harmonics to model and de-noise the illumination functions from images of a reference object with known shape, and a fast grid technique to invert those functions and recover the surface normal for each point of the target object. The depth coordinate is obtained by weighted multi-scale integration of these normals, using an integration weight mask obtained automatically from the images themselves. We have applied these techniques to improve the PhotoFace system of Hansen et al. (2010).     

Three-dimensional shape from color photometric stereo

Computer vision systems can be used to determine the shapes of real three-dimensional objects for purposes of object recognition and pose estimation or for CAD applications. One method that has been developed is photometric stereo. This method uses several images taken from the same viewpoint, but with different lightings, to determine the three-dimensional shape of an object. Most previous work in photometric stereo has been with gray-tone images; color images have only been used for dielectric materials. In this paper we describe a procedure for color photometric stereo, which recovers the shape of a colored object from two or more color images of the object under white illumination. This method can handle different types of materials, such as composites and metals, and can employ various reflection models such as the Lambertian, dichromatic, and Torrance-Sparrow models. For composite materials, colored metals, and dielectrics, there are two advantages of utilizing color information: at each pixel, there are more constraints on the orientation, and the result is less sensitive to noise. Consequently, the shape can be found more accurately. The method has been tested on both artificial and real images of objects of various materials, and on real images of a multi-colored object.     

Simultaneous estimation of shape and reflectance map from photometric stereo

This paper investigates whether the shape of an object and certain parameters of its reflectance map can be simultaneously estimated using photometric stereo. This problem has been addressed in the literature for the case where the Lambertian and non-Lambertian components in the image can be easily separated. No such separability is assumed in this paper. A class of reflectance maps for modeling diffusely reflecting surfaces is proposed. This class is based on the physics of scattering from real world surfaces. Next, the problem of joint estimation of some parameters of the map along with the surface shape is analyzed. A bound is obtained on the number of light sources necessary for a unique solution to the problem. The analysis also reveals that some of the estimates can be obtained by a nonparametric method. The behavior of the estimates in the presence of noise is also investigated. It is shown that simultaneous estimation is ill-posed. Regularizing the estimates yields good reconstructions from real world data.     

3D Shape Recovery of Smooth Surfaces: Dropping the Fixed-Viewpoint Assumption

We present a new method for recovering the 3D shape of a featureless smooth surface from three or more calibrated images illuminated by different light sources (three of them are independent). This method is unique in its ability to handle images taken from unconstrained perspective viewpoints and unconstrained illumination directions. The correspondence between such images is hard to compute and no other known method can handle this problem locally from a small number of images. Our method combines geometric and photometric information in order to recover dense correspondence between the images and accurately computes the 3D shape. Only a single pass starting at one point and local computation are used. This is in contrast to methods that use the occluding contours recovered from many images to initialize and constrain an optimization process. The output of our method can be used to initialize such processes. In the special case of fixed viewpoint, the proposed method becomes a new perspective photometric stereo algorithm. Nevertheless, the introduction of the multiview setup, self-occlusions, and regions close to the occluding boundaries are better handled, and the method is more robust to noise than photometric stereo. Experimental results are presented for simulated and real images.     

Acquiring a radiance distribution to superimpose virtual objectsonto a real scene

This paper describes a new method for superimposing virtual objects with correct shadings onto an image of a real scene. Unlike the previously proposed methods, our method can measure a radiance distribution of a real scene automatically and use it for superimposing virtual objects appropriately onto a real scene. First, a geometric model of the scene is constructed from a pair of omnidirectional images by using an omnidirectional stereo algorithm. Then, radiance of the scene is computed from a sequence of omnidirectional images taken with different shutter speeds and mapped onto the constructed geometric model. The radiance distribution mapped onto the geometric model is used for rendering virtual objects superimposed onto the scene image. As a result, even for a complex radiance distribution, our method can superimpose virtual objects with convincing shadings and shadows cast onto the real scene. We successfully tested the proposed method by using real images to show its effectiveness     

Determining Generative Models of Objects Under Varying Illumination: Shape and Albedo from Multiple Images Using SVD and Integrability

We describe a method of learning generative models of objects from a set of images of the object under different, and unknown, illumination. Such a model allows us to approximate the objects' appearance under a range of lighting conditions. This work is closely related to photometric stereo with unknown light sources and, in particular, to the use of Singular Value Decomposition (SVD) to estimate shape and albedo from multiple images up to a linear transformation (Hayakawa, 1994). Firstly we analyze and extend the SVD approach to this problem. We demonstrate that it applies to objects for which the dominant imaging effects are Lambertian reflectance with a distant light source and a background ambient term. To determine that this is a reasonable approximation we calculate the eigenvectors of the SVD on a set of real objects, under varying lighting conditions, and demonstrate that the first few eigenvectors account for most of the data in agreement with our predictions. We then analyze the linear ambiguities in the SVD approach and demonstrate that previous methods proposed to resolve them (Hayakawa, 1994) are only valid under certain conditions. We discuss alternative possibilities and, in particular, demonstrate that knowledge of the object class is sufficient to resolve this problem. Secondly, we describe the use of surface consistency for putting constraints on the possible solutions. We prove that this constraint reduces the ambiguities to a subspace called the generalized bas relief ambiguity (GBR) which is inherent in the Lambertian reflectance function (and which can be shown to exist even if attached and cast shadows are present (Belhumeur et al., 1997)). We demonstrate the use of surface consistency to solve for the shape and albedo up to a GBR and describe, and implement, a variety of additional assumptions to resolve the GBR. Thirdly, we demonstrate an iterative algorithm that can detect and remove some attached shadows from the objects thereby increasing the accuracy of the reconstructed shape and albedo.     

Shape Reconstruction of 3D Bilaterally Symmetric Surfaces

The paper presents a new approach for shape recovery based on integrating geometric and photometric information. We consider 3D bilaterally symmetric objects, that is, objects which are symmetric with respect to a plane (e.g., faces), and their reconstruction from a single image. Both the viewpoint and the illumination are not necessarily frontal. Furthermore, no correspondence between symmetric points is required.The basic idea is that an image taken from a general, non frontal viewpoint, under non-frontal illumination can be regarded as a pair of images. Each image of the pair is one half of the object, taken from different viewing positions and with different lighting directions. Thus, one-image-variants of geometric stereo and of photometric stereo can be used. Unlike the separate invocation of these approaches, which require point correspondence between the two images, we show that integrating the photometric and geometric information suffice to yield a dense correspondence between pairs of symmetric points, and as a result, a dense shape recovery of the object. Furthermore, the unknown lighting and viewing parameters, are also recovered in this process.Unknown distant point light source, Lambertian surfaces, unknown constant albedo, and weak perspective projection are assumed. The method has been implemented and tested experimentally on simulated and real data.     

Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.     

Stereo Image Analysis of Non-Lambertian Surfaces

Stereo image analysis is based on establishing correspondences between a pair of images by determining similarity measures for potentially corresponding image parts. Such similarity criteria are only strictly valid for surfaces with Lambertian (diffuse) reflectance characteristics. Specular reflections are viewpoint dependent and may thus cause large intensity differences at corresponding image points. In the presence of specular reflections, traditional stereo approaches are often unable to establish correspondences at all, or the inferred disparity values tend to be inaccurate, or the established correspondences do not belong to the same physical surface point. The stereo image analysis framework for non-Lambertian surfaces presented in this contribution combines geometric cues with photometric and polarimetric information into an iterative scheme that allows to establish stereo correspondences in accordance with the specular reflectance behaviour and at the same time to determine the surface gradient field based on the known photometric and polarimetric reflectance properties. The described approach yields a dense 3D reconstruction of the surface which is consistent with all observed geometric and photopolarimetric data. Initially, a sparse 3D point cloud of the surface is computed by traditional blockmatching stereo. Subsequently, a dense 3D profile of the surface is determined in the coordinate system of camera 1 based on the shape from photopolarimetric reflectance and depth technique. A synthetic image of the surface is rendered in the coordinate system of camera 2 using the illumination direction and reflectance properties of the surface material. Point correspondences between the rendered image and the observed image of camera 2 are established with the blockmatching technique. This procedure yields an increased number of 3D points of higher accuracy, compared to the initial 3D point cloud. The improved 3D point cloud is used to compute a refined dense 3D surface profile. These steps are iterated until convergence of the 3D reconstruction. An experimental evaluation of our method is provided for areas of several square centimetres of forged and cast iron objects with rough surfaces displaying both diffuse and significant specular reflectance components, where traditional stereo image analysis largely fails. A comparison to independently measured ground truth data reveals that the root-mean-square error of the 3D reconstruction results is typically of the order 30–100 μm at a lateral pixel resolution of 86 μm. For two example surfaces, the number of stereo correspondences established by the specular stereo algorithm is several orders of magnitude higher than the initial number of 3D points. For one example surface, the number of stereo correspondences decreases by a factor of about two, but the 3D point cloud obtained with the specular stereo method is less noisy, contains a negligible number of outliers, and shows significantly more surface detail than the initial 3D point cloud. For poorly known reflectance parameters we observe a graceful degradation of the accuracy of 3D reconstruction.     

Overcoming shadows in 3-source photometric stereo

Light occlusions are one of the most significant difficulties of photometric stereo methods. When three or more images are available without occlusion, the local surface orientation is overdetermined so that shape can be computed and the shadowed pixels can be discarded. In this paper, we look at the challenging case when only two images are available without occlusion, leading to a one degree of freedom ambiguity per pixel in the local orientation. We show that, in the presence of noise, integrability alone cannot resolve this ambiguity and reconstruct the geometry in the shadowed regions. As the problem is ill-posed in the presence of noise, we describe two regularization schemes that improve the numerical performance of the algorithm while preserving the data. Finally, the paper describes how this theory applies in the framework of color photometric stereo where one is restricted to only three images and light occlusions are common. Experiments on synthetic and real image sequences are presented.     

Capture and Synthesis of 3D Surface Texture

We present and compare five approaches for capturing, synthesising and relighting real 3D surface textures. Unlike 2D texture synthesis techniques they allow the captured textures to be relit using illumination conditions that differ from those of the original. We adapted a texture quilting method due to Efros and combined this with five different relighting representations, comprising: a set of three photometric images; surface gradient and albedo maps; polynomial texture maps; and two eigen based representations using 3 and 6 base images.We used twelve real textures to perform quantitative tests on the relighting methods in isolation. We developed a qualitative test for the assessment of the complete synthesis systems. Ten observers were asked to rank the images obtained from the five methods using five real textures. Statistical tests were applied to the rankings.The six-base-image eigen method produced the best quantitative relighting results and in particular was better able to cope with specular surfaces. However, in the qualitative tests there were no significant performance differences detected between it and the other two top performers. Our conclusion is therefore that the cheaper gradient and three-base-image eigen methods should be used in preference, especially where the surfaces are Lambertian or near Lambertian.     

Translational photometric alignment of single-view image sequences

Photometric stereo is a well-established method to estimate surface normals of an object. When coupled with depth-map estimation, it can be used to reconstruct an object’s height field. Typically, photometric stereo requires an image sequence of an object under the same viewpoint but with differing illumination directions. One crucial assumption of this configuration is perfect pixel correspondence across images in the sequence. While this assumption is often satisfied, certain setups are susceptible to translational errors or misalignments across images. Current methods to align image sequences were not designed specifically for single-view photometric stereo. Thus, they either struggle to account for changing illumination across images, require training sets, or are overly complex for these conditions. However, the unique nature of single-view photometric stereo allows one to model misaligned image sequences using the underlying image formation model and a set of translational shifts. This paper introduces such a technique, entitled translational photometric alignment, that employs the Lambertian model of image formation. This reduces the alignment problem to minimizing a nonlinear sum-squared error function in order to best reconcile the observed images with the generative model. Thus, the end goal of translational photometric alignment is not only to align image sequences, but also to produce the best surface-normal estimates given the observed images. Controlled experiments on the Yale Face Database B demonstrate the high accuracy of translational photometric alignment. The utility and benefits of the technique are further illustrated by additional experiments on image sequences suffering from uncontrolled real-world misalignments.     

Fusing multiview and photometric stereo for 3D reconstruction under uncalibrated illumination

We propose a method to obtain a complete and accurate 3D model from multiview images captured under a variety of unknown illuminations. Based on recent results showing that for Lambertian objects, general illumination can be approximated well using low-order spherical harmonics, we develop a robust alternating approach to recover surface normals. Surface normals are initialized using a multi-illumination multiview stereo algorithm, then refined using a robust alternating optimization method based on the l(1) metric. Erroneous normal estimates are detected using a shape prior. Finally, the computed normals are used to improve the preliminary 3D model. The reconstruction system achieves watertight and robust 3D reconstruction while neither requiring manual interactions nor imposing any constraints on the illumination. Experimental results on both real world and synthetic data show that the technique can acquire accurate 3D models for Lambertian surfaces, and even tolerates small violations of the Lambertian assumption.     

Error analysis of surface normals determined by radiometry

Surface normals can be computed from three images of a workpiece taken under three distinct lighting conditions without requiring surface continuity. Radiometric methods are susceptible to systematic errors such as: errors in the measurement of light source orientations; mismatched light source irradiance; detector nonlinearity; the presence of specular reflection or shadows; the spatial and spectral distribution of incident light; surface size, material, and microstructure; and the length and properties of the light source to target path. Typically, a 1° error in surface orientation of a Lambertian workpiece is caused by a 1 percent change in image intensity due to variations in incident light intensity or a 1° change in orientation of a collimated light source. Tests on a white nylon sphere indicate that by using modest error prevention and calibration schemes, surface angles off the camera axis can be computed within 5°, except at edge pixels. Equations for the sensitivity of surface normals to major error sources have been derived. Results of surface normal estimation and edge extraction experiments on various non-Lambertian and textured workpieces are also presented.     

Dynamic adjustment of stereo display parameters

This paper reports on an experimental approach to adjusting stereo parameters automatically and thereby providing a low eye strain, easily accommodated stereo view for computer graphics applications. To this end, the concept of virtual eye separation is defined. Experiment 1 shows that dynamic changes in virtual eye separation are not noticed if they occur over a period of a few seconds. Experiment 2 shows that when subjects are given control over their virtual eye separation, they change it depending on the amount of depth in the scene. Based partly on these results, an algorithm is presented for enhancing stereo depth cues for moving computer generated 3D images. It has the effect of doubling the stereo depth in flat scenes and limiting the stereo depth for deep scenes. It also reduces the occurrence of double images and the discrepancy between focus and vergence. The algorithm is applied dynamically in real time with an optional damping factor applied so the disparities never change too abruptly. Finally, Experiment 3 provides a qualitative assessment of the algorithm with a dynamic “flight” over a digital elevation map