Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.     

An approximation technique for photometric stereo

Photometric stereo is a technique to determine the orientation of an object. It takes at least three images corresponding to three different light sources to determine the orientation and to eliminate the ambiguities which will arise if only two light sources are used. In this report we show that a plausible shape can be obtained based only on two light sources. The method is based on the approximation of a surface patch by a spherical patch. Two real images and one synthetic image are used to illustrate the method.     

Three-dimensional shape from color photometric stereo

Computer vision systems can be used to determine the shapes of real three-dimensional objects for purposes of object recognition and pose estimation or for CAD applications. One method that has been developed is photometric stereo. This method uses several images taken from the same viewpoint, but with different lightings, to determine the three-dimensional shape of an object. Most previous work in photometric stereo has been with gray-tone images; color images have only been used for dielectric materials. In this paper we describe a procedure for color photometric stereo, which recovers the shape of a colored object from two or more color images of the object under white illumination. This method can handle different types of materials, such as composites and metals, and can employ various reflection models such as the Lambertian, dichromatic, and Torrance-Sparrow models. For composite materials, colored metals, and dielectrics, there are two advantages of utilizing color information: at each pixel, there are more constraints on the orientation, and the result is less sensitive to noise. Consequently, the shape can be found more accurately. The method has been tested on both artificial and real images of objects of various materials, and on real images of a multi-colored object.     

Auto-calibrating photometric stereo using ring light constraints

We propose an auto-calibration method for photometric stereo. Our method exploits constraints placed on light sources to recover their positions and the surface normals of an object up to a scaling and planar rotation ambiguity. The ambiguity is resolved with multi-view consistency constraints leading to a Euclidean reconstruction of the geometric shape of the object. Auto-calibration methods are helpful to overcome the limitations imposed by calibrating objects. We evaluate our algorithm with experiments on real world scenes.     

Learning conditional photometric stereo with high-resolution features

Computational Visual Media - Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination....     

Overcoming shadows in 3-source photometric stereo

Light occlusions are one of the most significant difficulties of photometric stereo methods. When three or more images are available without occlusion, the local surface orientation is overdetermined so that shape can be computed and the shadowed pixels can be discarded. In this paper, we look at the challenging case when only two images are available without occlusion, leading to a one degree of freedom ambiguity per pixel in the local orientation. We show that, in the presence of noise, integrability alone cannot resolve this ambiguity and reconstruct the geometry in the shadowed regions. As the problem is ill-posed in the presence of noise, we describe two regularization schemes that improve the numerical performance of the algorithm while preserving the data. Finally, the paper describes how this theory applies in the framework of color photometric stereo where one is restricted to only three images and light occlusions are common. Experiments on synthetic and real image sequences are presented.     

Integrability disambiguates surface recovery in two-image photometric stereo

Two images of a Lambertian surface obtained under different illumination conditions, determine the local surface normals up to two possible orientations. We show that for smooth surfaces, the local integrability constraints usually resolve the problem of deciding between the two possibilities. We also provide a complete characterization of the surfaces that remain ambiguous under given illumination conditions.     

Surface reflectance and normal estimation from photometric stereo

In this paper, we propose a new photometric stereo method for estimating diffuse reflection and surface normal from color images. Using dichromatic reflection model, we introduce surface chromaticity as a matching invariant for photometric stereo, which serves as the foundation of the theory of this paper. An extremely simple and robust reflection components separation method is proposed based on the invariant. Our separation method differs from most previous methods which either assume dependencies among pixels or require segmentation. We also show that a linear relationship between the image color and the surface normal can be obtained based on this invariant. The linear relationship turns the surface normal estimation problem into a linear system that can be solved exactly or via least-squares optimization. We present experiments on both synthetic and real images, which demonstrate the effectiveness of our method.     

Robust 3D face capture using example-based photometric stereo

We show that using example-based photometric stereo, it is possible to achieve realistic reconstructions of the human face. The method can handle non-Lambertian reflectance and attached shadows after a simple calibration step. We use spherical harmonics to model and de-noise the illumination functions from images of a reference object with known shape, and a fast grid technique to invert those functions and recover the surface normal for each point of the target object. The depth coordinate is obtained by weighted multi-scale integration of these normals, using an integration weight mask obtained automatically from the images themselves. We have applied these techniques to improve the PhotoFace system of Hansen et al. (2010).     

Bayesian shape estimation: Shape-from-shading and photometric stereo revisited

We present a Bayesian approach to the machine vision processes of shape-from-shading and photometric stereo, also considering the associated question of the detection of shape discontinuities. The shape reconstruction problem is formulated as a maximum a posteriori (MAP) estimation from probability distributions of Gibbs form, and is solved via simulated annealing. In shape-from-shading, our formulation leads to a constrained optimization problem, where the constraints come from the image irradiance equation and from the incorporation of the necessary boundary conditions. In photometric stereo, we are able to estimate shape directly from degraded input images. We also propose an edge-detection algorithm that works cooperatively with the reconstruction process, employing the shape estimates to locate the discontinuities of the reconstructed surface. We show results of the application of our framework both to synthetic and to real imagery.     

Dense photometric stereo: a Markov random field approach

We address the problem of robust normal reconstruction by dense photometric stereo, in the presence of complex geometry, shadows, highlight, transparencies, variable attenuation in light intensities, and inaccurate estimation in light directions. The input is a dense set of noisy photometric images, conveniently captured by using a very simple set-up consisting of a digital video camera, a reflective mirror sphere, and a handheld spotlight. We formulate the dense photometric stereo problem as a Markov network and investigate two important inference algorithms for Markov Random Fields (MRFs)--graph cuts and belief propagation--to optimize for the most likely setting for each node in the network. In the graph cut algorithm, the MRF formulation is translated into one of energy minimization. A discontinuity-preserving metric is introduced as the compatibility function, which allows alpha-expansion to efficiently perform the maximum a posteriori (MAP) estimation. Using the identical dense input and the same MRF formulation, our tensor belief propagation algorithm recovers faithful normal directions, preserves underlying discontinuities, improves the normal estimation from one of discrete to continuous, and drastically reduces the storage requirement and running time. Both algorithms produce comparable and very faithful normals for complex scenes. Although the discontinuity-preserving metric in graph cuts permits efficient inference of optimal discrete labels with a theoretical guarantee, our estimation algorithm using tensor belief propagation converges to comparable results, but runs faster because very compact messages are passed and combined. We present very encouraging results on normal reconstruction. A simple algorithm is proposed to reconstruct a surface from a normal map recovered by our method. With the reconstructed surface, an inverse process, known as relighting in computer graphics, is proposed to synthesize novel images of the given scene under user-specified light source and direction. The synthesis is made to run in real time by exploiting the state-of-the-art graphics processing unit (GPU). Our method offers many unique advantages over previous relighting methods and can handle a wide range of novel light sources and directions.     

Evaluating the effect of diffuse light on photometric stereo reconstruction

Photometric stereo surface reconstruction requires each input image to be associated with a particular 3D illumination vector. This signifies that the subject should be illuminated in turn by various directional illumination sources. In real life, this directionality may be reduced by ambient illumination, which is typically present as a diffuse component of the incident light. This work assesses the photometric stereo reconstruction quality for various ratios of ambient to directional illuminance and provides a reference for the robustness of photometric stereo with respect to that illuminance ratio. In our analysis, we focus on the face reconstruction application of photometric stereo, as faces are convex objects with rich surface variation, thus providing a suitable platform for photometric stereo reconstruction quality evaluation. Results demonstrate that photometric stereo renders realistic reconstructions of the given surface for ambient illuminance as high as nine times the illuminance of the directional light component.     

Simultaneous estimation of shape and reflectance map from photometric stereo

This paper investigates whether the shape of an object and certain parameters of its reflectance map can be simultaneously estimated using photometric stereo. This problem has been addressed in the literature for the case where the Lambertian and non-Lambertian components in the image can be easily separated. No such separability is assumed in this paper. A class of reflectance maps for modeling diffusely reflecting surfaces is proposed. This class is based on the physics of scattering from real world surfaces. Next, the problem of joint estimation of some parameters of the map along with the surface shape is analyzed. A bound is obtained on the number of light sources necessary for a unique solution to the problem. The analysis also reveals that some of the estimates can be obtained by a nonparametric method. The behavior of the estimates in the presence of noise is also investigated. It is shown that simultaneous estimation is ill-posed. Regularizing the estimates yields good reconstructions from real world data.     

Uncalibrated flatfielding and illumination vector estimationfor photometric stereo face reconstruction

Within the context of photometric stereo reconstruction, flatfielding may be used to compensate for the effect of the inverse-square law of light propagation on the pixel brightness. This would require capturing a set of reference images at an off-line imaging session, which employs a calibrating device that should be captured under the exact conditions as the main session. Similarly, the illumination vectors, on which photometric stereo relies, are typically precomputed based on another dedicated calibration session. In practice, implementing such off-line sessions is inconvenient and often infeasible. This work aims at enabling accurate photometric stereo reconstruction for the case of non-interactive on-line capturing of human faces. We propose unsupervised methodologies, which extract all information that is required for accurate face reconstruction from the images of interest themselves. Specifically, we propose an uncalibrated flatfielding and an uncalibrated illumination vector estimation methodology, and we assess their effect on photometric stereo face reconstruction. Results demonstrate that incorporating our methodologies into the photometric stereo framework halves the reconstruction error, while eliminating the need of off-line calibration.     

An empirical study on the effects of translucency on photometric stereo

We present an empirical study on the effects of translucency on photometric stereo. Our study shows that the impact on the accuracy of the photometric normals is related to the relative size of the geometrical features and the mean free path. We show that under simplified conditions, the obtained photometric normals are a blurred version of the true surface normals, where the blur kernel is directly related to the subsurface scattering profile. We furthermore investigate the impact of scattering albedo, index of refraction, and single scattering on the accuracy. We perform our analysis using simulations, and demonstrate the validity on a real world example.     

Example-based photometric stereo: shape reconstruction with general, varying BRDFs

This paper presents a technique for computing the geometry of objects with general reflectance properties from images. For surfaces with varying material properties, a full segmentation into different material types is also computed. It is assumed that the camera viewpoint is fixed, but the illumination varies over the input sequence. It is also assumed that one or more example objects with similar materials and known geometry are imaged under the same illumination conditions. Unlike most previous work in shape reconstruction, this technique can handle objects with arbitrary and spatially-varying BRDFs. Furthermore, the approach works for arbitrary distant and unknown lighting environments. Finally, almost no calibration is needed, making the approach exceptionally simple to apply.     

Light source position calibration method for photometric stereo in capsule endoscopy

ABSTRACT

For photometric stereo in capsule endoscopy, calibration of light source is crucial for improving the precision of surface normal estimation. Therefore, this paper presents an improved planar-mirror-based light source position calibration method: from captured images of light source and detected poses of planar mirror, light paths are retraced from camera to light source, and position of light source is triangulated with least square method. The contribution of this paper is that a refraction model of the planar mirror is employed in the retracement of light paths, thus the bias of light paths caused by refraction can be compensated and the position of light source can be estimated more precisely. The results of simulation and experiment show that the proposed method provides higher calibration accuracy than the current planar mirror-based calibration method and can improve the precision of subsequent photometric stereo-based 3D reconstruction.     

利用控制点提高光度立体技术重建精度研究

基于光度立体技术的原理,使用物体表面方向梯度在局部范围内可以获得三维表面较高的精度,但由于重建过程中存在累积误差,在全局范围内存在偏移或扭曲.为此,提出了利用少量位置信息已知的控制点来校正误差以提高整体测量精度.由激光三角测量获取控制点的精确三维坐标,采用加权最小二乘法和改进的插补面法,得到了物体表面的三维形状.在合成球上的模拟结果表明,改进的插补面法可以更好地提高重建精度.