The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows

We present an algorithm for separating the local gradient information and Lambertian color by using 4-source color photometric stereo in the presence of highlights and shadows. We assume that the surface reflectance can be approximated by the sum of a Lambertian and a specular component. The conventional photometric method is generalized for color images. Shadows and highlights in the input images are detected using either spectral or directional cues and excluded from the recovery process, thus giving more reliable estimates of local surface parameters.     

Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.     

Robust 3D face capture using example-based photometric stereo

We show that using example-based photometric stereo, it is possible to achieve realistic reconstructions of the human face. The method can handle non-Lambertian reflectance and attached shadows after a simple calibration step. We use spherical harmonics to model and de-noise the illumination functions from images of a reference object with known shape, and a fast grid technique to invert those functions and recover the surface normal for each point of the target object. The depth coordinate is obtained by weighted multi-scale integration of these normals, using an integration weight mask obtained automatically from the images themselves. We have applied these techniques to improve the PhotoFace system of Hansen et al. (2010).     

Three-dimensional shape from color photometric stereo

Computer vision systems can be used to determine the shapes of real three-dimensional objects for purposes of object recognition and pose estimation or for CAD applications. One method that has been developed is photometric stereo. This method uses several images taken from the same viewpoint, but with different lightings, to determine the three-dimensional shape of an object. Most previous work in photometric stereo has been with gray-tone images; color images have only been used for dielectric materials. In this paper we describe a procedure for color photometric stereo, which recovers the shape of a colored object from two or more color images of the object under white illumination. This method can handle different types of materials, such as composites and metals, and can employ various reflection models such as the Lambertian, dichromatic, and Torrance-Sparrow models. For composite materials, colored metals, and dielectrics, there are two advantages of utilizing color information: at each pixel, there are more constraints on the orientation, and the result is less sensitive to noise. Consequently, the shape can be found more accurately. The method has been tested on both artificial and real images of objects of various materials, and on real images of a multi-colored object.     

An approximation technique for photometric stereo

Photometric stereo is a technique to determine the orientation of an object. It takes at least three images corresponding to three different light sources to determine the orientation and to eliminate the ambiguities which will arise if only two light sources are used. In this report we show that a plausible shape can be obtained based only on two light sources. The method is based on the approximation of a surface patch by a spherical patch. Two real images and one synthetic image are used to illustrate the method.     

Combinatorial photometric stereo and its application in 3D modeling of melanoma

This article concerns a new type of photometric stereo algorithm for which outliers such as highlights and shadows, including attached and cast shadow, are mixed with Lambertian data. The underlying motivation behind this algorithm is very simple: an axial symmetrical setup in 6-light photometric stereo can be used to offer advantages. We investigate why an axial symmetrical setup is useful and how it can be used to improve standard photometric stereo. The main result is summarized as a combinatorial photometric stereo algorithm which embeds a non-Lambertian detection procedure. To apply this algorithm, it involves three steps. First, it combines a group of reflectance intensities to make five virtual images, whose equivalence is guaranteed due to the axial symmetrical setup of the 6-source photometric stereo system. Second, comparison between these virtual images generates a five by five skew-symmetric matrix. The Frobenius norm of this matrix is then employed as an index to determine whether there is a non-Lambertian pixel present among the six pixels. Finally, after identification of non-Lambertian pixels, standard photometric stereo is performed to realize 3D modeling. Validation of this algorithm has been conducted with both synthetic and real images. The real images were obtained from a newly designed 3D imaging device, the Skin Analyzer, for clinical inspection of melanoma. Experimental study shows that combinatorial photometric stereo gives promising results in suppressing shadows and highlights, while improving 3D reconstruction results. Furthermore, error analysis illustrates how to determine an appropriate threshold value to enable the algorithm to achieve optimal performance.     

Integrability disambiguates surface recovery in two-image photometric stereo

Two images of a Lambertian surface obtained under different illumination conditions, determine the local surface normals up to two possible orientations. We show that for smooth surfaces, the local integrability constraints usually resolve the problem of deciding between the two possibilities. We also provide a complete characterization of the surfaces that remain ambiguous under given illumination conditions.     

Learning conditional photometric stereo with high-resolution features

Computational Visual Media - Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination....     

Auto-calibrating photometric stereo using ring light constraints

We propose an auto-calibration method for photometric stereo. Our method exploits constraints placed on light sources to recover their positions and the surface normals of an object up to a scaling and planar rotation ambiguity. The ambiguity is resolved with multi-view consistency constraints leading to a Euclidean reconstruction of the geometric shape of the object. Auto-calibration methods are helpful to overcome the limitations imposed by calibrating objects. We evaluate our algorithm with experiments on real world scenes.     

Fusing multiview and photometric stereo for 3D reconstruction under uncalibrated illumination

We propose a method to obtain a complete and accurate 3D model from multiview images captured under a variety of unknown illuminations. Based on recent results showing that for Lambertian objects, general illumination can be approximated well using low-order spherical harmonics, we develop a robust alternating approach to recover surface normals. Surface normals are initialized using a multi-illumination multiview stereo algorithm, then refined using a robust alternating optimization method based on the l(1) metric. Erroneous normal estimates are detected using a shape prior. Finally, the computed normals are used to improve the preliminary 3D model. The reconstruction system achieves watertight and robust 3D reconstruction while neither requiring manual interactions nor imposing any constraints on the illumination. Experimental results on both real world and synthetic data show that the technique can acquire accurate 3D models for Lambertian surfaces, and even tolerates small violations of the Lambertian assumption.     

光度立体技术的物体三维表面重建算法模拟与评价

为了提高基于光度立体法的物体三维表面重建精度,研究了几种常用三维表面重建算法的适用范围.介绍了几种常用重建算法(变分法、金字塔法、代数法)的基本原理及求解方法,指出了表面方向梯度的求解准确度是三维表面重建过程中的难点.在此基础上建立了朗伯半球体模型和实际拍摄的玻璃灯罩,从重建速度、重建精度等方面对几种算法进行了比较和评价.最后通过实验模拟和算法性能评价,表明了常用算法的重建精确度有限,并且时三维重建今后的研究进行了展望.     

3D face reconstructions from photometric stereo using near infrared and visible light

This paper seeks to advance the state-of-the-art in 3D face capture and processing via novel Photometric Stereo (PS) hardware and algorithms. The first contribution is a new high-speed 3D data capture system, which is capable of acquiring four raw images in approximately 20 ms. The results presented in this paper demonstrate the feasibility of deploying the device in commercial settings. We show how the device can operate with either visible light or near infrared (NIR) light. The NIR light sources offer the advantages of being less intrusive and more covert than most existing face recognition methods allow. Furthermore, our experiments show that the accuracy of the reconstructions is also better using NIR light. The paper also presents a modified four-source PS algorithm which enhances the surface normal estimates by assigning a likelihood measure for each pixel being in a shadowed region. This likelihood measure is determined by the discrepancies between measured pixel brightnesses and expected values. Where the likelihood of shadow is high, then one light source is omitted from the computation for that pixel, otherwise a weighted combination of pixels is used to determine the surface normal. This means that the precise shadow boundary is not required by our method. The results section of the paper provides a detailed analysis of the methods presented and a comparison to ground truth. We also analyse the reflectance properties of a small number of skin samples to test the validity of the Lambertian model and point towards potential improvements to our method using the Oren–Nayar model.     

Surface reflectance and normal estimation from photometric stereo

In this paper, we propose a new photometric stereo method for estimating diffuse reflection and surface normal from color images. Using dichromatic reflection model, we introduce surface chromaticity as a matching invariant for photometric stereo, which serves as the foundation of the theory of this paper. An extremely simple and robust reflection components separation method is proposed based on the invariant. Our separation method differs from most previous methods which either assume dependencies among pixels or require segmentation. We also show that a linear relationship between the image color and the surface normal can be obtained based on this invariant. The linear relationship turns the surface normal estimation problem into a linear system that can be solved exactly or via least-squares optimization. We present experiments on both synthetic and real images, which demonstrate the effectiveness of our method.     

Dense photometric stereo: a Markov random field approach

We address the problem of robust normal reconstruction by dense photometric stereo, in the presence of complex geometry, shadows, highlight, transparencies, variable attenuation in light intensities, and inaccurate estimation in light directions. The input is a dense set of noisy photometric images, conveniently captured by using a very simple set-up consisting of a digital video camera, a reflective mirror sphere, and a handheld spotlight. We formulate the dense photometric stereo problem as a Markov network and investigate two important inference algorithms for Markov Random Fields (MRFs)--graph cuts and belief propagation--to optimize for the most likely setting for each node in the network. In the graph cut algorithm, the MRF formulation is translated into one of energy minimization. A discontinuity-preserving metric is introduced as the compatibility function, which allows alpha-expansion to efficiently perform the maximum a posteriori (MAP) estimation. Using the identical dense input and the same MRF formulation, our tensor belief propagation algorithm recovers faithful normal directions, preserves underlying discontinuities, improves the normal estimation from one of discrete to continuous, and drastically reduces the storage requirement and running time. Both algorithms produce comparable and very faithful normals for complex scenes. Although the discontinuity-preserving metric in graph cuts permits efficient inference of optimal discrete labels with a theoretical guarantee, our estimation algorithm using tensor belief propagation converges to comparable results, but runs faster because very compact messages are passed and combined. We present very encouraging results on normal reconstruction. A simple algorithm is proposed to reconstruct a surface from a normal map recovered by our method. With the reconstructed surface, an inverse process, known as relighting in computer graphics, is proposed to synthesize novel images of the given scene under user-specified light source and direction. The synthesis is made to run in real time by exploiting the state-of-the-art graphics processing unit (GPU). Our method offers many unique advantages over previous relighting methods and can handle a wide range of novel light sources and directions.     

Bayesian shape estimation: Shape-from-shading and photometric stereo revisited

We present a Bayesian approach to the machine vision processes of shape-from-shading and photometric stereo, also considering the associated question of the detection of shape discontinuities. The shape reconstruction problem is formulated as a maximum a posteriori (MAP) estimation from probability distributions of Gibbs form, and is solved via simulated annealing. In shape-from-shading, our formulation leads to a constrained optimization problem, where the constraints come from the image irradiance equation and from the incorporation of the necessary boundary conditions. In photometric stereo, we are able to estimate shape directly from degraded input images. We also propose an edge-detection algorithm that works cooperatively with the reconstruction process, employing the shape estimates to locate the discontinuities of the reconstructed surface. We show results of the application of our framework both to synthetic and to real imagery.     

基于光度特性和多梯度分析的运动阴影去除法

提出了一种利用光度特性和多梯度分析去除运动阴影的算法。该算法是在准确地检测出运动目标的基础上,首先利用像素邻域内光度特性和连接成分分析法检测出候选阴影区域;其次比较分析候选阴影区域和背景中相应区域的梯度,并结合连接成分分析法去除阴影;最后使用边缘去除法进一步去除半影,并应用形态学运算完善结果。实验结果表明,同传统的阴影去除法相比,该算法有高检测率和高识别率。     

Light source position calibration method for photometric stereo in capsule endoscopy

ABSTRACT

For photometric stereo in capsule endoscopy, calibration of light source is crucial for improving the precision of surface normal estimation. Therefore, this paper presents an improved planar-mirror-based light source position calibration method: from captured images of light source and detected poses of planar mirror, light paths are retraced from camera to light source, and position of light source is triangulated with least square method. The contribution of this paper is that a refraction model of the planar mirror is employed in the retracement of light paths, thus the bias of light paths caused by refraction can be compensated and the position of light source can be estimated more precisely. The results of simulation and experiment show that the proposed method provides higher calibration accuracy than the current planar mirror-based calibration method and can improve the precision of subsequent photometric stereo-based 3D reconstruction.