Separation of Reflection Components Using Color and Polarization

Specular reflections and interreflections produce strong highlights in brightness images. These highlights can cause vision algorithms for segmentation, shape from shading, binocular stereo, and motion estimation to produce erroneous results. A technique is developed for separating the specular and diffuse components of reflection from images. The approach is to use color and polarization information, simultaneously, to obtain constraints on the reflection components at each image point. Polarization yields local and independent estimates of the color of specular reflection. The result is a linear subspace in color space in which the local diffuse component must lie. This subspace constraint is applied to neighboring image points to determine the diffuse component. In contrast to previous separation algorithms, the proposed method can handle highlights on surfaces with substantial texture, smoothly varying diffuse reflectance, and varying material properties. The separation algorithm is applied to several complex scenes with textured objects and strong interreflections. The separation results are then used to solve three problems pertinent to visual perception; determining illumination color, estimating illumination direction, and shape recovery.     

Dense Reconstruction of Transparent Objects by Altering Incident Light Paths Through Refraction

This paper addresses the problem of reconstructing the surface shape of transparent objects. The difficulty of this problem originates from the viewpoint dependent appearance of a transparent object, which quickly makes reconstruction methods tailored for diffuse surfaces fail disgracefully. In this paper, we introduce a fixed viewpoint approach to dense surface reconstruction of transparent objects based on refraction of light. We present a simple setup that allows us to alter the incident light paths before light rays enter the object by immersing the object partially in a liquid, and develop a method for recovering the object surface through reconstructing and triangulating such incident light paths. Our proposed approach does not need to model the complex interactions of light as it travels through the object, neither does it assume any parametric form for the object shape nor the exact number of refractions and reflections taken place along the light paths. It can therefore handle transparent objects with a relatively complex shape and structure, with unknown and inhomogeneous refractive index. We also show that for thin transparent objects, our proposed acquisition setup can be further simplified by adopting a single refraction approximation. Experimental results on both synthetic and real data demonstrate the feasibility and accuracy of our proposed approach.     

纹理物体表面反射参数的恢复

物体表面反射参数的恢复是新兴的逆向绘制技术的核心问题，提出一种恢复物体表面BRDF模型参数的方法，假定物体表面的BRDF模型可以近似分解为漫射纹理和整体不变的镜面反射分量，则每一点的反射参数可以用Phong模型来近似计算，首先选取物体表面的一片区域，利用该区域在不同视点光照条件下采样的几幅图像，计算得到该区域的平均镜面反射分量，作为整个物体表面的镜面反射分量，然后据此从物体表面各点的采样中分离出漫射分量，计算各点的漫反射参数，得到近似的BRDF模型，实验数据表明算法是有效的。     

Transparent and Specular Object Reconstruction

This state of the art report covers reconstruction methods for transparent and specular objects or phenomena. While the 3D acquisition of opaque surfaces with Lambertian reflectance is a well‐studied problem, transparent, refractive, specular and potentially dynamic scenes pose challenging problems for acquisition systems. This report reviews and categorizes the literature in this field. Despite tremendous interest in object digitization, the acquisition of digital models of transparent or specular objects is far from being a solved problem. On the other hand, real‐world data is in high demand for applications such as object modelling, preservation of historic artefacts and as input to data‐driven modelling techniques. With this report we aim at providing a reference for and an introduction to the field of transparent and specular object reconstruction. We describe acquisition approaches for different classes of objects. Transparent objects/phenomena that do not change the straight ray geometry can be found foremost in natural phenomena. Refraction effects are usually small and can be considered negligible for these objects. Phenomena as diverse as fire, smoke, and interstellar nebulae can be modelled using a straight ray model of image formation. Refractive and specular surfaces on the other hand change the straight rays into usually piecewise linear ray paths, adding additional complexity to the reconstruction problem. Translucent objects exhibit significant sub‐surface scattering effects rendering traditional acquisition approaches unstable. Different classes of techniques have been developed to deal with these problems and good reconstruction results can be achieved with current state‐of‐the‐art techniques. However, the approaches are still specialized and targeted at very specific object classes. We classify the existing literature and hope to provide an entry point to this exiting field.     

A probabilistic framework for specular shape-from-shading

One of the problems that hinders conventional methods for shape-from-shading is the presence of local specularities which may be misidentified as high curvature surface features. In this paper we address the problem of estimating the proportions of Lambertian and specular reflection components in order to improve the quality of surface normal information recoverable using shape-from-shading. The framework for our study is provided by the iterated conditional modes algorithm. We develop a maximum a posteriori probability (MAP) estimation method for estimating the mixing proportions for Lambertian and specular reflectance, and also, for recovering local surface normals. The MAP estimation scheme has two model ingredients. First, there are separate conditional measurement densities which describe the distributions of surface normal directions for the Lambertian and specular reflectance components. We experimentally compare three different models for the specular component. The second ingredient is a smoothness prior which models the distribution of surface normal directions over local image regions. We demonstrate the utility of method on real-world data. Ground truth data is provided by imagery obtained with crossed polaroid filters. This reveals not only that the method accurately estimates the proportion of specular reflection, but that it also results in good surface normal reconstruction in the proximity of specular highlights.     

A Framework for Automatically Recovering Object Shape,Reflectance and Light Sources from Calibrated Images

In this paper, we present a complete framework for recovering an object shape, estimating its reflectance properties and light sources from a set of images. The whole process is performed automatically. We use the shape from silhouette approach proposed by R. Szeliski (1993) combined with image pixels for reconstructing a triangular mesh according to the marching cubes algorithm. A classification process identifies regions of the object having the same appearance. For each region, a single point or directional light source is detected. Therefore, we use specular lobes, lambertian regions of the surface or specular highlights seen on images. An identification method jointly (i) decides what light sources are actually significant and (ii) estimates diffuse and specular coefficients for a surface represented by the modified Phong model (Lewis, 1994). In order to validate our algorithm efficiency, we present a case study with various objects, light sources and surface properties. As shown in the results, our system proves accurate even for real objects images obtained with an inexpensive acquisition system.     

Neural computation approach for developing a 3D shapereconstruction model

The shape from shading problem refers to the well-known fact that most real images usually contain specular components and are affected by unknown reflectivity. In this paper, these limitations are addressed and a new neural-based 3D shape reconstruction model is proposed. The idea behind this approach is to optimize a proper reflectance model by learning the parameters of the proposed neural reflectance model. In order to do this, new neural-based reflectance models are presented. The feedforward neural network (FNN) model is able to generalize the diffuse term, while the RBF model is able to generalize the specular term. A hybrid structure of FNN-based and RBF-based models is also presented because most real surfaces are usually neither Lambertian models nor ideally specular models. Experimental results, including synthetic and real images, are presented to demonstrate the performance of our approach given different specular effects, unknown illuminate conditions, and different noise environments.     

存在镜面反射时的立体匹配研究

传统的立体匹配方法建立在Lambertian的漫反射模型之上,漫反射模型的立体匹配在一个图像中大部分是有效的,但是在处理图像中包含镜面反射部分时结果会产生严重的匹配错误.为了解决个问题,根据二色反射模型引入一种漫反射和镜面反射的分离方法,匹配图像中存在镜面反射部分时先滤除掉镜面反射再进行匹配,在镜面反射部分也能匹配得到正确的视差.实验结果证明该方法很有效.     

Shape from interaction

We present “shape from interaction” (SfI), an approach to the problem of acquiring 3D representations of rigid objects through observing the activity of a human who handles a tool. SfI relies on the fact that two rigid objects cannot share the same physical space. The 3D reconstruction of the unknown object is achieved by tracking the known 3D tool and by carving out the space it occupies as a function of time. Due to this indirection, SfI reconstructs rigid objects regardless of their material and appearance properties and proves particularly useful for the cases of textureless, transparent, translucent, refractive and specular objects for which there exists no practical vision-based 3D reconstruction method. Additionally, object concavities that are not directly observable can also be reconstructed. The 3D tracking of the tool is formulated as an optimization problem that is solved based on visual input acquired by a multicamera system. Experimental results from a prototype implementation of SfI support qualitatively and quantitatively the effectiveness of the proposed approach.     

Shape and View Independent Reflectance Map from Multiple Views

We consider the problem of estimating the 3D shape and reflectance properties of an object made of a single material from a set of calibrated views. To model the reflectance, we propose to use the View Independent Reflectance Map (VIRM), which is a representation of the joint effect of the diffuse+specular Bidirectional Reflectance Distribution Function (BRDF) and the environment illumination. The object shape is parameterized using a triangular mesh. We pose the estimation problem as minimizing the cost of matching input images, and the images synthesized using the shape and VIRM estimates. We show that by enforcing a constant value of VIRM as a global constraint, we can minimize the cost function by iterating between the VIRM and shape estimation. Experimental results on both synthetic and real objects show that our algorithm can recover both the 3D shape and the diffuse/specular reflectance information. Our algorithm does not require the light sources to be known or calibrated. The estimated VIRM can be used to predict the appearances of objects with the same material from novel viewpoints and under transformed illumination. The support of National Science Foundation under grant ECS 02-25523 is gratefully acknowledged. Tianli Yu was supported in part by a Beckman Institute Graduate Fellowship.     

Detection and localization of specular surfaces using image motion cues

Successful identification of specularities in an image can be crucial for an artificial vision system when extracting the semantic content of an image or while interacting with the environment. We developed an algorithm that relies on scale and rotation invariant feature extraction techniques and uses motion cues to detect and localize specular surfaces. Appearance change in feature vectors is used to quantify the appearance distortion on specular surfaces, which has previously been shown to be a powerful indicator for specularity (Doerschner et al. in Curr Biol, 2011). The algorithm combines epipolar deviations (Swaminathan et al. in Lect Notes Comput Sci 2350:508–523, 2002) and appearance distortion, and succeeds in localizing specular objects in computer-rendered and real scenes, across a wide range of camera motions and speeds, object sizes and shapes, and performs well under image noise and blur conditions.     

Separation of diffuse and specular components of surface reflection by use of polarization and statistical analysis of images

The image of an opaque object is created by observing the reflection of the light incident on its surface. The dichromatic reflection model describes the surface reflection as the sum of two components, diffuse and specular terms. The specular reflection component is usually strong in its intensity and polarized significantly compared to the diffuse components. On the other hand, the intensity of the diffuse component is weak and it tends to be unpolarized except near occluding contours. Thus, the observation of an object through a rotating polarizer approximately yields images containing constant diffuse component and specular component of different intensity. In this paper, we show that diffuse and specular components of surface reflection can be separated as two independent components when we apply independent component analysis to the images observed through a polarizer of different orientations. We give a separation simulation of artificial data and also give some separation results of real scenes.     

Near‐Instant Capture of High‐Resolution Facial Geometry and Reflectance

We present a near‐instant method for acquiring facial geometry and reflectance using a set of commodity DSLR cameras and flashes. Our setup consists of twenty‐four cameras and six flashes which are fired in rapid succession with subsets of the cameras. Each camera records only a single photograph and the total capture time is less than the 67ms blink reflex. The cameras and flashes are specially arranged to produce an even distribution of specular highlights on the face. We employ this set of acquired images to estimate diffuse color, specular intensity, specular exponent, and surface orientation at each point on the face. We further refine the facial base geometry obtained from multi‐view stereo using estimated diffuse and specular photometric information. This allows final submillimeter surface mesostructure detail to be obtained via shape‐from‐specularity. The final system uses commodity components and produces models suitable for authoring high‐quality digital human characters.     

Estimation of surface orientation from directional derivatives of shading image

A new estimation approach is proposed to reconstruct the surface shape from shading images. The reflectance property of the object surface material is the uniform lambertian property; the light source produces a uniform parallel beam, and image projection is orthographic. The illuminating direction is varied slightly to obtain directional derivatives of image density. The new approach shows that the local surface orientation is reconstructed from the directional derivatives of the image density. Differently from other approaches, all components of the illuminating direction vector or the proportional constant are not necessary for this reconstruction. Computer simulations show the evaluation of errors in approximating and quantizing process. The effectiveness of this approach is demonstrated in the shape reconstruction simulations for the curved objects     

Shape From Shading by Using Neural Based Colour Reflectance Model

In this Letter, a new methodology for Colour Shape From Shading problem is proposed. The problem of colour SFS refers to the well-known fact that most real objects usually contain mixtures of diffuse and specular colour reflections. In this paper, these limitations are addressed and a new colour neural based model is proposed. The proposed approach focuses on developing a generalized neural based colour reflectance model. Experimental results on synthetic coloured objects and a real coloured object were performed to demonstrate the performance of the proposed methodology.     

Visual Servoing Based on Structure From Controlled Motion or on Robust Statistics

This paper focuses on the way to achieve accurate visual servoing tasks when the shape of the object being observed as well as the desired image are unknown. More precisely, we want to control the camera orientation with respect to the tangent plane at a certain object point corresponding to the center of a region of interest. We also want to observe this point at the principal point to fulfil a fixation task. A 3-D reconstruction phase must, therefore, be performed during the camera motion. Our approach is then close to the structure-from-motion problem. The reconstruction phase is based on the measurement of the 2-D motion in a region of interest and on the measurement of the camera velocity. Since the 2-D motion depends on the shape of the objects being observed, we introduce a unified motion model to cope both with planar and nonplanar objects. However, since this model is only an approximation, we propose two approaches to enlarge its domain of validity. The first is based on active vision, coupled with a 3-D reconstruction based on a continuous approach, and the second is based on statistical techniques of robust estimation, coupled with a 3-D reconstruction based on a discrete approach. Theoretical and experimental results compare both approaches.     

Acquiring a complete 3D model from specular motion under theillumination of circular-shaped light sources

We recover 3D models of objects with specular surfaces. An object is rotated and its continuous images are taken. Circular-shaped light sources that generate conic rays are used to illuminate the rotating object in such a way that highlighted stripes can be observed on most of the specular surfaces. Surface shapes can be computed from the motions of highlights in the continuous images; either specular motion stereo or single specular trace mode can be used. When the lights are properly set, each point on the object can be highlighted during the rotation. The shape for each rotation plane is measured independently using its corresponding epipolar plane image. A 3D shape model is subsequently reconstructed by combining shapes at different rotation planes. Computing a shape is simple and requires only the motion of highlight on each rotation plane. The novelty of this paper is the complete modeling of a general type of specular objects that has not been accomplished before     

Separating reflection components based on chromaticity and noise analysis

Many algorithms in computer vision assume diffuse only reflections and deem specular reflections to be outliers. However, in the real world, the presence of specular reflections is inevitable since there are many dielectric inhomogeneous objects which have both diffuse and specular reflections. To resolve this problem, we present a method to separate the two reflection components. The method is principally based on the distribution of specular and diffuse points in a two-dimensional maximum chromaticity-intensity space. We found that, by utilizing the space and known illumination color, the problem of reflection component separation can be simplified into the problem of identifying diffuse maximum chromaticity. To be able to identity the diffuse maximum chromaticity correctly, an analysis of the noise is required since most real images suffer from it. Unlike existing methods, the proposed method can separate the reflection components robustly for any kind of surface roughness and light direction.     

Rapid classification of specular and diffuse reflection from image velocities

We propose a method for rapidly classifying surface reflectance directly from the output of spatio-temporal filters applied to an image sequence of rotating objects. Using image data from only a single frame, we compute histograms of image velocities and classify these as being generated by a specular or a diffusely reflecting object. Exploiting characteristics of material-specific image velocities we show that our classification approach can predict the reflectance of novel 3D objects, as well as human perception.