Shape Reconstruction of 3D Bilaterally Symmetric Surfaces

The paper presents a new approach for shape recovery based on integrating geometric and photometric information. We consider 3D bilaterally symmetric objects, that is, objects which are symmetric with respect to a plane (e.g., faces), and their reconstruction from a single image. Both the viewpoint and the illumination are not necessarily frontal. Furthermore, no correspondence between symmetric points is required.The basic idea is that an image taken from a general, non frontal viewpoint, under non-frontal illumination can be regarded as a pair of images. Each image of the pair is one half of the object, taken from different viewing positions and with different lighting directions. Thus, one-image-variants of geometric stereo and of photometric stereo can be used. Unlike the separate invocation of these approaches, which require point correspondence between the two images, we show that integrating the photometric and geometric information suffice to yield a dense correspondence between pairs of symmetric points, and as a result, a dense shape recovery of the object. Furthermore, the unknown lighting and viewing parameters, are also recovered in this process.Unknown distant point light source, Lambertian surfaces, unknown constant albedo, and weak perspective projection are assumed. The method has been implemented and tested experimentally on simulated and real data.     

Three-dimensional shape from color photometric stereo

Computer vision systems can be used to determine the shapes of real three-dimensional objects for purposes of object recognition and pose estimation or for CAD applications. One method that has been developed is photometric stereo. This method uses several images taken from the same viewpoint, but with different lightings, to determine the three-dimensional shape of an object. Most previous work in photometric stereo has been with gray-tone images; color images have only been used for dielectric materials. In this paper we describe a procedure for color photometric stereo, which recovers the shape of a colored object from two or more color images of the object under white illumination. This method can handle different types of materials, such as composites and metals, and can employ various reflection models such as the Lambertian, dichromatic, and Torrance-Sparrow models. For composite materials, colored metals, and dielectrics, there are two advantages of utilizing color information: at each pixel, there are more constraints on the orientation, and the result is less sensitive to noise. Consequently, the shape can be found more accurately. The method has been tested on both artificial and real images of objects of various materials, and on real images of a multi-colored object.     

Photometric Stereo with General,Unknown Lighting

Work on photometric stereo has shown how to recover the shape and reflectance properties of an object using multiple images taken with a fixed viewpoint and variable lighting conditions. This work has primarily relied on known lighting conditions or the presence of a single point source of light in each image. In this paper we show how to perform photometric stereo assuming that all lights in a scene are distant from the object but otherwise unconstrained. Lighting in each image may be an unknown and may include arbitrary combination of diffuse, point and extended sources. Our work is based on recent results showing that for Lambertian objects, general lighting conditions can be represented using low order spherical harmonics. Using this representation we can recover shape by performing a simple optimization in a low-dimensional space. We also analyze the shape ambiguities that arise in such a representation. We demonstrate our method by reconstructing the shape of objects from images obtained under a variety of lightings. We further compare the reconstructed shapes against shapes obtained with a laser scanner.     

3D Surface Reconstruction Using Occluding Contours

This paper addresses the problem of 3D surface reconstruction using image sequences. It has been shown that shape recovery from three or more occluding contours of the surface is possible given a known camera motion. Several algorithms, which have been recently proposed, allow such a reconstruction under the assumption of a linear camera motion. A new approach is presented which deals with the reconstruction problem directly from a discrete point of view. First, a theoretical study of the epipolar correspondence between occluding contours is achieved. A correct depth formulation is then derived from a local approximation of the surface up to order two. This allows the local shape to be estimated, given three consecutive contours, without any constraints on the camera motion. Experimental results are presented for both synthetic and real data.     

Robust Shape Recovery from Occluding Contours Using a Linear Smoother

Recovering the shape of an object from two views fails at occluding contours of smooth objects because the extremal contours are view dependent. For three or more views, shape recovery is possible, and several algorithms have recently been developed for this purpose. We present a new approach to the multiframe stereo problem that does not depend on differential measurements in the image, which may be noise sensitive. Instead, we use a linear smoother to optimally combine all of the measurements available at the contours (and other edges) in all of the images. This allows us to extract a robust and reasonably dense estimate of surface shape, and to integrate shape information from both surface markings and occluding contours. Results are presented, which empirically show that in the presence of noise, smoothing over more than three views reduces the error even when the epipolar curve is nonplanar.     

An algebraic approach to shape-from-image problems

This paper presents a new method for recovering three-dimensional shapes of polyhedral objects from their single-view images. The problem of recovery is formulated in a constrained optimization problem, in which the constraints reflect the assumption that the scene is composed of polyhedral objects, and the objective function to be minimized is a weighted sum of quadric errors of surface information such as shading and texture. For practical purpose it is decomposed into the two more tractable problems: a linear programming problem and an unconstrained optimization problem. In the present method the global constraints placed by the polyhedron assumption are represented in terms of linear algebra, whereas similar constraints have usually been represented in terms of a gradient space. Moreover, superstrictness of the constraints can be circumvented by a new concept ‘position-free incidence structure’. For this reason the present method has several advantages: it can recover the polyhedral shape even if image data are incorrect due to vertex-position errors, it can deal with perspective projection as well as orthographic projection, the number of variables in the optimization problem is very small (three or a little greater than three), and any kinds of surface information can be incorporated in a unifying manner.     

Efficient generic face model fitting to images and videos

In this paper we present a robust and lightweight method for the automatic fitting of deformable 3D face models on facial images. Popular fitting techniques such as those based on statistical models of shape and appearance require a training stage based on a set of facial images and their corresponding facial landmarks, which have to be manually labeled. Therefore, new images in which to fit the model cannot differ too much in shape and appearance (including illumination variation, facial hair, wrinkles, etc.) from those used for training. By contrast, our approach can fit a generic face model in two steps: (1) the detection of facial features based on local image gradient analysis and (2) the backprojection of a deformable 3D face model through the optimization of its deformation parameters. The proposed approach can retain the advantages of both learning-free and learning-based approaches. Thus, we can estimate the position, orientation, shape and actions of faces, and initialize user-specific face tracking approaches, such as Online Appearance Models (OAMs), which have shown to be more robust than generic user tracking approaches. Experimental results show that our method outperforms other fitting alternatives under challenging illumination conditions and with a computational cost that allows its implementation in devices with low hardware specifications, such as smartphones and tablets. Our proposed approach lends itself nicely to many frameworks addressing semantic inference in face images and videos.     

利用光度立体法生成真实物体的三维模型

在计算机三维动画制作中，要制作一个具体的真实物体的三维模型，是一项复杂而困难的事情。本文根据由阴影恢复形状的视觉理论，利用光度立体视觉技术，给出了由３幅光移图象计算物体表面方向和深度信息，并利用贝塞尔曲线拟合获得物体表面的结构模型的方法。     

A 3D Imaging Framework Based on High-Resolution Photometric-Stereo and Low-Resolution Depth

This paper introduces a 3D imaging framework that combines high-resolution photometric stereo and low-resolution depth. Our approach targets imaging scenarios based on either macro-lens photography combined with focal stacking or a large-format camera that are able to image objects with more than 600 samples per mm $^2$ . These imaging techniques allow photometric stereo algorithms to obtain surface normals at resolutions that far surpass corresponding depth values obtained with traditional approaches such as structured-light, passive stereo, or depth-from-focus. Our work offers two contributions for 3D imaging based on these scenarios. The first is a multi-resolution, patched-based surface reconstruction scheme that can robustly handle the significant resolution difference between our surface normals and depth samples. The second is a method to improve the initial normal estimation by using all the available focal information for images obtained using a focal stacking technique.     

3-D reconstruction using mirror images based on a plane symmetryrecovering method

Three-dimensional reconstruction from a perspective 2D image using mirrors is addressed. The mirrors are used to form symmetrical relations between the direct image and mirror images. By finding correspondences between them, the 3D shape can be reconstructed by means of plane symmetry recovering method using the vanishing point. Two constraints are used in determining the correspondence. In the case where only one mirror is used, invisible parts both in the direct image and in the mirror image may still remain. Using multiple mirrors, however, occluded parts will decrease or disappear, and occlusion-free object reconstruction becomes possible     

基于形变模型由立体序列图象恢复物体的3D形状

结合立体视觉和形变模型提出了一种新的物体3D形状的恢复方法。采用立体视觉方法导出物体表面的3D坐标;利用光流模型估计物体的3D运动,根据此运动移动形变模型,使其对准物体的表面块;由形变模型将由各幅图象得到的离散的3D点融为一起,得到物体的表面形状。实验结果表明该方法能用于形状复杂的物体恢复。     

Modeling Surperspective Projection of Landscapes for Geographical Guide-Map Generation

It is still challenging to generate hand-drawn pictures because they differ from ordinary photographs in that they are often drawn as seen from multiple viewpoints. This paper presents a new approach for modeling such surperspective projection based on shape deformation techniques. Specifically, surperspective landscape images for guide-maps are generated from 3D geographical elevation data. Our method first partitions a target geographical surface into feature areas to provide designers with landmarks suitable for editing. The system takes as input 2D visual effects, which are converted to 3D geometric constraints for geographical surface deformation. Using ordinary perspective projection, the deformed shape is then transformed into a target guide-map image where each landmark enjoys its own vista points. An algorithm for calculating such 2D visual effects semi-automatically from the geographical shape features is also considered.     

双摄象机大转角立体图象的深度计算

本文就具有普遍性的“大转角双目视觉”的三维恢复问题,提出了一种包括自动匹配问题在内的求取物体三维坐标的系列最小二乘法.这种方法不仅能够克服由于大转角变化及遮挡等原因造成的求取图象对应点的困难,而且在数据准确性、鲁棒性、计算复杂性、保证旋转矩阵正交性及三维运动的直接恢复程度方面都优于现存的方法;用实际图象数据进行计算,结果满足实际要求.     

基于二维自动主动形状模型的椎间盘核磁共振图像分割算法

针对椎间盘手动建模主观耗时以及现有分割方法不够准确的问题,提出了一种二维自动主动形状模型(2D-AASM)方法,由基于最小描述长度的椎间盘自动统计形状建模、二维局部梯度建模和分割三部分组成。将25组脊柱核磁共振图像(MRI)的椎间盘专家分割结果作为训练集,采用基于最小描述长度的方法确定点对应关系,建立椎间盘T4-5的统计形状模型和二维局部梯度模型,生成形状模型的方差和目标函数值均小于手工和弧长参数方法。模型建立后,通过3组脊柱MRI数据测试提出的分割方法,与传统主动形状模型（ASM）和加入一维局部梯度模型的ASM方法相比,其分割结果具有更高的戴斯系数值,更低的过分割率和欠分割率。实验结果表明,所提方法建立的模型更准确,分割结果更精确。     

Correspondence with cumulative similarity transforms

A local image transform based on cumulative similarity measures is defined and is shown to enable efficient correspondence and tracking near occluding boundaries. Unlike traditional methods, this transform allows correspondences to be found when the only contrast present is the occluding boundary itself and when the sign of contrast along the boundary is possibly reversed. The transform is based on the idea of a cumulative similarity measure which characterizes the shape of local image homogeneity; both the value of an image at a particular point and the shape of the region with locally similar and connected values is captured. This representation is insensitive to structure beyond an occluding boundary but is sensitive to the shape of the boundary itself, which is often an important cue. We show results comparing this method to traditional least-squares and robust correspondence matching     

Affine Structure and Motion from Points,Lines and Conics

In this paper several new methods for estimating scene structure and camera motion from an image sequence taken by affine cameras are presented. All methods can incorporate both point, line and conic features in a unified manner. The correspondence between features in different images is assumed to be known.Three new tensor representations are introduced describing the viewing geometry for two and three cameras. The centred affine epipoles can be used to constrain the location of corresponding points and conics in two images. The third order, or alternatively, the reduced third order centred affine tensors can be used to constrain the locations of corresponding points, lines and conics in three images. The reduced third order tensors contain only 12 components compared to the 16 components obtained when reducing the trifocal tensor to affine cameras.A new factorization method is presented. The novelty lies in the ability to handle not only point features, but also line and conic features concurrently. Another complementary method based on the so-called closure constraints is also presented. The advantage of this method is the ability to handle missing data in a simple and uniform manner. Finally, experiments performed on both simulated and real data are given, including a comparison with other methods.     

Combinatorial photometric stereo and its application in 3D modeling of melanoma

This article concerns a new type of photometric stereo algorithm for which outliers such as highlights and shadows, including attached and cast shadow, are mixed with Lambertian data. The underlying motivation behind this algorithm is very simple: an axial symmetrical setup in 6-light photometric stereo can be used to offer advantages. We investigate why an axial symmetrical setup is useful and how it can be used to improve standard photometric stereo. The main result is summarized as a combinatorial photometric stereo algorithm which embeds a non-Lambertian detection procedure. To apply this algorithm, it involves three steps. First, it combines a group of reflectance intensities to make five virtual images, whose equivalence is guaranteed due to the axial symmetrical setup of the 6-source photometric stereo system. Second, comparison between these virtual images generates a five by five skew-symmetric matrix. The Frobenius norm of this matrix is then employed as an index to determine whether there is a non-Lambertian pixel present among the six pixels. Finally, after identification of non-Lambertian pixels, standard photometric stereo is performed to realize 3D modeling. Validation of this algorithm has been conducted with both synthetic and real images. The real images were obtained from a newly designed 3D imaging device, the Skin Analyzer, for clinical inspection of melanoma. Experimental study shows that combinatorial photometric stereo gives promising results in suppressing shadows and highlights, while improving 3D reconstruction results. Furthermore, error analysis illustrates how to determine an appropriate threshold value to enable the algorithm to achieve optimal performance.     

Projective reconstruction and invariants from multiple images

This correspondence investigates projective reconstruction of geometric configurations seen in two or more perspective views, and the computation of projective invariants of these configurations from their images. A basic tool in this investigation is the fundamental matrix that describes the epipolar correspondence between image pairs. It is proven that once the epipolar geometry is known, the configurations of many geometric structures (for instance sets of points or lines) are determined up to a collineation of projective 3-space 𝒫³ by their projection in two independent images. This theorem is the key to a method for the computation of invariants of the geometry. Invariants of six points in 𝒫³ and of four lines in 𝒫³ are defined and discussed. An example with real images shows that they are effective in distinguishing different geometrical configurations. Since the fundamental matrix is a basic tool in the computation of these invariants, new methods of computing the fundamental matrix from seven-point correspondences in two images or six-point correspondences in three images are given     

Real-time face tracking and recognition by sparse eigentracker with associative mapping to 3D shape

This paper proposes a novel framework of real-time face tracking and recognition by combining two eigen-based methods. The first method is a novel extension of eigenface called augmented eigenface and the second method is a sparse 3D eigentemplate tracker controlled by a particle filter. The augmented eigenface is an eigenface augmented by an associative mapping to 3D shape that is specified by a set of volumetric face models. This paper discusses how to make up the augmented eigenface and how it can be used for inference of 3D shape from partial images. The associative mapping is also generalized to subspace-to-one mappings to cover photometric image changes for a fixed shape. A novel technique, called photometric adjustment, is introduced for simple implementation of associative mapping when an image subspace should be combined to a shape. The sparse 3D eigentemplate tracker is an extension of the 3D template tracker proposed by Oka et al. In combination with the augmented eigenface, the sparse 3D eigentemplate tracker facilitates real-time 3D tracking and recognition when a monocular image sequence is provided. In the tracking, sparse 3D eigentemplate is updated by the augmented eigenface while face pose is estimated by the sparse eigentracker. Since the augmented eigenface is constructed on the conventional eigenfaces, face identification and expression recognition are also accomplished efficiently during the tracking. In the experiment, an augmented eigenface was constructed from 25 faces where 24 images were taken in different lighting conditions for each face. Experimental results show that the augmented eigenface works with the 3D eigentemplate tracker for real-time tracking and recognition.