Restoring warped document images through 3D shape modeling

Scanning a document page from a thick bound volume often results in two kinds of distortions in the scanned image, i.e., shade along the "spine" of the book and warping in the shade area. In this paper, we propose an efficient restoration method based on the discovery of the 3D shape of a book surface from the shading information in a scanned document image. From a technical point of view, this shape from shading (SFS) problem in real-world environments is characterized by 1) a proximal and moving light source, 2) Lambertian reflection, 3) nonuniform albedo distribution, and 4) document skew. Taking all these factors into account, we first build practical models (consisting of a 3D geometric model and a 3D optical model) for the practical scanning conditions to reconstruct the 3D shape of the book surface. We next restore the scanned document image using this shape based on deshading and dewarping models. Finally, we evaluate the restoration results by comparing our estimated surface shape with the real shape as well as the OCR performance on original and restored document images. The results show that the geometric and photometric distortions are mostly removed and the OCR results are improved markedly.     

Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions

In this paper, we present a new method to modify the appearance of a face image by manipulating the illumination condition, when the face geometry and albedo information is unknown. This problem is particularly difficult when there is only a single image of the subject available. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using a spherical harmonic representation. Moreover, morphable models are statistical ensembles of facial properties such as shape and texture. In this paper, we integrate spherical harmonics into the morphable model framework by proposing a 3D spherical harmonic basis morphable model (SHBMM). The proposed method can represent a face under arbitrary unknown lighting and pose simply by three low-dimensional vectors, i.e., shape parameters, spherical harmonic basis parameters, and illumination coefficients, which are called the SHBMM parameters. However, when the image was taken under an extreme lighting condition, the approximation error can be large, thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion-based framework that uses a Markov random field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to extreme lighting conditions, but also insensitive to partial occlusions. The performance of our framework is demonstrated through various experimental results, including the improved rates for face recognition under extreme lighting conditions.     

Super-vector coding features extracted from both depth buffer and view-normal-angle images for part-based 3D shape retrieval

We have witnessed 3D shape models abundant in many application fields including 3D CAD/CAM, augmented/mixed reality (AR/MR), and entertainment. Creating 3D shape models from scratch is still very expensive. Efficient and accurate methods for shape retrieval is essential for 3D shape models to be reused. To retrieve similar 3D shape models, one must provide an arbitrary 3D shape as a query. Most of the research on 3D shape retrieval has been conducted with a “whole” shape as a query (aka whole-to-whole shape retrieval), while a “part” shape (aka part-to-whole shape retrieval) is more practically requested as a query especially by mechanical engineering with 3D CAD/CAM applications. A “part” shape is naturally constructed by a 3D range scanner as an input device. In this paper, we focus on the efficient method for part-to-whole shape retrieval where the “part” shape is assumed to be given by a 3D range scanner. Specifically, we propose a Super-Vector coding feature with SURF local features extracted from the View-Normal-Angle image, or the image synthesized by taking account of the angle between the view vector and the surface normal vector, together with the depth-buffered image, for part-to-whole shape retrieval. In addition, we propose a weighted whole-to-whole re-ranking method taking advantage of global information based on the result of part-to-whole shape retrieval. Through experiments we demonstrate that our proposed method outperforms the previous methods with or without re-ranking.     

基于多视角的高精度三维人脸重建

提出了一种多阶段优化的方法来解决基于多视角图片在未知姿态、表情以及光照条件下的高精度三维人脸重建问题.首先,通过重新渲染合成的方法将参数化模型拟合到输入的多视角图片,然后在纹理域上求解一个光流问题来获取不同视角之间的对应关系.通过对应关系可以恢复出人脸的点云,并利用基于明暗恢复几何的方法来恢复人脸细节.在真实数据以及合成数据下的实验结果表明,文中方法能够恢复出带有几何细节的高精度的三维人脸模型,并且提高了现有方法的重建精度.     

Driving 3D morphable models using shading cues

In this paper we show how surface orientation information inferred using shape-from-shading can be used to aid the process of fitting a 3D morphable model to an image of a face. We consider the problem of model dominance and show how shading constraints can be used to refine morphable model shape estimates, offering the possibility of exceeding the maximum possible accuracy of the model. We use this observation to motivate an optimisation scheme based on surface normal error. This ensures the fullest possible use of the information conveyed by the shading in an image. Moreover, our framework allows estimation of per-vertex albedo and bump maps which are not constrained to lie within the span of the model. This means the recovered model is capable of describing shape and reflectance phenomena not present in the training set. We show reconstruction and synthesis results and demonstrate that the shape and albedo estimates can be used for illumination insensitive recognition using only a single gallery image.     

Direct Differential Photometric Stereo Shape Recovery of Diffuse and Specular Surfaces

Recovering the 3D shape of an object from shading is a challenging problem due to the complexity of modeling light propagation and surface reflections. Photometric Stereo (PS) is broadly considered a suitable approach for high-resolution shape recovery, but its functionality is restricted to a limited set of object surfaces and controlled lighting setup. In particular, PS models generally consider reflection from objects as purely diffuse, with specularities being regarded as a nuisance that breaks down shape reconstruction. This is a serious drawback for implementing PS approaches, since most common materials have prominent specular components. In this paper, we propose a PS model that solves the problem for both diffuse and specular components aimed at shape recovery of generic objects with the approach being independent of the albedo values thanks to the image ratio formulation used. Notably, we show that by including specularities, it is possible to solve the PS problem for a minimal number of three images using a setup with three calibrated lights and a standard industrial camera. Even if an initial separation of diffuse and specular components is still required for each input image, experimental results on synthetic and real objects demonstrate the feasibility of our approach for shape reconstruction of complex geometries.     

Reconstructing 3D Shape,Albedo and Illumination from a Single Face Image

The morphable model has been employed to efficiently describe 3D face shape and the associated albedo with a reduced set of basis vectors. The spherical harmonics (SH) model provides a compact basis to well approximate the image appearance of a Lambertian object under different illumination conditions. Recently, the SH and morphable models have been integrated for 3D face shape reconstruction. However, the reconstructed 3D shape is either inconsistent with the SH bases or obtained just from landmarks only. In this work, we propose a geometrically consistent algorithm to reconstruct the 3D face shape and the associated albedo from a single face image iteratively by combining the morphable model and the SH model. The reconstructed 3D face geometry can uniquely determine the SH bases, therefore the optimal 3D face model can be obtained by minimizing the error between the input face image and a linear combination of the associated SH bases. In this way, we are able to preserve the consistency between the 3D geometry and the SH model, thus refining the 3D shape reconstruction recursively. Furthermore, we present a novel approach to recover the illumination condition from the estimated weighting vector for the SH bases in a constrained optimization formulation independent of the 3D geometry. Experimental results show the effectiveness and accuracy of the proposed face reconstruction and illumination estimation algorithm under different face poses and multiple‐light‐source illumination conditions.     

Near-lighting Photometric Stereo for unknown scene distance and medium attenuation

Photometric Stereo in murky water is subject to light attenuation and near-field illumination, and the resulting image formation model is complex. Apart from the scene normals and albedo, the incident illumination varies per-pixel and it depends on the scene depth and the attenuation coefficient of the medium. When these are unknown, e.g. in a realistic scenario where a robotic platform explores an underwater scene (unknown shape and distance) within the dynamic subsea environment (unknown scattering level), Photometric Stereo becomes ambiguous. Previous approaches have tackled the problem by assuming distant-lighting and resorting to external hardware for estimating the unknown model variables. In our work, we show that the Photometric Stereo problem can be determined as soon as some additional constraints regarding the scene albedo and the presence of pixels with local intensity maxima within the image are incorporated into the optimization framework. Our proposed solution leads to effective Photometric Stereo and yields detailed 3D reconstruction of objects in murky water when the scene distance and the medium attenuation are unknown. We evaluate our work using both numerical simulations and real experiments in the controlled environment of a water tank and real port water using a remotely operated vehicle.     

Non-Rigid Stereo Factorization

In this paper we address the problem of recovering 3D non-rigid structure from a sequence of images taken with a stereo pair. We have extended existing non-rigid factorization algorithms to the stereo camera case and presented an algorithm to decompose the measurement matrix into the motion of the left and right cameras and the 3D shape, represented as a linear combination of basis-shapes. The added constraints in the stereo camera case are that both cameras are viewing the same structure and that the relative orientation between both cameras is fixed. Our focus in this paper is on the recovery of flexible 3D shape rather than on the correspondence problem. We propose a method to compute reliable 3D models of deformable structure from stereo images. Our experiments with real data show that improved reconstructions can be achieved using this method. The algorithm includes a non-linear optimization step that minimizes image reprojection error and imposes the correct structure to the motion matrix by choosing an appropriate parameterization. We show that 3D shape and motion estimates can be successfully disambiguated after bundle adjustment and demonstrate this on synthetic and real image sequences. While this optimization step is proposed for the stereo camera case, it can be readily applied to the case of non-rigid structure recovery using a monocular video sequence. Electronic supplementary material Electronic supplementary material is available for this article at and accessible for authorised users.     

A shape-from-shading method of polyhedral objects using prior information

We propose a new method for recovering the 3D shape of a polyhedral object from its single 2D image using the shading information contained in the image and the prior information on the object. In a strict sense, we cannot recover the shape of a polyhedron from an incorrect line drawing, even if it is practically almost correct. In order to overcome this problem, we propose a flexible face positioning method that can permit inconsistencies in the recovered shape that arise from vertex-position errors contained in incorrect line drawings. Also, we propose to use prior information about the horizontality and verticality of special faces and the convex and concave properties of the edges in order to attain good solutions and present a method of formulating such prior information as physical constraints. The shape-from-shading method is formulated as a minimization problem of a nonlinear cost function with the nonlinear constraints and its solution is searched by a global optimization algorithm. In the experiments with a synthetic image and three kinds of real images, shapes that are similar to those of the actual objects were recovered in all cases. As a result, the proposed method has proven to be effective in the shape recovery of simple-shape polyhedral objects.     

超球流形约束下的3DMM分步全局优化

通过综合运用人脸空间的超球流形约束、基于梯度的启发式全局优化、光照的球面谐波描述以及凸包可见点集的直接消隐方法,提出一种三维可形变模型的图像匹配方法.首先通过形状超球流形约束下的全局优化算法求解摄像机参数和形状参数,然后使用以上参数和凸包点集的直接消隐方法确定物像点对应关系,最后根据物像点对应关系由反射率超球流形约束下的全局优化算法求解光照参数和反射率参数.定量的对比实验结果表明,该方法无需借助分区域拟合、人为估计参数值、层次匹配策略或复杂的特征组合,即可由单幅图像恢复三维可形变模型(3DMM)的全部参数.     

Performance evaluation of memetic approaches in 3D reconstruction of forensic objects

Different tasks in forensics require the use of 3D models of forensic objects (skulls, bones, corpses, etc.) captured by 3D range scanners. Since a whole object cannot be completely scanned in a single image using a range scanner, multiple acquisitions from different views are needed to supply the information to construct the 3D model by a range image registration method. There is an increasing interest in adopting evolutionary algorithms as the optimization technique for image registration methods. However, the image registration community tends to separate global and local searches in two different stages, named sequential hybridization approach, which is opposite to the scheme adopted by the memetic framework. In this work, we aim to analyze the capabilities of memetic algorithms (Moscato in On evolution, search, optimization, genetic algorithms and martial arts: towards memeticalgorithms. Report 826, Caltech Concurrent Computation Program, Pasadena, 1989) for tackling a really complex and challenging real-world problem as the 3D reconstruction of forensic objects. Our intention is threefold: firstly, designing new memetic-based methods for tackling a real-world problem and subsequently carrying out a performance and behavioral analysis of the results; secondly, comparing their performance with the one achieved by other methods based on the classical sequential hybridization approach; and thirdly, concluding the experimental study by highlighting the outcomes achieved by the best method in tackling the real-world problem. Several real-world 3D reconstruction problems from the Physical Anthropology Lab at the University of Granada, Spain, were used to support the evaluation study.     

Real‐time recognition and tracking for augmented reality books

An augmented reality book (AR book) is an application in which such multimedia elements as virtual 3D objects, movie clips, or sound clips are augmented to a conventional book using augmented reality technology. It can provide better understanding about the contents and visual impressions for users. For AR books, this paper presents a markerless tracking method, which recognizes and tracks a large number of pages in real‐time, even on PCs with low computation power. For fast recognition with respect to a large number of pages, we propose a generic randomized forest that is an extension of a randomized forest. In addition, we define the spatial locality of the subregions in an image to resolve the problem of a dropping recognition rate under a complex background. For tracking with minimal jittering, we also propose the adaptive keyframe‐based tracking method, which automatically updates the current frame as a keyframe when it describes the page better than the existing one. Copyright © 2011 John Wiley & Sons, Ltd.     

三维重建中曲面反照率值的估计与应用

从单幅图像获得物体的表面高度是计算机视觉中的一个重要研究领域,其中一种重要的方法就是从明暗恢复形状(ShapefromShading,简称SFS)。在SFS的各种不同算法中都需要曲面的反照率值,反照率值的估算是否准确直接影响了三维重建的效果。针对反照率值的估算,已经产生了很多有效的算法。文中讨论了三种反照率值的估计算法及其优缺点,并将局部反照率估计算法引入到三维重建中,解决了由全局反照率值重建的弊端。     

Estimating Facial Reflectance Properties Using Shape-from-Shading

In this paper we show how to estimate facial surface reflectance properties (a slice of the BRDF and the albedo) in conjunction with the facial shape from a single image. The key idea underpinning our approach is to iteratively interleave the two processes of estimating reflectance properties based on the current shape estimate and updating the shape estimate based on the current estimate of the reflectance function. For frontally illuminated faces, the reflectance properties can be described by a function of one variable which we estimate by fitting a curve to the scattered and noisy reflectance samples provided by the input image and estimated shape. For non-frontal illumination, we fit a smooth surface to the scattered 2D reflectance samples. We make use of a novel statistical face shape constraint which we term ‘model-based integrability’ which we use to regularise the shape estimation. We show that the method is capable of recovering accurate shape and reflectance information from single grayscale or colour images using both synthetic and real world imagery. We use the estimated reflectance measurements to render synthetic images of the face in varying poses. To synthesise images under novel illumination, we show how to fit a parametric model of reflectance to the estimated reflectance function.     

Shape Reconstruction of 3D Bilaterally Symmetric Surfaces

The paper presents a new approach for shape recovery based on integrating geometric and photometric information. We consider 3D bilaterally symmetric objects, that is, objects which are symmetric with respect to a plane (e.g., faces), and their reconstruction from a single image. Both the viewpoint and the illumination are not necessarily frontal. Furthermore, no correspondence between symmetric points is required.The basic idea is that an image taken from a general, non frontal viewpoint, under non-frontal illumination can be regarded as a pair of images. Each image of the pair is one half of the object, taken from different viewing positions and with different lighting directions. Thus, one-image-variants of geometric stereo and of photometric stereo can be used. Unlike the separate invocation of these approaches, which require point correspondence between the two images, we show that integrating the photometric and geometric information suffice to yield a dense correspondence between pairs of symmetric points, and as a result, a dense shape recovery of the object. Furthermore, the unknown lighting and viewing parameters, are also recovered in this process.Unknown distant point light source, Lambertian surfaces, unknown constant albedo, and weak perspective projection are assumed. The method has been implemented and tested experimentally on simulated and real data.     

Mumford-Shah on the Move: Region-Based Segmentation on Deforming Manifolds with Application to 3-D Reconstruction of Shape and Appearance from Multi-View Images

We address the problem of estimating the shape and appearance of a scene made of smooth Lambertian surfaces with piecewise smooth albedo. We allow the scene to have self-occlusions and multiple connected components. This class of surfaces is often used as an approximation of scenes populated by man-made objects. We assume we are given a number of images taken from different vantage points. Mathematically this problem can be posed as an extension of Mumford and Shah’s approach to static image segmentation to the segmentation of a function defined on a deforming surface. We propose an iterative procedure to minimize a global cost functional that combines geometric priors on both the shape of the scene and the boundary between smooth albedo regions. We carry out the numerical implementation in the level set framework.     

A low cost 3D scanner based on structured light

Automatic 3D acquisition devices (often called 3D scanners) allow to build highly accurate models of real 3D objects in a cost- and time-effective manner. We have experimented this technology in a particular application context: the acquisition of Cultural Heritage artefacts. Specific needs of this domain are: medium-high accuracy, easy of use, affordable cost of the scanning device, self-registered acquisition of shape and color data, and finally operational safety for both the operator and the scanned artefacts. According to these requirements, we designed a low-cost 3D scanner based on structured light which adopts a new, versatile colored stripe pattern approach. We present the scanner architecture, the software technologies adopted, and the first results of its use in a project regarding the 3D acquisition of an archeological statue.     

激光三维扫描仪的硬件设计与实现

非接触三维测量技术是研究物体形态，特别是研究软组织物体表面形态的基础，如何在非接触条件下精确地采集三维数据，这一直是人们所期待要解决的问题，激光三维扫描仪较好地解决了这一问题，可方便、快速、精确地采集各类物体表面的三维数据。利用这组三维数据，计算机可再现出物体任一角度的三维图像，从而为研究物体表面形态打下良好的基础。     

基于网格变形的从图像重建三维人脸

从图像重建高质量三维人脸一直是计算机视觉和图形学的一个重要研究问题.不同于传统的基于立体匹配的窄基线多视几何和数据驱动的人脸形变方法,提出一种结合网格变形技术和立体视觉原理的、从图像重建高质量三维人脸模型方法.给定从不同视角拍摄的几幅人脸图像,基于健壮图像特征获得可靠的相机外部参数和稀疏三维点;在此基础上,提出一种结合几何细节保持和图像一致性约束的三维人脸变形算法重建三维人脸,通过对人脸模板的网格变形,使得变形人脸在多幅图像中的可见投影具有一致性的图像颜色强度.基于模板的人脸变形可以有效地解决三维模型成像中的遮挡问题,采用健壮估计法消除噪声、离群点和光照对目标函数收敛性的影响,对目标函数的多次非线性优化求解进一步改进了人脸重建的质量.采用合成人脸图像和真实人脸图像重建三维人脸的实验结果表明,文中算法可以从几幅宽基线图像重建高质量的三维人脸模型.