基于网格变形的从图像重建三维人脸

从图像重建高质量三维人脸一直是计算机视觉和图形学的一个重要研究问题.不同于传统的基于立体匹配的窄基线多视几何和数据驱动的人脸形变方法,提出一种结合网格变形技术和立体视觉原理的、从图像重建高质量三维人脸模型方法.给定从不同视角拍摄的几幅人脸图像,基于健壮图像特征获得可靠的相机外部参数和稀疏三维点;在此基础上,提出一种结合几何细节保持和图像一致性约束的三维人脸变形算法重建三维人脸,通过对人脸模板的网格变形,使得变形人脸在多幅图像中的可见投影具有一致性的图像颜色强度.基于模板的人脸变形可以有效地解决三维模型成像中的遮挡问题,采用健壮估计法消除噪声、离群点和光照对目标函数收敛性的影响,对目标函数的多次非线性优化求解进一步改进了人脸重建的质量.采用合成人脸图像和真实人脸图像重建三维人脸的实验结果表明,文中算法可以从几幅宽基线图像重建高质量的三维人脸模型.     

A Semi‐Lagrangian Closest Point Method for Deforming Surfaces

We present an Eulerian method for the real‐time simulation of intrinsic fluid dynamics effects on deforming surfaces. Our method is based on a novel semi‐Lagrangian closest point method for the solution of partial differential equations on animated triangle meshes. We describe this method and demonstrate its use to compute and visualize flow and wave propagation along such meshes at high resolution and speed. Underlying our technique is the efficient conversion of an animated triangle mesh into a time‐dependent implicit representation based on closest surface points. The proposed technique is unconditionally stable with respect to the surface deformation and, in contrast to comparable Lagrangian techniques, its precision does not depend on the level of detail of the surface triangulation.     

Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video

3D video [IEEE Multimedia (1997) 18] is the ultimate image media recording dynamic visual events in the real world as is; it records time varying 3D object shape with high fidelity surface properties (i.e., color and texture). Its applications cover wide varieties of personal and social human activities: entertainment (e.g., 3D game and 3D TV), education (e.g., 3D animal picture books), sports (e.g., sport performance analysis), medicine (e.g., 3D surgery monitoring), culture (e.g., 3D archive of traditional dances), and so on. In this paper, we propose: (1) a PC cluster system for real-time reconstruction of dynamic 3D object action from multi-view video images, (2) a deformable 3D mesh model for reconstructing the accurate dynamic 3D object shape, and (3) an algorithm of rendering natural-looking texture on the 3D object surface from the multi-view video images. Experimental results with quantitative performance evaluations demonstrate the effectiveness of these methods in generating high fidelity 3D video from multi-view video images.     

Reconstructing Animated Meshes from Time‐Varying Point Clouds

In this paper, we describe a novel approach for the reconstruction of animated meshes from a series of time‐deforming point clouds. Given a set of unordered point clouds that have been captured by a fast 3‐D scanner, our algorithm is able to compute coherent meshes which approximate the input data at arbitrary time instances. Our method is based on the computation of an implicit function in ?⁴ that approximates the time‐space surface of the time‐varying point cloud. We then use the four‐dimensional implicit function to reconstruct a polygonal model for the first time‐step. By sliding this template mesh along the time‐space surface in an as‐rigid‐as‐possible manner, we obtain reconstructions for further time‐steps which have the same connectivity as the previously extracted mesh while recovering rigid motion exactly. The resulting animated meshes allow accurate motion tracking of arbitrary points and are well suited for animation compression. We demonstrate the qualities of the proposed method by applying it to several data sets acquired by real‐time 3‐D scanners.     

Feature Preserving Mesh Generation from 3D Point Clouds

We address the problem of generating quality surface triangle meshes from 3D point clouds sampled on piecewise smooth surfaces. Using a feature detection process based on the covariance matrices of Voronoi cells, we first extract from the point cloud a set of sharp features. Our algorithm also runs on the input point cloud a reconstruction process, such as Poisson reconstruction, providing an implicit surface. A feature preserving variant of a Delaunay refinement process is then used to generate a mesh approximating the implicit surface and containing a faithful representation of the extracted sharp edges. Such a mesh provides an enhanced trade‐off between accuracy and mesh complexity. The whole process is robust to noise and made versatile through a small set of parameters which govern the mesh sizing, approximation error and shape of the elements. We demonstrate the effectiveness of our method on a variety of models including laser scanned datasets ranging from indoor to outdoor scenes.     

Direct Reconstruction of a Displaced Subdivision Surface from Unorganized Points

In this paper we describe the generation of a displaced subdivision surface directly from a set of unorganized points. The displaced subdivision surface is an efficient mesh representation that defines a detailed mesh with a displacement map over a smooth domain surface and has many benefits including compression, rendering, and animation, which overcome limitations of an irregular mesh produced by an ordinary mesh reconstruction scheme. Unlike previous displaced subdivision surface reconstruction methods, our method does not rely on a highly detailed reconstructed mesh. Instead, we efficiently create a coarse base mesh, which is used to sample displacements directly from unorganized points, and this results in a simple process and fast calculation. We suggest a shrink-wrapping-like shape approximation and a point-based mesh simplification method that uses the distance between a set of points and a mesh as an error metric to generate a domain surface that optimally approximates the given points. We avoid time-consuming energy minimization by employing a local subdivision surface fitting scheme. Finally, we show several reconstruction results that demonstrate the usability of our algorithm.     

A level-set approach for the metamorphosis of solid models

We present a new approach to 3D shape metamorphosis. We express the interpolation of two shapes as a process where one shape deforms to maximize its similarity with another shape. The process incrementally optimizes an objective function while deforming an implicit surface model. We represent the deformable surface as a level set (iso-surface) of a densely sampled scalar function of three dimensions. Such level-set models have been shown to mimic conventional parametric deformable surface models by encoding surface movements as changes in the grayscale values of a volume data set. Thus, a well-founded mathematical structure leads to a set of procedures that describes how voxel values can be manipulated to create deformations that are represented as a sequence of volumes. The result is a 3D morphing method that offers several advantages over previous methods, including minimal need for user input, no model parameterization, flexible topology, and subvoxel accuracy     

Free-viewpoint video relighting from multi-view sequence under general illumination

We proposed an approach to create plausible free-viewpoint relighting video using multi-view cameras array under general illumination. Given the multi-view video dataset recorded using a set of industrial cameras under general uncontrolled and unknown illumination, we first reconstruct 3D model of the captured target using existing multi-view stereo approach. Using the coarse geometry reconstruction, we estimate the spatially varying surface reflectance in the spherical harmonics domain considering the spatial and temporal coherence. With the estimated geometry and reflectance, the 3D target is relit to the novel illumination with the environment map of the target environment. Relit performance is enhanced using a flow- and quotient-based transfer strategy to achieve detailed and plausible performance relighting. Finally, the free-viewpoint video is generated using a view-dependent rendering strategy. Experimental results on various dataset show that our approach enables plausible free-view relighting, and opens up a path towards relightable free-viewpoint video using less complex acquisition setups.     

3D reconstruction for featureless scenes with curvature hints

We present a novel interactive framework for improving 3D reconstruction starting from incomplete or noisy results obtained through image-based reconstruction algorithms. The core idea is to enable the user to provide localized hints on the curvature of the surface, which are turned into constraints during an energy minimization reconstruction. To make this task simple, we propose two algorithms. The first is a multi-view segmentation algorithm that allows the user to propagate the foreground selection of one or more images both to all the images of the input set and to the 3D points, to accurately select the part of the scene to be reconstructed. The second is a fast GPU-based algorithm for the reconstruction of smooth surfaces from multiple views, which incorporates the hints provided by the user. We show that our framework can turn a poor-quality reconstruction produced with state of the art image-based reconstruction methods into a high- quality one.     

Multi-Level Partition of Unity Algebraic Point Set Surfaces

We present a multi-level partition of unity algebraic set surfaces (MPU-APSS) for surface reconstruction which can be represented by either a projection or in an implicit form. An algebraic point set surface (APSS) defines a smooth surface from a set of unorganized points using local moving least-squares (MLS) fitting of algebraic spheres. However, due to the local nature, APSS does not work well for geometry editing and modeling. Instead, our method builds an implicit approximation function for the scattered point set based on the partition of unity approach. By using an octree subdivision strategy, we first adaptively construct local algebraic spheres for the point set, and then apply weighting functions to blend together these local shape functions. Finally, we compute an error-controlled approximation of the signed distance function from the surface. In addition, we present an efficient projection operator which makes our representation suitable for point set filtering and dynamic point resampling. We demonstrate the effectiveness of our unified approach for both surface reconstruction and geometry modeling such as surface completion.     

Multi‐View Stereo on Consistent Face Topology

We present a multi‐view stereo reconstruction technique that directly produces a complete high‐fidelity head model with consistent facial mesh topology. While existing techniques decouple shape estimation and facial tracking, our framework jointly optimizes for stereo constraints and consistent mesh parameterization. Our method is therefore free from drift and fully parallelizable for dynamic facial performance capture. We produce highly detailed facial geometries with artist‐quality UV parameterization, including secondary elements such as eyeballs, mouth pockets, nostrils, and the back of the head. Our approach consists of deforming a common template model to match multi‐view input images of the subject, while satisfying cross‐view, cross‐subject, and cross‐pose consistencies using a combination of 2D landmark detection, optical flow, and surface and volumetric Laplacian regularization. Since the flow is never computed between frames, our method is trivially parallelized by processing each frame independently. Accurate rigid head pose is extracted using a PCA‐based dimension reduction and denoising scheme. We demonstrate high‐fidelity performance capture results with challenging head motion and complex facial expressions around eye and mouth regions. While the quality of our results is on par with the current state‐of‐the‐art, our approach can be fully parallelized, does not suffer from drift, and produces face models with production‐quality mesh topologies.     

Modeling Cumulus Cloud Scenes from High‐resolution Satellite Images

We present a reconstruction framework, which fits physically‐based constraints to model large‐scale cloud scenes from satellite images. Applications include weather phenomena visualization, flight simulation, and weather spotter training. In our method, the cloud shape is assumed to be composed of a cloud top surface and a nearly flat cloud base surface. Based on this, an effective method of multi‐spectral data processing is developed to obtain relevant information for calculating the cloud base height and the cloud top height, including ground temperature, cloud top temperature and cloud shadow. A lapse rate model is proposed to formulate cloud shape as an implicit function of temperature lapse rate and cloud base temperature. After obtaining initial cloud shapes, we enrich the shapes by a fractal method and represent reconstructed clouds by a particle system. Experiment results demonstrate the capability of our method in generating physically sound large‐scale cloud scenes from high‐resolution satellite images.     

Automatic segmentation of point clouds from multi-view reconstruction using graph-cut

In multi-view reconstruction systems, the recovered point cloud often consists of numerous unwanted background points. We propose a graph-cut based method for automatically segmenting point clouds from multi-view reconstruction. Based on the observation that the object of interest is likely to be central to the intended multi-view images, our method requires no user interaction except two roughly estimated parameters of objects covering in the central area of images. The proposed segmentation process is carried out in two steps: first, we build a weighted graph whose nodes represent points and edges that connect each point to its k-nearest neighbors. The potentials of each point being object and background are estimated according to distances between its projections in images and the corresponding image centers. The pairwise potentials between each point and its neighbors are computed according to their positions, colors and normals. Graph-cut optimization is then used to find the initial binary segmentation of object and background points. Second, to refine the initial segmentation, Gaussian mixture models (GMMs) are created from the color and density features of points in object and background classes, respectively. The potentials of each point being object and background are re-calculated based on the learned GMMs. The graph is updated and the segmentation of point clouds is improved by graph-cut optimization. The second step is iterated until convergence. Our method requires no manual labeling points and employs available information of point clouds from multi-view systems. We test the approach on real-world data generated by multi-view reconstruction systems.     

Optimizing human model reconstruction from RGB-D images based on skin detection

This paper reconstructs human model from multi-view RGB-D images of an Xbox One Kinect. We preprocess the depth images by implicit surface de-noising and then part-wisely register them into a point cloud. A template model is selected from the human model database to fit the registered point cloud of a human body by Laplacian deformation. Skin detection of RGB-D images helps to tightly constrain the skin parts of human body in template fitting step in order to get more precise and lifelike human model. We propose a robust skin detection method that is not affected by clothing pattern and background. Experiments demonstrate the effectiveness of our method.     

Dynamic-scale model construction from range imagery

The construction of a surface model from range data may be undertaken at any point in a continuum of scales that reflects the level of detail of the resulting model. This continuum relates the construction parameters to the scale of the model. We propose methods to dynamically reprocess range data at different scales. The construction result from a single scale is automatically evaluated, causing reconstruction at a different scale when user-defined criteria are not met. We demonstrate our methods in constructing a planar b-rep space envelope (a scene representation) for over 400 range images. The experiments demonstrate the ability to construct 100 percent valid models, with the scale of detail within specified requirements     

Model-based human shape reconstruction from multiple views

Image-based modelling allows the reconstruction of highly realistic digital models from real-world objects. This paper presents a model-based approach to recover animated models of people from multiple view video images. Two contributions are made, a multiple resolution model-based framework is introduced that combines multiple visual cues in reconstruction. Second, a novel mesh parameterisation is presented to preserve the vertex parameterisation in the model for animation. A prior humanoid surface model is first decomposed into multiple levels of detail and represented as a hierarchical deformable model for image fitting. A novel mesh parameterisation is presented that allows propagation of deformation in the model hierarchy and regularisation of surface deformation to preserve vertex parameterisation and animation structure. The hierarchical model is then used to fuse multiple shape cues from silhouette, stereo and sparse feature data in a coarse-to-fine strategy to recover a model that reproduces the appearance in the images. The framework is compared to physics-based deformable surface fitting at a single resolution, demonstrating an improved reconstruction accuracy against ground-truth data with a reduced model distortion. Results demonstrate realistic modelling of real people with accurate shape and appearance while preserving model structure for use in animation.     

On stochastic methods for surface reconstruction

In this article, we present and discuss three statistical methods for surface reconstruction. A typical input to a surface reconstruction technique consists of a large set of points that has been sampled from a smooth surface and contains uncertain data in the form of noise and outliers. We first present a method that filters out uncertain and redundant information yielding a more accurate and economical surface representation. Then we present two methods, each of which converts the input point data to a standard shape representation; the first produces an implicit representation while the second yields a triangle mesh.     

CPU Ray Tracing of Tree-Based Adaptive Mesh Refinement Data

Adaptive mesh refinement (AMR) techniques allow for representing a simulation's computation domain in an adaptive fashion. Although these techniques have found widespread adoption in high-performance computing simulations, visualizing their data output interactively and without cracks or artifacts remains challenging. In this paper, we present an efficient solution for direct volume rendering and hybrid implicit isosurface ray tracing of tree-based AMR (TB-AMR) data. We propose a novel reconstruction strategy, Generalized Trilinear Interpolation (GTI), to interpolate across AMR level boundaries without cracks or discontinuities in the surface normal. We employ a general sparse octree structure supporting a wide range of AMR data, and use it to accelerate volume rendering, hybrid implicit isosurface rendering and value queries. We demonstrate that our approach achieves artifact-free isosurface and volume rendering and provides higher quality output images compared to existing methods at interactive rendering rates.     

Feature‐Preserving Surface Reconstruction From Unoriented,Noisy Point Data

We propose a robust method for surface mesh reconstruction from unorganized, unoriented, noisy and outlier‐ridden 3D point data. A kernel‐based scale estimator is introduced to estimate the scale of inliers of the input data. The best tangent planes are computed for all points based on mean shift clustering and adaptive scale sample consensus, followed by detecting and removing outliers. Subsequently, we estimate the normals for the remaining points and smooth the noise using a surface fitting and projection strategy. As a result, the outliers and noise are removed and filtered, while the original sharp features are well preserved. We then adopt an existing method to reconstruct surface meshes from the processed point data. To preserve sharp features of the generated meshes that are often blurred during reconstruction, we describe a two‐step approach to effectively recover original sharp features. A number of examples are presented to demonstrate the effectiveness and robustness of our method.     

Using surface variability characteristics for segmentation of deformable 3D objects with application to piecewise statistical deformable model

To cope with the small sample size problem in the construction of Statistical Deformable Models (SDM), this paper proposes two novel measures that quantify the similarity of the variability characteristics among deforming 3D meshes. These measures are used as the basis of our proposed technique for partitioning a 3D mesh for the construction of piecewise SDM in a divide-and-conquer strategy. Specifically, the surface variability information is extracted by performing a global principal component analysis on the set of sample meshes. An iterative face clustering algorithm is developed for segmenting a mesh that favors grouping triangular faces having similar variability characteristics into a same mesh component. We apply the proposed mesh segmentation algorithm to the construction of piecewise SDM and evaluate the representational ability of the resulting piecewise SDM through the reconstruction of unseen meshes. Experimental results show that our approach outperforms several state-of-the-art methods in terms of the representational ability of the resulting piecewise SDM as evaluated by the reconstruction accuracy.