Data‐Driven Shape Analysis and Processing

Data‐driven methods serve an increasingly important role in discovering geometric, structural and semantic relationships between shapes. In contrast to traditional approaches that process shapes in isolation of each other, data‐driven methods aggregate information from 3D model collections to improve the analysis, modelling and editing of shapes. Data‐driven methods are also able to learn computational models that reason about properties and relationships of shapes without relying on hard‐coded rules or explicitly programmed instructions. Through reviewing the literature, we provide an overview of the main concepts and components of these methods, as well as discuss their application to classification, segmentation, matching, reconstruction, modelling and exploration, as well as scene analysis and synthesis. We conclude our report with ideas that can inspire future research in data‐driven shape analysis and processing.     

State of the Art on Monocular 3D Face Reconstruction,Tracking, and Applications

The computer graphics and vision communities have dedicated long standing efforts in building computerized tools for reconstructing, tracking, and analyzing human faces based on visual input. Over the past years rapid progress has been made, which led to novel and powerful algorithms that obtain impressive results even in the very challenging case of reconstruction from a single RGB or RGB‐D camera. The range of applications is vast and steadily growing as these technologies are further improving in speed, accuracy, and ease of use. Motivated by this rapid progress, this state‐of‐the‐art report summarizes recent trends in monocular facial performance capture and discusses its applications, which range from performance‐based animation to real‐time facial reenactment. We focus our discussion on methods where the central task is to recover and track a three dimensional model of the human face using optimization‐based reconstruction algorithms. We provide an in‐depth overview of the underlying concepts of real‐world image formation, and we discuss common assumptions and simplifications that make these algorithms practical. In addition, we extensively cover the priors that are used to better constrain the under‐constrained monocular reconstruction problem, and discuss the optimization techniques that are employed to recover dense, photo‐geometric 3D face models from monocular 2D data. Finally, we discuss a variety of use cases for the reviewed algorithms in the context of motion capture, facial animation, as well as image and video editing.     

基于深度学习的三维数据分析理解方法研究综述

基于深度学习的三维数据分析理解是数字几何领域的一个研究热点.不同于基于深度学习的图像分析理解,基于深度学习的三维数据分析理解需要解决的首要问题是数据表达的多样性.相较于规则的二维图像,三维数据有离散表达和连续表达的方法,目前基于深度学习的相关工作多基于三维数据的离散表示,不同的三维数据表达方法与不同的数字几何处理任务对深度学习网络的要求也不同.本文首先汇总了常用的三维数据集与特定任务的评价指标,并分析了三维模型特征描述符.然后从特定任务出发,就不同的三维数据表达方式,对现有的基于深度学习的三维数据分析理解网络进行综述,对各类方法进行对比分析,并从三维数据表达方法的角度进一步汇总现有工作.最后基于国内外研究现状,讨论了亟待解决的挑战性问题,展望了未来发展的趋势.     

Three-dimensional surface relief completion via nonparametric techniques

Common 3D acquisition techniques, such as laser scanning and stereo capture, are realistically only 2.5D in nature. Here we consider the automated completion of hidden or missing portions in 3D scenes originally acquired from 2.5D (or 3D) capture. We propose an approach based on the non-parametric propagation of available scene knowledge from the known (visible) scene areas to these unknown (invisible) 3D regions in conjunction with an initial underlying geometric surface completion.     

DeepGarment : 3D Garment Shape Estimation from a Single Image

3D garment capture is an important component for various applications such as free‐view point video, virtual avatars, online shopping, and virtual cloth fitting. Due to the complexity of the deformations, capturing 3D garment shapes requires controlled and specialized setups. A viable alternative is image‐based garment capture. Capturing 3D garment shapes from a single image, however, is a challenging problem and the current solutions come with assumptions on the lighting, camera calibration, complexity of human or mannequin poses considered, and more importantly a stable physical state for the garment and the underlying human body. In addition, most of the works require manual interaction and exhibit high run‐times. We propose a new technique that overcomes these limitations, making garment shape estimation from an image a practical approach for dynamic garment capture. Starting from synthetic garment shape data generated through physically based simulations from various human bodies in complex poses obtained through Mocap sequences, and rendered under varying camera positions and lighting conditions, our novel method learns a mapping from rendered garment images to the underlying 3D garment model. This is achieved by training Convolutional Neural Networks (CNN‐s) to estimate 3D vertex displacements from a template mesh with a specialized loss function. We illustrate that this technique is able to recover the global shape of dynamic 3D garments from a single image under varying factors such as challenging human poses, self occlusions, various camera poses and lighting conditions, at interactive rates. Improvement is shown if more than one view is integrated. Additionally, we show applications of our method to videos.     

基于多任务学习的有限样本多视角三维形状识别算法

随着三维扫描技术的快速发展,三维形状分析得到了学术界的广泛关注;尤其是深度学习在计算机视觉上取得的显著成功,使得基于多视图的三维形状识别方法成为了目前三维模型识别的主流方式。已有研究表明,三维数据集的数量对于最终的分类精度是一个非常重要的影响条件。然而,由于专业三维扫描设备的限制,三维形状数据难以采集。实际上,现有的公共基准三维数据集的规模远远小于二维数据集,三维形状分析的发展因此受到阻碍。为了解决这一问题,文中主要研究在极小数据样本情况下,三维形状识别问题的优化解策略。受多任务学习的启发,搭建了多分支的网络结构,并引入基于度量学习的辅助比较模块,用于挖掘类内和类间的相似性和差异性信息。网络模型包括主支路与辅助支路,分别使用不同的损失函数对应不同的训练任务,并使用权值超参数平衡多项损失。主支路获得预测分类,使用交叉熵损失函数进行更新;辅助支路得到不同样本间的相似性得分,使用均方差损失函数进行更新。为保证特征向量被投影到同一个空间中,主、辅助支路共享相同的特征提取模块,在训练阶段共同更新参数,在测试阶段仅使用主支路获得的分类结果。在两个公开的三维形状基准数据集上的大量实验结果表明,所提网络结构与训练策略相比传统方法,在少样本的情况下可以显著提高特征模块对不同类别的区分能力,获得更优的识别结果。     

Recent Trends,Applications, and Perspectives in 3D Shape Similarity Assessment

The recent introduction of 3D shape analysis frameworks able to quantify the deformation of a shape into another in terms of the variation of real functions yields a new interpretation of the 3D shape similarity assessment and opens new perspectives. Indeed, while the classical approaches to similarity mainly quantify it as a numerical score, map‐based methods also define (dense) shape correspondences. After presenting in detail the theoretical foundations underlying these approaches, we classify them by looking at their most salient features, including the kind of structure and invariance properties they capture, as well as the distances and the output modalities according to which the similarity between shapes is assessed and returned. We also review the usage of these methods in a number of 3D shape application domains, ranging from matching and retrieval to annotation and segmentation. Finally, the most promising directions for future research developments are discussed.     

Silhouette‐Aware Warping for Image‐Based Rendering

Image‐based rendering (IBR) techniques allow capture and display of 3D environments using photographs. Modern IBR pipelines reconstruct proxy geometry using multi‐view stereo, reproject the photographs onto the proxy and blend them to create novel views. The success of these methods depends on accurate 3D proxies, which are difficult to obtain for complex objects such as trees and cars. Large number of input images do not improve reconstruction proportionally; surface extraction is challenging even from dense range scans for scenes containing such objects. Our approach does not depend on dense accurate geometric reconstruction; instead we compensate for sparse 3D information by variational image warping. In particular, we formulate silhouette‐aware warps that preserve salient depth discontinuities. This improves the rendering of difficult foreground objects, even when deviating from view interpolation. We use a semi‐automatic step to identify depth discontinuities and extract a sparse set of depth constraints used to guide the warp. Our framework is lightweight and results in good quality IBR for previously challenging environments.     

Performance characterization of a high-speed stereo vision sensor for acquisition of time-varying 3D shapes

Acquisition of dynamic dense 3D shape data is of increasing importance in computer vision with applications in various disciplines. In this paper, we investigate the performance of a unique high-speed range sensor based on the stereo vision principle for 3D shape acquisition of animals. The investigation reveals some characteristics of the current version of the sensor with respect to its physical parameters, which suggest an more appropriate configuration of the sensor in real data acquisition scenarios. Due to the novelty of the sensor and the application, we believe that our evaluation of the sensor’s performance will inspire new applications to follow using the dynamic 3D acquisition technology of similar types.     

A diffusion wavelet approach for 3-D model matching

This paper proposes a new 3D shape retrieval approach based on diffusion wavelets which generalize wavelet analysis and associated signal processing techniques to functions on manifolds and graphs. Unlike current works on 3D matching, which are based either on the topological information of the model or its scatter point distribution information, this approach uses both information for more effective matching. Diffusion wavelets enable both global and local analyses on graphs, and can capture the topology of a surface with the diffusion map of its mesh representation. As a result, both multi-scale properties of the 3D geometric model and the topology among the meshes can be extracted for use in 3D geometric model retrieval. Tests using 3D benchmarks demonstrate that the approach based on diffusion wavelets is effective and performs better than those by spherical wavelet and spherical harmonics in 3D model matching.     

基于光点阵列的三维表面数据获取技术及实现

现有的三维表面数据获取技术主要有光飞跃时间、激光扫描和结构光3种,均存在结构复杂、价格昂贵、不易使用的局限.基于光点阵列的三维数据获取完全不同于上述技术,其通过向待测物体表面投射二维光点阵列,用2个相机从不同角度获取图像,利用光点在2幅图像中的视差计算出其空间坐标.由于仅需要一次拍摄,因此可实现瞬时、动态的三维数据获取.该技术结合软件的自动配准和实时显示,可简化人工操作,实现三维测量的高速化、自动化和傻瓜化.     

A Survey on Data‐driven Dictionary‐based Methods for 3D Modeling

Dictionaries are very useful objects for data analysis, as they enable a compact representation of large sets of objects through the combination of atoms. Dictionary‐based techniques have also particularly benefited from the recent advances in machine learning, which has allowed for data‐driven algorithms to take advantage of the redundancy in the input dataset and discover relations between objects without human supervision or hard‐coded rules. Despite the success of dictionary‐based techniques on a wide range of tasks in geometric modeling and geometry processing, the literature is missing a principled state‐of‐the‐art of the current knowledge in this field. To fill this gap, we provide in this survey an overview of data‐driven dictionary‐based methods in geometric modeling. We structure our discussion by application domain: surface reconstruction, compression, and synthesis. Contrary to previous surveys, we place special emphasis on dictionary‐based methods suitable for 3D data synthesis, with applications in geometric modeling and design. Our ultimate goal is to enlight the fact that these techniques can be used to combine the data‐driven paradigm with design intent to synthesize new plausible objects with minimal human intervention. This is the main motivation to restrict the scope of the present survey to techniques handling point clouds and meshes, making use of dictionaries whose definition depends on the input data, and enabling shape reconstruction or synthesis through the combination of atoms.     

3D video and free viewpoint video—From capture to display

This paper gives an end-to-end overview of 3D video and free viewpoint video, which can be regarded as advanced functionalities that expand the capabilities of a 2D video. Free viewpoint video can be understood as the functionality to freely navigate within real world visual scenes, as it is known for instance from virtual worlds in computer graphics. 3D video shall be understood as the functionality that provides the user with a 3D depth impression of the observed scene, which is also known as stereo video. In that sense as functionalities, 3D video and free viewpoint video are not mutually exclusive but can very well be combined in a single system. Research in this area combines computer graphics, computer vision and visual communications. It spans the whole media processing chain from capture to display and the design of systems has to take all parts into account, which is outlined in different sections of this paper giving an end-to-end view and mapping of this broad area. The conclusion is that the necessary technology including standard media formats for 3D video and free viewpoint video is available or will be available in the future, and that there is a clear demand from industry and user for such advanced types of visual media. As a consequence we are witnessing these days how such technology enters our everyday life     

A Part-aware Surface Metric for Shape Analysis

The notion of parts in a shape plays an important role in many geometry problems, including segmentation, correspondence, recognition, editing, and animation. As the fundamental geometric representation of 3D objects in computer graphics is surface-based, solutions of many such problems utilize a surface metric, a distance function defined over pairs of points on the surface, to assist shape analysis and understanding. The main contribution of our work is to bring together these two fundamental concepts: shape parts and surface metric. Specifically, we develop a surface metric that is part-aware. To encode part information at a point on a shape, we model its volumetric context – called the volumetric shape image (VSI) – inside the shape's enclosed volume, to capture relevant visibility information. We then define the part-aware metric by combining an appropriate VSI distance with geodesic distance and normal variation. We show how the volumetric view on part separation addresses certain limitations of the surface view, which relies on concavity measures over a surface as implied by the well-known minima rule. We demonstrate how the new metric can be effectively utilized in various applications including mesh segmentation, shape registration, part-aware sampling and shape retrieval.     

Computer‐vision determination of 3‐D geometric parameters of LDL particles via cryogenic transmission electron microscopy

Abstract— Previous research has shown that the size of the LDL macromolecules can have an effect on cardiovascular health and that LDL macromolecules may be non‐spherical in shape. Some of these studies, however, used methods that are not conducive to automatic determination of the 3‐D parameters of the particles. In particular, the prior methods used for determination of geometric‐parameter determination were either centrifugal separations or manual determination of parameters from cryogenic transmission electron micrographs. An application of computer‐vision techniques to automatically determine the 3‐D parameters from cryogenic transmission electron microscopy (CTEM) images will be described. Correlation of computer‐generated geometric models to the orthonormal projection CTEM imagery were investigated to determine the applicability of finding the pertinent geometric parameters of the expected discoid shape of the LDL particles. The processing showed that the discoid shape can be verified using small‐angle rotations that are more amenable to the limitations of CTEM imaging.     

Single‐image Tomography: 3D Volumes from 2D Cranial X‐Rays

As many different 3D volumes could produce the same 2D x‐ray image, inverting this process is challenging. We show that recent deep learning‐based convolutional neural networks can solve this task. As the main challenge in learning is the sheer amount of data created when extending the 2D image into a 3D volume, we suggest firstly to learn a coarse, fixed‐resolution volume which is then fused in a second step with the input x‐ray into a high‐resolution volume. To train and validate our approach we introduce a new dataset that comprises of close to half a million computer‐simulated 2D x‐ray images of 3D volumes scanned from 175 mammalian species. Future applications of our approach include stereoscopic rendering of legacy x‐ray images, re‐rendering of x‐rays including changes of illumination, view pose or geometry. Our evaluation includes comparison to previous tomography work, previous learning methods using our data, a user study and application to a set of real x‐rays.     

一种三角网格模型的轮廓生成方法

从三维网格模型中提取轮廓信息是一个具有挑战性的过程.现有的方法一般是基于局部形状特征分析,如曲面的曲率和相邻面法向之间的夹角,但其特性通常对模型中的局部特征变化敏感.为了解决这个问题,提出一种基于三维形状几何近似的轮廓提取方法.利用完善的变分几何分割算法来得到一套完整的描述性特征曲线,首先基于变分几何近似方法划分模型为...     

Geometric Snakes for Triangular Meshes

Feature detection is important in various mesh processing techniques, such as mesh editing, mesh morphing, mesh compression, and mesh signal processing. In spite of much research in computer vision, automatic feature detection even for images still remains a difficult problem. To avoid this difficulty, semi‐automatic or interactive techniques for image feature detection have been investigated. In this paper, we propose a geometric snake as an interactive tool for feature detection on a 3D triangular mesh. A geometric snake is an extension of an image snake, which is an active contour model that slithers from its initial position specified by the user to a nearby feature while minimizing an energy functional. To constrain the movement of a geometric snake onto the surface of a mesh, we use the parameterization of the surrounding region of a geometric snake. Although the definition of a feature may vary among applications, we use the normal changes of faces to detect features on a mesh. Experimental results demonstrate that geometric snakes can successfully capture nearby features from user‐specified initial positions.     

动态水面数据采集与重建方法

自然现象模拟是计算机图形学中的一个重要研究热点.如何快速逼真地模拟自然现象是此类研究的主要目的.传统的解决思路大多采用基于物理的建模方法,而随着采集设备的快速发展,基于采集图像的重建方法得到了广泛关注与研究.本文以液体为研究对象,总结了近年来基于采集图像的重建方法的部分研究成果.针对动态水面,提出了一种动态水面数据采集与重建方法.首先,设计并搭建一套基于多相机的动态水面数据采集系统,采集得到多视图下不同水面运动现象的连续帧图像.其次,通过提取采集图像序列中每幅图像的亚像素级特征点,进行特征点匹配并建立特征点与物理空间中三维点的映射关系.然后,结合水介质的光学折射特性迭代求解水面上三维点阵的高度场和法向量.最终获得动态水面的重建结果.实验结果表明该方法能快速生成与采集水面可视效果相近的三维重建结果,可在计算机游戏、医学、科学研究可视化等领域具有应用前景.     

Database‐Assisted Object Retrieval for Real‐Time 3D Reconstruction

In recent years, real‐time 3D scanning technology has developed significantly and is now able to capture large environments with considerable accuracy. Unfortunately, the reconstructed geometry still suffers from incompleteness, due to occlusions and lack of view coverage, resulting in unsatisfactory reconstructions. In order to overcome these fundamental physical limitations, we present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real‐time. With this approach, we are able to replace scanned RGB‐D data with complete, hand‐modeled objects from shape databases. We align and scale retrieved models to the input data to obtain a high‐quality virtual representation of the real‐world environment that is quite faithful to the original geometry. In contrast to previous methods, we are able to retrieve objects in cluttered and noisy scenes even when the database contains only similar models, but no exact matches. In addition, we put a strong focus on object retrieval in an interactive scanning context — our algorithm runs directly on 3D scanning data structures, and is able to query databases of thousands of models in an online fashion during scanning.