基于深度学习的单视图三维重建

单视图三维重建在计算机视觉领域中是一个具有挑战性的问题.为了提升现有三维重建算法重建后三维模型的精度,本文除了提取图像全局特征之外还提取图像局部特征,结合全局特征和局部特征并选取SDF (signed distance function)作为重建后的三维物体表达方式,不仅提高了模型的精度,生成了更高质量的3D形状,还增强了模型的泛化能力,使得深度模型可以以较高质量重建出其他物体种类.实验结果表明,本文提出的深度网络结构和3D形状表示方法与当今最先进的重建算法相比,无论在重建后三维模型的效果还是新型物体的泛化中都有更好的表现.     

基于深度学习的单视图三维重建综述

计算机视觉是计算机领域研究的热门方向,而三维重建在计算机视觉方面具有很高的研究价值.近年来,随着深度学习、人工智能不断的发展,基于深度学习的单视图三维重建工作受到大量学者的关注.介绍了基于深度学习的单视图三维重建的三种常用方法、三维重建工作常用数据集和其具体的应用领域,并对基于深度学习的单视图三维重建进行了小结与展望.     

ST-Rec3D：基于结构和目标感知的三维重建

基于视图的三维重建旨在从二维图像恢复出其对应的三维形状。现有方法主要通过编码器-解码器结构,结合二元交叉熵函数及其变形,完成三维重建,取得较好的重建结果。然而,编码器在编码过程中缺乏对输入视图的结构感知能力,造成重建的三维模型几何细节不准确;以二元交叉熵函数为主的损失函数在体素分布不均衡的情况下,目标感知能力较差,导致其重建结果存在断裂、缺失等不完整性问题。针对此类问题,提出了一种具有结构和目标感知能力的三维重建网络(ST-Rec3D),以单视图或多视图为输入,由粗到细地重建出三维模型;结合注意力机制提出了一种具有空间结构感知能力的编码器,即结构编码器,以充分捕捉输入视图中的空间结构信息,有效感知重建物体的几何细节;将 IoU 损失引入到三维体素模型重建中,在体素分布不均衡的情况下,精准感知目标物体,确保重建物体的完整性和准确性。在 ShapeNet和 Pix3D 数据集上的对比结果表明,ST-Rec3D 在单视图和多视图上重建的三维模型的完整性和准确性均优于当前方法。     

视觉深度学习的三维重建方法综述

近年来,三维重建作为计算机视觉的重要任务之一,得到广泛的关注和深入的研究。重点分析近年来使用深度学习重建通用对象的三维形状的研究进展。以深度学习进行三维重建环节为脉络,根据三维重建过程中数据深度特征表示方法将三维重建研究分为体素、点云、曲面网格、隐式曲面四类。再根据输入二维图像的数目分为单视图三维重建和多视图三维重建两类,根据网络架构以及它们使用的训练机制进行具体细分,在讨论每个类别的研究进展的同时,分析每种训练方法的发展前景及优缺点。研究近年来在特定三维重建领域的新热点,例如动态人体三维重建和不完整几何数据的三维补全,对一些关键论文进行比较,总结了这些领域存在的问题。介绍现阶段的三维数据集的重点应用场景和参数。总结现阶段三维重建领域存在数据集缺失、模型训练复杂、缺少特定领域针对性识别等问题。对三维重建在未来的具体应用领域发展前景进行了例证分析,并对三维重建的研究方向进行了展望。     

基于深度学习的三维点云修复技术综述

三维点云是最常用的三维场景/物体表示方法之一.根据点云修复侧重点不同,将基于深度学习的三维点云修复技术划分为密集重建、补全重建和去噪重建3类;详细分析了相关典型修复模型及关键技术,如特征编码、特征扩展和损失函数设计;总结了常用的网络模块、点云数据集和评估准则;最后讨论了3类修复技术之间的关系,并从旋转不变性特征提取、细节信息修复、拓扑关系保持、几何算法应用和多模态数据融合5个方面探讨了点云修复技术面临的挑战及未来发展趋势.     

深度学习在基于单幅图像的物体三维重建中的应用

基于单幅图像的物体三维重建是计算机视觉领域的一个重要问题，近几十年来得到了广泛的关注.随着深度学习的不断发展，近年来基于单幅图像的物体三维重建取得了显著进展.本文对深度学习在基于单幅图像的物体三维重建领域的研究进展及具体应用进行了综述.首先介绍了基于单幅图像的三维重建的研究背景及其传统方法的研究现状，其次简要介绍了深度学习并详细综述了深度学习在基于单幅图像的物体三维重建中的应用，随后简要概述了三维物体重建的常用公共数据集，最后进行了分析与总结，指出了目前存在的问题及未来的研究方向.     

信号博弈的三维重建形变识别算法

现有双目立体视觉算法常常需要双目相机位置固定,在现实应用中,这类算法难以重构空间三维几何关系.为此,本文提出了一种不受位置限制的多视角图像三维重建与形变检测算法.该算法首先采用sift算法获取成对图像的特征点,以获取形变前后比对点对图像的特征信息;其次,利用信号博弈方法确定图像拍摄时相机的空间位置与视角,以准确获取图像的空间位置坐标;再次,依据上述信息完成物体的三维点云重建;进而,利用三维数据信息比对实现物体形变识别.最后,本文利用真实物体的实验,验证了三维重建形变识别算法的有效性.     

结构感知深度学习的三维形状分类方法

为了解决复杂、海量三维模型的形状识别问题,提出了一种结构感知深度学习的三维形状分类方法.通过联合学习三维模型的几何结构和空间结构,生成具有结构感知的特征向量表示,该特征向量具有更强的识别力与稳定性,在三维形状分类中取得显著的效果.首先,提取优化的多尺度热核特征,并通过CNN学习模型,有效地获取三维形状的几何结构特征;其次,建立多视图卷积学习网络提取三维形状的空间结构特征;最后,通过联合优化学习生成具有结构感知的深度特征表示.文中采用了C++,Matlab,TensorFlow框架实现,并在公开的三维数据库中进行了大量实验,实验结果表明,文中方法获取的深层结构特征对于复杂拓扑结构、大尺度几何形变的三维形状具有稳定性;与相关方法对比,在三维形状分类中具有更高的分类精度.     

基于多视图深度采样的自然场景三维重建

自然场景的多视图三维重建一直是计算机视觉领域的基本问题,有着广泛的应用.随着深度获取设备的日益普及,如何有效地利用多视点的深度图信息重建场景的三维模型已成为一个重要的研究课题.为了自动剔除输入深度图中的错误深度信息,恢复高质量的场景模型,提出一种多视图深度采样方法来实现自然场景的三维几何重建.首先对深度图进行非均匀采样以获得每帧的三维点集,并剔除部分深度误差较大的三维点;然后通过深度置信度估计的方法对多帧的三维点集进行融合并剔除多帧之间重复冗余的三维点,从而获得整体场景的三维点云;最后基于融合后的三维点云生成完整的场景几何模型.一系列复杂的自然场景实例证明了该方法的正确性和鲁棒性,其不仅能够重建小规模的物体,也同样适用于大规模场景的三维几何重建.     

RGBD融合明暗恢复形状的全视角三维重建技术研究

为了高效、高精度、低成本地实现对物体的全视角三维重建, 提出一种使用深度相机融合光照约束实现全视角三维重建的方法。该重建方法中,在进行单帧重建时采用RGBD深度图像融合明暗恢复形状(Shape from shading,SFS)的重建方法, 即在原有的深度数据上加上额外的光照约束来优化深度值; 在相邻两帧配准时, 采用快速点特征直方图（Fast point feature histograms, FPFH）特征进行匹配并通过随机采样一致性（Random sample consensus, RANSAC）滤除错误的匹配点对求解粗配准矩阵, 然后通过迭代最近点（Iterative closest point, ICP）算法进行精配准得出两帧间的配准矩阵; 在进行全视角的三维重建时, 采用光束平差法优化相机位姿, 从而消除累积误差使首尾帧完全重合, 最后融合生成一个完整的模型。该方法融入了物体表面的光照信息,因此生成的三维模型更为光顺,也包含了更多物体表面的细节信息,提高了重建精度;同时该方法仅通过单张照片就能在自然光环境下完成对多反射率三维物体的重建,适用范围更广。本文方法的整个实验过程通过手持深度相机就能完成,不需要借助转台,操作更加方便。     

Weakly Supervised Part-wise 3D Shape Reconstruction from Single-View RGB Images

In order for the deep learning models to truly understand the 2D images for 3D geometry recovery, we argue that single-view reconstruction should be learned in a part-aware and weakly supervised manner. Such models lead to more profound interpretation of 2D images in which part-based parsing and assembling are involved. To this end, we learn a deep neural network which takes a single-view RGB image as input, and outputs a 3D shape in parts represented by 3D point clouds with an array of 3D part generators. In particular, we devise two levels of generative adversarial network (GAN) to generate shapes with both correct part shape and reasonable overall structure. To enable a self-taught network training, we devise a differentiable projection module along with a self-projection loss measuring the error between the shape projection and the input image. The training data in our method is unpaired between the 2D images and the 3D shapes with part decomposition. Through qualitative and quantitative evaluations on public datasets, we show that our method achieves good performance in part-wise single-view reconstruction.     

基于形变模型的3D表面自适应重建

结合形变模型和ＡＣＤ方法提出了基于变形模型的３Ｄ表面自适应重建方法。同时引入了与图象统计特性有关的外力,使得表面重建结果与模型的初始位置无关,利用ＡＣＤ方法使模型自适应地改变其拓扑结构;为了提高表面重建的程度和鲁棒性,提出了多尺度重建算法,该方法适用于形状、结构复杂的物体重建,实验结果证明了该方法的有效性。     

基于卷积盲降噪的混合式核磁共振成像

为了解决图像压缩感知重建研究领域中通过有效的图像先验信息重构与原图相似性高且保留细节消除伪影的高质量图像的问题, 针对不足采样的K空间数据, 在经典的CNN算法CBDNet算法的基础上, 通过融合深度学习先验信息及传统图像恢复各自优势的方法, 研究了基于深度神经网络去噪先验和BM3D块压缩感知算法的混合式重构算法. 该算法采用交互式方法训练多尺度残差网络抑制噪声水平, 借优化选择的方式将深度学习与传统块匹配多尺度结合以提取图像不同尺度的特征数据从而实现抑制伪影、快速重建高质量MRI. 实结果表明深度学习结合BM3D在MR图像重构领域能够有效降低伪影保留细节信息, 加强重构效果. 与此同时, 通过采用GPU的加速运算, 算法的计算复杂度较使用单一算法并未增加很多. 可见基于卷积盲降噪的混合式核磁共振成像效果更佳.     

Fast Mesh Reconstruction from Single View Based on GCN and Topology Modification

3D reconstruction based on single view aims to reconstruct the entire 3D shape of an object from one perspective. When existing methods reconstruct the mesh surface of complex objects, the surface details are difficult to predict and the reconstruction visual effect is poor because the mesh representation is not easily integrated into the deep learning framework; the 3D topology is easily limited by predefined templates and inflexible, and unnecessary mesh self-intersections and connections will be generated when reconstructing complex topology, thus destroying the surface details; the training of the reconstruction network is limited by the large amount of information attached to the mesh vertices, and the training time of the reconstructed network is too long. In this paper, we propose a method for fast mesh reconstruction from single view based on Graph Convolutional Network (GCN) and topology modification. We use GCN to ensure the generation of high-quality mesh surfaces and use topology modification to improve the flexibility of the topology. Meanwhile, a feature fusion method is proposed to make full use of the features of each stage of the image hierarchically. We use 3D open dataset ShapeNet to train our network and add a new weight parameter to speed up the training process. Extensive experiments demonstrate that our method can not only reconstruct object meshes on complex topological surfaces, but also has better qualitative and quantitative results.     

Surface reconstruction with data-driven exemplar priors

In this paper, we propose a framework to reconstruct 3D models from raw scanned points by learning the prior knowledge of a specific class of objects. Unlike previous work that heuristically specifies particular regularities and defines parametric models, our shape priors are learned directly from existing 3D models under a framework based on affinity propagation. Given a database of 3D models within the same class of objects, we build a comprehensive library of 3D local shape priors. We then formulate the problem to select as-few-as-possible priors from the library, referred to as exemplar priors. These priors are sufficient to represent the 3D shapes of the whole class of objects from where they are generated. By manipulating these priors, we are able to reconstruct geometrically faithful models with the same class of objects from raw point clouds. Our framework can be easily generalized to reconstruct various categories of 3D objects that have more geometrically or topologically complex structures. Comprehensive experiments exhibit the power of our exemplar priors for gracefully solving several problems in 3D shape reconstruction such as preserving sharp features, recovering fine details and so on.     

Novel View Synthesis Of Transparent Object From a Single Image

We propose a method for converting a single image of a transparent object into multi-view photo that enables users observing the object from multiple new angles, without inputting any 3D shape. The complex light paths formed by refraction and reflection makes it challenging to compute the lighting effects of transparent objects from a new angle. We construct an encoder–decoder network for normal reconstruction and texture extraction, which enables synthesizing novel views of transparent object from a set of new views and new environment maps using only one RGB image. By simultaneously considering the optical transmission and perspective variation, our network learns the characteristics of optical transmission and the change of perspective as guidance to the conversion from RGB colours to surface normals. A texture extraction subnetwork is proposed to alleviate the contour loss phenomenon during normal map generation. We test our method using 3D objects within and without our training data, including real 3D objects that exists in our lab, and completely new environment maps that we take using our phones. The results show that our method performs better on view synthesis of transparent objects in complex scenes using only a single-view image.     

单幅植物叶片图像的3维重建

目的 植物叶片形态复杂,在虚拟场景中很难真实表现。为了从信息量有限的单幅图像中恢复植物叶片的3维形状,本文基于从明暗恢复形状（shape from shading,SFS）的方法,利用亮度统计规律和植物形态特征恢复叶片的3维形状。方法 在SFS的基础上,设计基于图像骨架的距离场偏置加强表面细节;针对SFS对恢复宏观几何形状的不足,提出根据图像亮度统计分布选取控制点控制表面宏观形状变化,并利用叶片中轴的距离场约束恢复宏观几何形状,每种方法对于表面宏观几何形状恢复的权重基于恢复的反射图和输入图像间的相似度设定;将表面细节添加到宏观几何形状上得到目标对象的3维形状。结果 选取植物叶片图像进行实验,并与其他方法进行比较,实验结果表明本文方法增强了表面细节显示,并有明显的宏观几何形状变化。同时为了验证本文方法对其他物体表面细节恢复的适用性,分别对硬币和恐龙恢复表面细节,实验结果表明提出的增强表面细节的方法同样适用于其他物体。结论 针对单幅植物叶片图像的3维重建,在SFS的基础上提出了根据骨架特征加强表面细节,根据图像亮度统计分布和叶片中轴距离场约束共同恢复表面宏观几何形状的算法,实验结果验证了本文方法的可行性。     

Three-Dimensional Measurement Using Structured Light Based on Deep Learning

Three-dimensional (3D) reconstruction using structured light projection has the characteristics of non-contact, high precision, easy operation, and strong real-time performance. However, for actual measurement, projection modulated images are disturbed by electronic noise or other interference, which reduces the precision of the measurement system. To solve this problem, a 3D measurement algorithm of structured light based on deep learning is proposed. The end-to-end multi-convolution neural network model is designed to separately extract the coarse- and fine-layer features of a 3D image. The point-cloud model is obtained by nonlinear regression. The weighting coefficient loss function is introduced to the multi-convolution neural network, and the point-cloud data are continuously optimized to obtain the 3D reconstruction model. To verify the effectiveness of the method, image datasets of different 3D gypsum models were collected, trained, and tested using the above method. Experimental results show that the algorithm effectively eliminates external light environmental interference, avoids the influence of object shape, and achieves higher stability and precision. The proposed method is proved to be effective for regular objects.     

The shape variational autoencoder: A deep generative model of part‐segmented 3D objects

We introduce a generative model of part‐segmented 3D objects: the shape variational auto‐encoder (ShapeVAE). The ShapeVAE describes a joint distribution over the existence of object parts, the locations of a dense set of surface points, and over surface normals associated with these points. Our model makes use of a deep encoder‐decoder architecture that leverages the part‐decomposability of 3D objects to embed high‐dimensional shape representations and sample novel instances. Given an input collection of part‐segmented objects with dense point correspondences the ShapeVAE is capable of synthesizing novel, realistic shapes, and by performing conditional inference enables imputation of missing parts or surface normals. In addition, by generating both points and surface normals, our model allows for the use of powerful surface‐reconstruction methods for mesh synthesis. We provide a quantitative evaluation of the ShapeVAE on shape‐completion and test‐set log‐likelihood tasks and demonstrate that the model performs favourably against strong baselines. We demonstrate qualitatively that the ShapeVAE produces plausible shape samples, and that it captures a semantically meaningful shape‐embedding. In addition we show that the ShapeVAE facilitates mesh reconstruction by sampling consistent surface normals.     

基于稀疏特征改进的单视图表面重建

基于深度学习的单视图三维重建是当前的研究热点。为重建出更多的高频细节,SDF-SRN算法引入了位置编码,但在缺乏精确监督时,网络容易过拟合而导致凹凸不平的重建结果。针对这个问题,提出一种基于稀疏特征的网络模型,该模型凭借残差学习机制,令容易过拟合的网络预测高频残差。通过特征提取网络得到稀疏特征和全局特征,稀疏特征输入到一个超网络中生成预测浅头,该浅头负责预测符号距离函数的低频部分,而全局特征输入到另一个超网络生成另一个浅头来预测高频残差,这两部分通过权重因子构成最终的符号距离函数。频谱分析表明实验结果达到了相应的设计目的;与不同平滑表面重建方案对比,基于残差学习的平滑重建方案可以实现更平滑的表面重建,克服了SDF-SRN过拟合的问题,同时保留足够的细节;与其他先进的单视图重建方法的定性和定量对比结果证明了该方法的优越性。