基于深度卷积神经网络的三维模型检索

为了更有效地利用三维模型数据集进行特征的自主学习,提出一种使用自然图像作为输入源,以三维模型的较优视图集为基础,通过深度卷积神经网络的训练提取深度特征用于检索的三维模型检索方法。首先,从多个视点对三维模型进行视图提取,并根据灰度熵的排序选取较优视图;然后,通过深度卷积神经网络对视图集进行训练,从而提取较优视图的深度特征并进行降维,同时,对输入的自然图像提取边缘轮廓图,经过相似度匹配获得一组三维模型;最后,基于检索结果中同类模型总数占检索列表长度的比例对列表进行重排序,从而获得最终的检索结果。实验结果表明,该算法能够有效利用深度卷积神经网络对三维模型的视图进行深度特征提取,同时降低了输入源的获取难度,有效提高了检索效果。     

3D human modeling from a single depth image dealing with self-occlusion

This paper presents a 2D to 3D conversion scheme to generate a 3D human model using a single depth image with several color images. In building a complete 3D model, no prior knowledge such as a pre-computed scene structure and photometric and geometric calibrations is required since the depth camera can directly acquire the calibrated geometric and color information in real time. The proposed method deals with a self-occlusion problem which often occurs in images captured by a monocular camera. When an image is obtained from a fixed view, it may not have data for a certain part of an object due to occlusion. The proposed method consists of following steps to resolve this problem. First, the noise in a depth image is reduced by using a series of image processing techniques. Second, a 3D mesh surface is constructed using the proposed depth image-based modeling method. Third, the occlusion problem is resolved by removing the unwanted triangles in the occlusion region and filling the corresponding hole. Finally, textures are extracted and mapped to the 3D surface of the model to provide photo-realistic appearance. Comparison results with the related work demonstrate the efficiency of our method in terms of visual quality and computation time. It can be utilized in creating 3D human models in many 3D applications.     

基于图像辅助的三维模型形状检索

随着3D建模技术的快速发展,互联网上可用的3D模型库出现了爆炸式增长,越来越多的3D模型可以方便地通过网络下载使用.这直接促使了3D形状检索技术的发展,即给出特定的搜索信息,要求系统搜索出符合要求的、相似的3D模型.文章提出了一种新的3D形状检索方法,以3D模型作为输入,系统将会从模型数据库中自动检索出与输入形状最相似的模型.对于给定的输入模型以及数据库中的每一个模型,首先由计算机生成多幅在不同视角下的2D草图;然后,应用Gabor滤波器对每一幅2D草图提取图像上的局部特征,并对特征进行量化,从而得到代表该图像特征的直方图,这样对于每一个3D模型将得到多个代表该模型的直方图;最终,通过对比两个模型之间直方图的相似性,可以得到它们的相似性值,从而检索出与输入模型最相似的模型.文章所提出的方法通过采取2D图像分析方法提取能反映3D模型的特征并计算出模型之间的相似性值.经过测试,在一些公开的数据集上得到了较好的效果.     

3D street art illusions: embedding chalk stylized rendering of 3D objects into a pavement photo

Three dimensional street art illusions have become more popular in recent years. Many of them are drawn on pavement such as streets, sidewalks, and town squares. It is often known as 3D chalk art, where a 2D artwork is drawn on the street, giving the viewer a 3D optical illusion from a certain perspective. This paper supplies chalk stylized renderings for 3D models and synthesizes 3D objects into a realistic photograph. Users can input one photo for the 3D model. First, a camera position is achieved by using a camera calibration algorithm. Second, a chalk stylized rendering was applied to create an artistic image of the 3D models using the same camera position. Then, the non‐photorealistic rendering image is composited into a source photograph using a modified Poisson approach. All of the enhanced pavement texture details are also blended into the object's image. The major contribution of this paper is providing a user to create any interesting and attractive 3D illusionary art without physically drawing pictures on the pavement. Finally, the proposed method is demonstrated using various experimental 3D street art illusion images. Copyright © 2014 John Wiley & Sons, Ltd.     

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.     

3D model retrieval via single image based on feature mapping

With the development of manufacture, more and more 3D models are generated by users and many differnet factories. 3D model retrieval has been receiving more and more attention in computer vision and the field of data analysis. In this paper, we propose a novel 3D model retrieval algorithm by cross-modal feature mapping (CMFM), which utilize one single image as query information to address 3D model retrieval problem. Specifically, in this paper, we first proposed to leverage 2D image to handle 3d model retrieval problem, which is one new problem in this field. The proposed feature learning method can benefit: 1) avoiding the interference of query image recorded by different visual sensor; 2) handling cross-modal data retrieval by simple computer vision technologies, which can guarantee the performance of retrieval and also control that the retrieval time hold a low level; 3) the low complexity of this method can guarantee that this method can be applied in many fields. Finally, we validate the retrieval method on three popular datasets. Extensive comparison experiments show the superiority of the proposed mehtod. To the best of our knowledge, it is the first method to handle 3D model retreival based on one single 2D image.     

Multi-View AAM Fitting and Construction

Active Appearance Models (AAMs) are generative, parametric models that have been successfully used in the past to model deformable objects such as human faces. The original AAMs formulation was 2D, but they have recently been extended to include a 3D shape model. A variety of single-view algorithms exist for fitting and constructing 3D AAMs but one area that has not been studied is multi-view algorithms. In this paper we present multi-view algorithms for both fitting and constructing 3D AAMs. Fitting an AAM to an image consists of minimizing the error between the input image and the closest model instance; i.e. solving a nonlinear optimization problem. In the first part of the paper we describe an algorithm for fitting a single AAM to multiple images, captured simultaneously by cameras with arbitrary locations, rotations, and response functions. This algorithm uses the scaled orthographic imaging model used by previous authors, and in the process of fitting computes, or calibrates, the scaled orthographic camera matrices. In the second part of the paper we describe an extension of this algorithm to calibrate weak perspective (or full perspective) camera models for each of the cameras. In essence, we use the human face as a (non-rigid) calibration grid. We demonstrate that the performance of this algorithm is roughly comparable to a standard algorithm using a calibration grid. In the third part of the paper, we show how camera calibration improves the performance of AAM fitting. A variety of non-rigid structure-from-motion algorithms, both single-view and multi-view, have been proposed that can be used to construct the corresponding 3D non-rigid shape models of a 2D AAM. In the final part of the paper, we show that constructing a 3D face model using non-rigid structure-from-motion suffers from the Bas-Relief ambiguity and may result in a “scaled” (stretched/compressed) model. We outline a robust non-rigid motion-stereo algorithm for calibrated multi-view 3D AAM construction and show how using calibrated multi-view motion-stereo can eliminate the Bas-Relief ambiguity and yield face models with higher 3D fidelity. Electronic Supplementary Material The online version of this article () contains supplementary material, which is available to authorized users.     

3D遮挡模型引导的光场图像深度获取

目的 光场相机可以通过单次曝光同时从多个视角采样单个场景,在深度估计领域具有独特优势。消除遮挡的影响是光场深度估计的难点之一。现有方法基于2D场景模型检测各视角遮挡状态,但是遮挡取决于所采样场景的3D立体模型,仅利用2D模型无法精确检测,不精确的遮挡检测结果将降低后续深度估计精度。针对这一问题,提出了3D遮挡模型引导的光场图像深度获取方法。方法 向2D模型中的不同物体之间添加前后景关系和深度差信息,得到场景的立体模型,之后在立体模型中根据光线的传输路径推断所有视角的遮挡情况并记录在遮挡图（occlusion map）中。在遮挡图引导下,在遮挡和非遮挡区域分别使用不同成本量进行深度估计。在遮挡区域,通过遮挡图屏蔽被遮挡视角,基于剩余视角的成像一致性计算深度;在非遮挡区域,根据该区域深度连续特性设计了新型离焦网格匹配成本量,相比传统成本量,该成本量能够感知更广范围的色彩纹理,以此估计更平滑的深度图。为了进一步提升深度估计的精度,根据遮挡检测和深度估计的依赖关系设计了基于最大期望（exception maximization,EM）算法的联合优化框架,在该框架下,遮挡图和深度图通过互相引导的方式相继提升彼此精度。结果 实验结果表明,本文方法在大部分实验场景中,对于单遮挡、多遮挡和低对比度遮挡在遮挡检测和深度估计方面均能达到最优结果。均方误差（mean square error,MSE）对比次优结果平均降低约19.75%。结论 针对遮挡场景的深度估计,通过理论分析和实验验证,表明3D遮挡模型相比传统2D遮挡模型在遮挡检测方面具有一定优越性,本文方法更适用于复杂遮挡场景的深度估计。     

3D CAD model matching from 2D local invariant features

The matching of particular types of CAD models to existing physical models can provide invaluable support to the process of CAD design and reuse. To meet the demand for fast and robust algorithms to detect predefined models in database, an local invariant model matching approach is proposed in this paper. It first maps the 3D CAD model to 2D principal image plane by its first two principal components, and then finds affine invariant key points in the 2D image. The CAD model matching problem is implemented as key points matching. Experimental results show the proposed 3D model retrieval method performs fairly well in retrieving similar models from a database of 3D CAD models.     

Least Committed Splines in 3D Modelling of Free Form Objects from Intensity Images

Generating 3D models of objects from video sequences is an important problem in many multimedia applications ranging from teleconferencing to virtual reality. In this paper, we present a method of estimating the 3D face model from a monocular image sequence, using a few standard results from the affine camera geometry literature in computer vision, and spline fitting techniques using a modified non parametric regression technique. We use the bicubic spline functions to model the depth map, given a set of observation depth maps computed from frame pairs in a video sequence. The minimal number of splines are chosen on the basis of the Schwartz's Criterion. We extend the spline fitting algorithm to hierarchical splines. Note that the camera calibration parameters and the prior knowledge of the object shape is not required by the algorithm. The system has been successfully demonstrated to extract 3D face structure of humans as well as other objects, starting from their image sequences.     

基于单张照片的三维人脸重建优化算法

传统人脸三维重建算法难以确定人脸形状,并且计算复杂。针对此问题,提出一种以水平集方法获取人脸轮廓并结合明暗恢复形状(SFS)算法重建三维模型的方法,该方法仅需单张正面人脸照片。首先采用主动形状模型确定人脸轮廓,将其作为水平集的初始演化曲线,分割出完整的人脸形状;然后对人脸区域进行灰度变换,求出灰度图像;最后通过SFS算法重建已知光照条件的人脸图像的三维模型,将该模型作为参考与灰度图像匹配,进而确定其光照条件和三维模型。实验结果表明,与基于网格模型的算法相比,该方法可快速地重建具有完整形状的人脸模型。     

Convolutional deep learning for 3D object retrieval

In recent years, with the development of 3D technologies, 3D model retrieval has become a hot topic. The key point of 3D model retrieval is to extract robust feature for 3D model representation. In order to improve the effectiveness of method on 3D model retrieval, this paper proposes a feature extraction model based on convolutional neural networks (CNN). First, we extract a set of 2D images from 3D model to represent each 3D object. SIFT detector is utilized to detect interesting points from each 2D image and extract interesting patches to represent local information of each 3D model. X-means is leveraged to generate the CNN filters. Second, a single CNN layer learns low-level features which are then given as inputs to multiple recursive neural networks (RNN) in order to compose higher order features. RNNs can generate the final feature for 2D image representation. Finally, nearest neighbor is used to compute the similarity between different 3D models in order to handle the retrieval problem. Extensive comparison experiments were on the popular ETH and MV-RED 3D model datasets. The results demonstrate the superiority of the proposed method.     

3D model retrieval using principal plane analysis and dynamic programming

Three dimensional models play an important role in many applications; the problem is how to select the appropriate models from a 3D database rapidly and accurately. In recent years, a variety of shape representations, statistical methods, and geometric algorithms have been proposed for matching 3D shapes or models. In this paper, we propose a 3D shape representation scheme based on a combination of principal plane analysis and dynamic programming. The proposed 3D shape representation scheme consists of three steps. First, a 3D model is transformed into a 2D image by projecting the vertices of the model onto its principal plane. Second, the convex hall of the 2D shape of the model is further segmented into multiple disjoint triangles using dynamic programming. Finally, for each triangle, a projection score histogram and moments are extracted as the feature vectors for similarity searching. Experimental results showed the robustness of the proposed scheme, which resists translation, rotation, scaling, noise, and destructive attacks. The proposed 3D model retrieval method performs fairly well in retrieving models having similar characteristics from a database of 3D models.     

3D depth information extraction with omni-directional camera

This paper presents a novel 3D depth information extraction method without calibration. Firstly, this paper develops an omni-directional 3D camera system, which consists of a CCD camera, hyperbolic mirror, infrared laser diodes and diffractive of element (DOE). Secondly, a depth measurement model is proposed to obtain the 3D depth information. Finally, in order to calculate the speckle shift accurately between the reference image and the object image, a dot matrix pattern and sequence coding algorithm are designed to find the corresponding speckles in the two images. Experimental results show that the reconstructed depth data have a good correlation with the actual distance. The accuracy of the data is also found to be influenced by the distance between the object and the camera.     

基于相关型图像传感器3维人脸成像的3维AAMs人脸识别方法的研究

主动外观模型是基于统计分析建立物体2维模型的有效方法,它融合了目标的形状和纹理信息。在基于相关型图像传感器3维人脸成像的基础上,提出了一种建立3维人脸模型的方法,该方法利用由相关型图像传感器得到的深度信息和与之对应的亮度信息将2维AAMs扩展为3维AAMs,融合人脸的形状,纹理和深度信息来构建3维人脸模型。人脸识别实验结果表明,该方法在不同人脸姿态,表情和光照条件下识别效果要优于Eigenface和2维AAMs。     

基于SFM算法的三维人脸模型重建

提出了一种根据两幅正面人脸图像和一幅侧面图像重建人脸三维模型的算法，该算法主要包括4个步骤：寻找匹配点；采用SFM算法计算出特征点的三维坐标，并组成稀疏的三维网格结构；采用分步紧支撑径向基函数进行三维插值，得到三维模型；最后根据多分辨图像拼接算法生成纹理图像并将其映射到三维模型上，从而增强真实感，与其它算法相比，该算法最大的不同之处在于匹配点的寻找，匹配点的准确与否直接影响SFM算法结果的正确性，许多寻找匹配点的算法如角点匹配算法，在处理人脸图像时得到的结果并不稳定，这是因为人脸图像上包含了许多低纹理和重复纹理区域，大多数算法将代表人脸结构基本特征的基准模型运用在重建过程的最后一步，通过三维逼近运算，得到最终的重建模型，而该算法将反映人脸共性特征的几何对称性和规律性运用到匹配点的寻找中，能够快速准确地找出SFM算法需要的匹配点，用户使用普通照相机拍摄到的图像经本算法的处理后就可以得到相应的三维人脸结构。     

基于整数中轴骨架的3维模型检索算法

针对3维模型检索算法性能较低的问题,提出了一种基于整数中轴骨架的3维模型检索算法。在对3维模型进行姿态调整和各向同向性预处理后,提取模型的整数中轴骨架,并记录每个骨架点相应的几何信息,对提取的骨架按不同的空间区域划分,形成模型骨架二叉树。为了能够描述骨架二叉树的不同节点对模型整体相似性匹配的影响程度,为每个节点定义一个特征权值,其大小由该节点对应的骨架区域大小所决定。最后,采用由粗到细逐步淘汰的策略计算不同模型的相似度。对一个标准3维模型测试数据库的检索实验结果表明,由于将模型的拓扑结构和统计特征相结合,该算法可以得到较好的检索性能。     

实时交互的带浮雕纹理的三维模型构建方法

为了快速生成带浮雕纹理的三维模型,提出一种实时交互的浮雕纹理模型构建方法。方法分两步：第一步,将生成浮雕的源模型或图像转换为初始深度图,并进一步转换为梯度图,再通过梯度域的压缩、过滤,求解线性方程重建出整体连续的浮雕深度图;第二步,借助基于网格求交的浮雕纹理映射算法将浮雕深度图贴在目标模型表面,并通过移动、旋转、缩放等操作实时在目标模型三维空间上修改浮雕效果,最终重建目标模型网格,生成浮雕纹理模型。实验表明,所提方法可快速实现在一个目标模型上生成凹浮雕、凸浮雕、多浮雕等效果,所得模型无需经过其他处理,可直接应用于3D打印,打印效果较好。     

深度学习在单图像三维模型重建的应用

针对基于单图像重建的三维模型具有高度不确定性问题,提出了一种基于深度图像估计、球面投影映射、三维对抗生成网络相结合的网络模型算法。首先,通过深度估计器得到输入图像的深度图像,这有利于对图像进一步的分析;其次,将得到的深度图像通过球面投影映射转换为三维模型;最后,利用三维对抗生成网络对重建的三维模型的真实性进行判断,建立更逼真的三维模型。理论分析和仿真实验表明,与学习先验知识生成三维模型的算法LVP相比,所提模型在真实三维模型与重建三维模型的交并比（IoU）上提高了20.1%,倒角距离（CD）缩小了13.2%。实验结果表明,所提模型在单视图三维模型重建中具有良好的泛化能力。