Semi‐Supervised Learning in Reconstructed Manifold Space for 3D Caricature Generation

Recently, automatic 3D caricature generation has attracted much attention from both the research community and the game industry. Machine learning has been proven effective in the automatic generation of caricatures. However, the lack of 3D caricature samples makes it challenging to train a good model. This paper addresses this problem by two steps. First, the training set is enlarged by reconstructing 3D caricatures. We reconstruct 3D caricatures based on some 2D caricature samples with a Principal Component Analysis (PCA)‐based method. Secondly, between the 2D real faces and the enlarged 3D caricatures, a regressive model is learnt by the semi‐supervised manifold regularization (MR) method. We then predict 3D caricatures for 2D real faces with the learnt model. The experiments show that our novel approach synthesizes the 3D caricature more effectively than traditional methods. Moreover, our system has been applied successfully in a massive multi‐user educational game to provide human‐like avatars.     

Free-viewpoint video relighting from multi-view sequence under general illumination

We proposed an approach to create plausible free-viewpoint relighting video using multi-view cameras array under general illumination. Given the multi-view video dataset recorded using a set of industrial cameras under general uncontrolled and unknown illumination, we first reconstruct 3D model of the captured target using existing multi-view stereo approach. Using the coarse geometry reconstruction, we estimate the spatially varying surface reflectance in the spherical harmonics domain considering the spatial and temporal coherence. With the estimated geometry and reflectance, the 3D target is relit to the novel illumination with the environment map of the target environment. Relit performance is enhanced using a flow- and quotient-based transfer strategy to achieve detailed and plausible performance relighting. Finally, the free-viewpoint video is generated using a view-dependent rendering strategy. Experimental results on various dataset show that our approach enables plausible free-view relighting, and opens up a path towards relightable free-viewpoint video using less complex acquisition setups.     

3D reconstruction from 2D images with hierarchical continuous simplices

This paper presents an effective framework for the reconstruction of volumetric data from a sequence of 2D images. The 2D images are first aligned to generate an initial 3D volume, followed by the creation of a tetrahedral domain using the Carver algorithm. The resulting tetrahedralization preserves both the geometry and topology of the original dataset. Then a solid model is reconstructed using simplex splines with fitting and faring procedures. The reconstructed heterogenous volumetric model can be quantitatively analyzed and easily visualized. Our experiments demonstrated that our approach can achieve high accuracy in the data reconstruction. The novel techniques and algorithms proposed in this paper can be applied to reconstruct a heterogeneous solid model with complex geometry and topology from other visual data.     

基于深度学习的多模态融合三维人脸识别

二维人脸识别受光照、遮挡和姿态的影响较大.为了克服二维人脸识别的缺点,本文提出了一种基于深度学习的多模态融合三维人脸识别算法.该方法首先使用卷积自编码器将彩色图像和深度图进行融合,将融合后的图像作为网络的输入进行预训练,并且设计了一种新的损失函数cluster loss,结合Softmax损失,预训练了一个精度非常高的模型.之后使用迁移学习将预训练的模型进行微调,得到了一个轻量级神经网络模型.将原始数据集进行一系列处理,使用处理之后的数据集作为测试集,测试的识别准确率为96.37%.实验证明,该方法弥补了二维人脸识别的一些缺点,受光照和遮挡的影响非常小,并且相对于使用高精度三维人脸图像的三维人脸识别,本文提出的算法速度快,并且鲁棒性高.     

Silhouette and stereo fusion for 3D object modeling

In this paper, we present a new approach to high quality 3D object reconstruction. Starting from a calibrated sequence of color images, the algorithm is able to reconstruct both the 3D geometry and the texture. The core of the method is based on a deformable model, which defines the framework where texture and silhouette information can be fused. This is achieved by defining two external forces based on the images: a texture driven force and a silhouette driven force. The texture force is computed in two steps: a multi-stereo correlation voting approach and a gradient vector flow diffusion. Due to the high resolution of the voting approach, a multi-grid version of the gradient vector flow has been developed. Concerning the silhouette force, a new formulation of the silhouette constraint is derived. It provides a robust way to integrate the silhouettes in the evolution algorithm. As a consequence, we are able to recover the contour generators of the model at the end of the iteration process. Finally, a texture map is computed from the original images for the reconstructed 3D model.     

单目视频中人体三维运动的迭代优化估计

提出并实现了一种从单目视频流中重建人体三维运动的方法．该方法通过交互定制得到个性化的人体骨架模型和视频序列每一帧中人体各关节点的二维坐标后,分别针对单帧和连续多帧进行优化并迭代求解,得到每一帧的比例因子的最优解;最后反求各关节点的三维坐标,重建人体三维运动序列．对包含复杂和快速多变的人体运动的视频进行的实验表明,该方法简单有效,适用于包括体育、影视等在内的实际视频源．     

基于卷积神经网络的视差图生成技术

针对裸眼三维中视差图生成过程中存在的高成本、长耗时以及容易出现背景空洞的问题，提出了一种基于卷积神经网络（CNN）学习预测的算法。首先通过对数据集的训练学习，掌握数据集中的变化规律；然后对输入卷积神经网络中的左视图进行特征提取和预测，得到深度值连续的深度图像；其次将预测所得到的每一个深度图和原图进行卷积，将生成的多个立体图像对进行叠加，最终形成右视图。仿真结果表明：该算法的像素重构尺寸误差相比基于水平视差的三维显示算法和深度图像视点绘制的算法降低了12.82%和10.52%，且背景空洞、背景粘连等问题都得到了明显改善。实验结果表明，卷积神经网络能提高视差图生成的图像质量。     

Frontal and Semi-Frontal Facial Caricature Synthesis Using Non-Negative Matrix Factorization

In this paper, we present a novel approach to synthesizing frontal and semi-frontal cartoon-like facial caricatures from an image. The caricature is generated by warping the input face from the original feature points to the corresponding exaggerated feature points. A 3D mean face model is incorporated to facilitate face to caricatures by inferring the depth of 3D feature points and the spatial transformation. Then the 3D face is deformed by using non-negative matrix factorization and projected back to image plane for future warping. To efficiently solve the nonlinear spatial transformation, we propose a novel initialization scheme to set up Levenberg-Marquardt optimization. According to the spatial transformation, exaggeration is applied to the most salient features by exaggerating their normalized difference from the mean. Non-photorealistic rendering (NPR) based stylization completes the cartoon caricature. Experiments demonstrate that our method outperforms existing methods in terms of view angles and aesthetic visual quality.     

基于动态序列图像的流形学习研究

动态场景的外形或表观在很大程度上往往受到一个潜在低维动态过程的控制。基于视频序列之间的时间相干特性,引入一种称为自编码(autoencoder)的特殊双向深层神经网络,采用CRBM（continuous restricted Boltzmann machine）的网络结构,用来学习序列图像的低维流形结构。将autoencoder 用于人体步态序列的实验表明,该方法能提供从高维视频帧到具有一定物理意义过程的低维序列的映射,并能从低维描述中恢复高维图像序列。     

Leave-one-out manifold regularization

The manifold regularization (MR) based semi-supervised learning could explore structural relationships from both labeled and unlabeled data. However, the model selection of MR seriously affects its predictive performance due to the inherent additional geometry regularizer of labeled and unlabeled data. In this paper, two continuous and two inherent discrete hyperparameters are selected as optimization variables, and a leave-one-out cross-validation (LOOCV) based Predicted REsidual Sum of Squares (PRESS) criterion is first presented for model selection of MR to choose appropriate regularization coefficients and kernel parameters. Considering the inherent discontinuity of the two hyperparameters, the minimization process is implemented by using a improved Nelder-Mead simplex algorithm to solve the inherent discrete and continues hybrid variables set. The manifold regularization and model selection algorithm are applied to six synthetic and real-life benchmark dataset. The proposed approach, leveraged by effectively exploiting the embedded intrinsic geometric manifolds and unbiased LOOCV estimation, outperforms the original MR and supervised learning approaches in the empirical study.     

Identifying and constructing elemental parts of shafts based on conditional random fields model

Semantic information is very important for understanding 2D engineering drawings. However, this kind of information is implicit so that it is hard to be extracted and understood by computers. In this paper, we aim to identify the semantic information of shafts from their 2D drawings, and then reconstruct the 3D models. The 2D representations of shafts are diverse. By analyzing the characteristics of 2D drawings of shafts, we find that there is always a view which represents the projected outline of the shaft, and each loop in this view corresponds to an elemental part. The conditional random fields (CRFs) model is a classification technique which can automatically integrate various features, rather than manually organizing of heuristic rules. We first use a CRFs model to identify elemental parts with semantic information. The 3D elemental parts are then constructed by a parameters template method. Compared with the existing 3D reconstruction methods, our approach can obtain both geometrical information and semantic information of each part of shafts from 2D drawings. Several examples are provided to demonstrate that our algorithm can accurately handle diverse 2D drawings of shafts.     

单幅人脸图像的全景纹理图生成方法

目的针对从单幅人脸图像中恢复面部纹理图时获得的信息不完整、纹理细节不够真实等问题,提出一种基于生成对抗网络的人脸全景纹理图生成方法。方法将2维人脸图像与3维人脸模型之间的特征关系转换为编码器中的条件参数,从图像数据与人脸条件参数的多元高斯分布中得到隐层数据的概率分布,用于在生成器中学习人物的头面部纹理特征。在新创建的人脸纹理图数据集上训练一个全景纹理图生成模型,利用不同属性的鉴别器对输出结果进行评估反馈,提升生成纹理图的完整性和真实性。结果实验与当前最新方法进行了比较,在Celeb A-HQ和LFW(labled faces in the wild)数据集中随机选取单幅正面人脸测试图像,经生成结果的可视化对比及3维映射显示效果对比,纹理图的完整度和显示效果均优于其他方法。通过全局和面部区域的像素量化指标进行数据比较,相比于UVGAN,全局峰值信噪比(peak signal to noise ratio,PSNR)和全局结构相似性(structural similarity index,SSIM)分别提高了7.9 d B和0.088,局部PSNR和局部SSIM分别提高了2.8 d B和0...     

Monocular Vision for Mobile Robot Localization and Autonomous Navigation

This paper presents a new real-time localization system for a mobile robot. We show that autonomous navigation is possible in outdoor situation with the use of a single camera and natural landmarks. To do that, we use a three step approach. In a learning step, the robot is manually guided on a path and a video sequence is recorded with a front looking camera. Then a structure from motion algorithm is used to build a 3D map from this learning sequence. Finally in the navigation step, the robot uses this map to compute its localization in real-time and it follows the learning path or a slightly different path if desired. The vision algorithms used for map building and localization are first detailed. Then a large part of the paper is dedicated to the experimental evaluation of the accuracy and robustness of our algorithms based on experimental data collected during two years in various environments.     

Robust deformable shape reconstruction from monocular video with manifold forests

Existing approaches to recover structure of 3D deformable objects and camera motion parameters from an uncalibrated images assume the object’s shape could be modelled well by a linear subspace. These methods have been proven effective and well suited when the deformations are relatively small, but fail to reconstruct the objects with relatively large deformations. This paper describes a novel approach for 3D non-rigid shape reconstruction, based on manifold decision forest technique. The use of this technique can be justified by noting that a specific type of shape variations might be governed by only a small number of parameters, and therefore can be well represented in a low-dimensional manifold. The key contributions of this work are the use of random decision forests for the shape manifold learning and robust metric for calculation of the re-projection error. The learned manifold defines constraints imposed on the reconstructed shapes. Due to a nonlinear structure of the learned manifold, this approach is more suitable to deal with large and complex object deformations when compared to the linear constraints. The robust metric is applied to reduce the effect of measurement outliers on the quality of the reconstruction. In many practical applications outliers cannot be completely removed and therefore the use of robust techniques is of particular practical interest. The proposed method is validated on 2D points sequences projected from the 3D motion capture data for ground truth comparison and also on real 2D video sequences. Experiments show that the newly proposed method provides better performance compared to previously proposed ones, including the robustness with respect to measurement noise, missing measurements and outliers present in the data.     

基于时序上采样卷积神经网络的风机叶片结冰检测

风电机组叶片结冰检测对风电机组的安全性、可靠性以及经济性具有非常重要的现实意义.针对风电机组运行观测数据的非平衡和单点无时序性问题,提出一种基于过采样和时序上采样卷积神经网络的风机叶片结冰检测方法.首先,采用数据自适应综合过采样算法对原始非平衡数据集进行重采样,实现对非平衡数据集的均衡;然后,提出并构建一种时序上采样卷积神经网络模型,将原始单点向量型数据进行重构并上采样为二维网格型数据,同时将其自动映射成为稀疏的特征表示,实现准确的风机叶片结冰检测功能;最后,将所提出方法在真实风场采集的数据集上进行验证.实验结果表明,所提出的风机叶片结冰检测方法在数据集非平衡且采集条件有限(单点无时序性数据)的情况下,具有一定的有效性、稳定性和可行性.     

基于实拍图像的人脸真实感重建

给出了基于实拍人脸图像的三维逼真人脸模型的重建算法,该算法首先在两幅人脸图像上交互标识特征点对和输入摄像机的广角参数来实现摄像机定标,进而匹配出两幅人脸图像上的其它对应点,实现模型的三维重建,作者用半自动垭达到匹配目的。用手工编辑建立的二维对应网格,得到初始人脸外开和鲁棒的最大拟然立体虎法自动匹配出稠密的对应点,重建出表示人脸的散乱三维数据点团;最后利用这些稠密的三维数据点去迭代矫正和自适应细分手     

Three-dimensional surface reconstruction using meshing growing neural gas (MGNG)

The neural network method, a relatively new method in reverse engineering (RE), has the potential to reconstruct 3D models accurately and fast. A neural network (NN) is a set of interconnected neurons, in which each neuron is capable of making autonomous arithmetic and geometric calculations. Moreover, each neuron is affected by its surrounding neurons through the structure of the network. This work proposes a new approach that utilizes growing neural gas neural network (GNG NN) techniques to reconstruct a triangular manifold mesh. This method has the advantage of reconstructing the surface of an n-genus freeform object without a priori knowledge regarding the original object, its topology or its shape. The resulting mesh can be improved by extending the MGNG into an adaptive algorithm. The proposed method was also extended for micro-structure modeling. The feasibility of the proposed method is demonstrated on several examples of freeform objects with complex topologies.     

Decomposition of complex line drawings with hidden lines for 3D planar-faced manifold object reconstruction

Three-dimensional object reconstruction from a single 2D line drawing is an important problem in computer vision. Many methods have been presented to solve this problem, but they usually fail when the geometric structure of a 3D object becomes complex. In this paper, a novel approach based on a divide-and-conquer strategy is proposed to handle the 3D reconstruction of a planar-faced complex manifold object from its 2D line drawing with hidden lines visible. The approach consists of four steps: 1) identifying the internal faces of the line drawing, 2) decomposing the line drawing into multiple simpler ones based on the internal faces, 3) reconstructing the 3D shapes from these simpler line drawings, and 4) merging the 3D shapes into one complete object represented by the original line drawing. A number of examples are provided to show that our approach can handle 3D reconstruction of more complex objects than previous methods.     

基于切空间判别学习的流形降维算法

在基于图像集的流形降维问题中,许多算法的核心思想都是把一个高维的流形直接降到一个维数相对较低、同时具有的判别信息更加充分的流形上.投影度量学习（projection metric learning,简称PML）是一种Grassmann流形降维算法.该算法是基于投影度量,并且使用RCG（Riemannian conjugate gradient）算法优化目标函数,其在多个数据集上都取得了较好的实验结果,但是对于复杂的人脸数据集,如YTC其实验结果相对较差,只取得了66.69%的正确率.同时,RCG算法的时间效率较差.基于上述原因,提出了基于切空间判别学习的流形降维算法.该算法首先对于PML中的投影矩阵添加扰动,使其成为对称正定（symmetric positive definite,简称SPD）矩阵;然后,使用LEM（log-euclidean metric）将其映射到切空间中;最后,利用基于特征值分解的迭代优化算法构造判别函数,得到变换矩阵.对提算法在多个标准数据集上进行了实验验证,并取得了较好的实验结果,从而验证了该算法的有效性.