Cross-view information interaction and feedback network for face hallucination

Hallucinating a photo-realistic frontal face image from a low-resolution (LR) non-frontal face image is beneficial for a series of face-related applications. However, previous efforts either focus on super-resolving high-resolution (HR) face images from nearly frontal LR counterparts or frontalizing non-frontal HR faces. It is necessary to address all these challenges jointly for real-world face images in unconstrained environment. In this paper, we develop a novel Cross-view Information Interaction and Feedback Network (CVIFNet), which simultaneously handles the non-frontal LR face image super-resolution (SR) and frontalization in a unified framework and interacts them with each other to further improve their performance. Specifically, the CVIFNet is composed of two feedback sub-networks for frontal and profile face images. Considering the reliable correspondence between frontal and non-frontal face images can be crucial and contribute to face hallucination in a different manner, we design a cross-view information interaction module (CVIM) to aggregate HR representations of different views produced by the SR and frontalization processes to generate finer face hallucination results. Besides, since 3D rendered facial priors contain rich hierarchical features, such as low-level (e.g., sharp edge and illumination) and perception level (e.g., identity) information, we design an identity-preserving consistency loss based on 3D rendered facial priors, which can ensure that the high-frequency details of frontal face hallucination result are consistent with the profile. Extensive experiments demonstrate the effectiveness and advancement of CVIFNet.     

Face completion with Hybrid Dilated Convolution

Image completion is a challenging task which aims to fill the missing or masked regions in images with plausibly synthesized contents. In this paper, we focus on face image inpainting tasks, aiming at reconstructing missing or damaged regions of an incomplete face image given the context information. We specially design the U-Net architecture to tackle the problem. The proposed U-Net based method combines Hybrid Dilated Convolution (HDC) and spectral normalization to fill in missing regions of any shape with sharp structures and fine-detailed textures. We perform both qualitative and quantitative evaluation on two challenging face datasets. Experimental results demonstrate that our method outperforms previous learning-based inpainting methods. The proposed method can generate realistic and semantically plausible images.     

基于形状匹配变形模型的三维人脸重构

为了从多幅人脸图像构造三维人脸结构,通常需要自动提取不同图像中的对应特征点,这往往是很难完成的.为了避免这个困难,本文建立了一个基于形状匹配的三维变形模型,在保证形状最佳匹配的条件下,实现对人脸图像姿态的估计和三维人脸重构.模型采用径向基函数对通用头部模型进行变形,用形状上下文来描述点之间的形状相似性,形状距离用来描述头部模型和人脸图像整体形状上的相似性,从而实现形状最佳匹配意义上的三维重构.实验表明,本文的算法只需要在人脸图像中提取特征点集,不需进行配准,就可以恢复出令人满意的三维头部结构.     

基于深度卷积网络的3D人脸重构算法

基于单张人脸图片的3D人脸模型重构,无论是在计算机图形领域还是可见光成像领域都是一个极具挑战性的研究方向,对于人脸识别、人脸成像、人脸动画等实际应用更是具有重要意义。针对目前算法复杂度较高、运算量较大且存在局部最优解和初始化不良等问题,本文提出了一种基于深度卷积神经网络的单张图片向3D人脸自动重构算法。该算法首先基于3D转换模型来提取2D人脸图像的密集信息,然后构建深度卷积神经网络架构、设计总体损失函数,直接学习2D人脸图像从像素到3D坐标的映射,从而实现了3D人脸模型的自动构建。算法对比与仿真实验表明,该算法在3D人脸重建上的归一化平均误差更低,且仅需一张2D人脸图像便可自动重构生成3D人脸模型。所生成的3D人脸模型鲁棒性好,重构准确,完整保留表情细节,并且对不同姿态的人脸也具有较好的重建效果,能够在三维空间中无死角自由呈现,将满足更多实际应用需求。     

Multi-pose 3D face recognition based on 2D sparse representation

The increasing availability of 3D facial data offers the potential to overcome the difficulties inherent with 2D face recognition, including the sensitivity to illumination conditions and head pose variations. In spite of their rapid development, many 3D face recognition algorithms in the literature still suffer from the intrinsic complexity in representing and processing 3D facial data. In this paper, we propose the intrinsic 3D facial sparse representation (I3DFSR) algorithm for multi-pose 3D face recognition. In this algorithm, each 3D facial surface is first mapped homeomorphically onto a 2D lattice, where the value at each site is the depth of the corresponding vertex on the 3D surface. Each 2D lattice is then interpolated and converted into a 2D facial attribute image. Next, the sparse representation is applied to those attribute images. Finally, the identity of each query face can be obtained by using the corresponding sparse coefficients. The innovation of our approach lies in the strategy of converting irregular 3D facial surfaces into regular 2D attribute images such that 3D face recognition problem can be solved by using the sparse representation of those attribute images. We compare the proposed algorithm to three widely used 3D face recognition algorithms in the GavabDB database, to six state-of-the-art algorithms in the FRGC2.0 database, and to three baseline algorithms in the NPU3D database. Our results show that the proposed I3DFSR algorithm can substantially improve the accuracy and efficiency of multi-pose 3D face recognition.     

2D facial expression recognition via 3D reconstruction and feature fusion

In this paper, a novel feature extraction method is proposed for facial expression recognition by extracting the feature from facial depth and 3D mesh alongside texture. Accordingly, the 3D Facial Expression Generic Elastic Model (3D FE-GEM) method is used to reconstruct an expression-invariant 3D model from the human face. Then, the texture, depth and mesh are extracted from the reconstructed face model. Afterwards, the Local Binary Pattern (LBP), proposed 3D High-Low Local Binary Pattern (3DH-LLBP) and Local Normal Binary Patterns (LNBPs) are applied to texture, depth and mesh of the face, respectively, to extract the feature from 2D images. Finally, the final feature vectors are generated through feature fusion and are classified by the Support Vector Machine (SVM). Convincing results are acquired for facial expression recognition on the CK+, CK, JAFFE and Bosphorus image databases compared to several state-of-the-art methods.     

Semantic prior-driven fused contextual transformation network for image inpainting

Recent advances in image inpainting have achieved impressive performance for generating plausible visual details on small regular image defects or simple backgrounds. However, current solution suffers from the lack of semantic priors for the image and the inability to deduce the image content from distant background, leading to distorted structures and artifacts in the results when inpainting large random irregular complicated images. To address these problems, a semantic prior-driven fused contextual transformation network for image inpainting is proposed as a promise solution. First, the semantic prior generator is put forward to map the semantic features of ground truth images and the low-level features of broken images to semantic priors. Subsequently, an image split-transform-aggregated strategy, named fusion context transformation block, is presented to infer rich multi-scale remote texture features and thus to improve the restored image finesse. Thereafter, an aggregated semantic attention-aware module, consisting of spatially adaptive normalization and enhanced spatial attention is designed to aggregate semantic priors and multi-scale texture features into the decoder to restore reasonable structure. Finally, the mask guided discriminator is developed to effectively discriminate between real and false pixels in the output image to improve the capability of the discriminator and hence to reduce the probability of artifacts containing in the output image. Comprehensive experimental results on CelebA-HQ, Paris Street View, and Places2 datasets demonstrate the superiority of the proposed network over the state-of-the-arts, whose PSNR, SSIM and MAE are improved about 20 %, 12.6 %, and 42 % gains, respectively.     

三维人脸特征提取方法综述

有效提取人脸特征是人脸识别技术的关键组成部分。传统的二维图像容易受到光照、姿态及表情的影响,而三维数据被认为具有光照姿态不变性。文中从局部特征和整体特征两个角度,对三维人脸特征提取进行综述,对部分方法进行比较,并分析了方法的有效性,总结了三维人脸特征提取方法的优势和困难。     

基于稀疏形变模型的三维卡通人脸生成

 随着数字娱乐产业的发展,由照片生成卡通人脸的技术将取得广泛应用.此前的方法主要集中在平面卡通化的领域,风格较为单一.对于三维人脸,尽管形变模型方法可以由照片合成各种属性的三维人脸,但它计算量较大,不适用于实时应用场合.本文提出了一种基于稀疏形变模型的三维卡通人脸生成方法,提高了计算速度,且只需要单幅正面人脸照片.首先由稀疏形变模型拟合照片人脸获得特定的稀疏人脸模型;然后将一个一般人脸模型变形到特定人脸并合成纹理;最后对三维人脸进行卡通化.实验结果证明本文方法能够快速自动地合成生动的三维卡通人脸.     

Using 3D face priors for depth recovery

For repairing inaccurate depth measurements from commodity RGB-D sensors, existing depth recovery methods primarily rely on low-level and rigid prior information. However, as the depth quality deteriorates, the recovered depth maps become increasingly unreliable, especially for non-rigid objects. Thus, additional high-level and non-rigid information is needed to improve the recovery quality. Taking as a starting point the human face that is the primary prior available in many high-level tasks, in this paper, we incorporate face priors into the depth recovery process. In particular, we propose a joint optimization framework that consists of two main steps: transforming the face model for better alignment and applying face priors for improved depth recovery. Face priors from both sparse and dense 3D face models are studied. By comparing with the baseline method on benchmark datasets, we demonstrate that the proposed method can achieve up to 23.8% improvement in depth recovery with more accurate face registrations, bringing inspirations to both non-rigid object modeling and analysis.     

Robust frontal view search using extended manifold learning

Many 2D face processing algorithms can perform better using frontal or near frontal faces. In this paper, we present a robust frontal view search method based on manifold learning, with the assumption that with the pose being the only variable, face images should lie in a smooth and low-dimensional manifold. In 2D embedding, we find that manifold geometry of face images with varying poses has the shape of a parabola with the frontal view in the vertex. However, background clutter and illumination variations make frontal view deviate from the vertex. To address this problem, we propose a pairwise K-nearest neighbor protocol to extend manifold learning. In addition, we present an illumination-robust localized edge orientation histogram to represent face image in the extended manifold learning. The experimental results show that the extended algorithms have higher search accuracy, even under varying illuminations.     

Robust face anti-spoofing with depth information

With the prevalence of face authentication applications, the prevention of malicious attack from fake faces such as photos or videos, i.e., face anti-spoofing, has attracted much attention recently. However, while an increasing number of works on the face anti-spoofing have been reported based on 2D RGB cameras, most of them cannot handle various attacking methods. In this paper we propose a robust representation jointly modeling 2D textual information and depth information for face anti-spoofing. The textual feature is learned from 2D facial image regions using a convolutional neural network (CNN), and the depth representation is extracted from images captured by a Kinect. A face in front of the camera is classified as live if it is categorized as live using both cues. We collected a face anti-spoofing experimental dataset with depth information, and reported extensive experimental results to validate the robustness of the proposed method.     

人脸图像中鼻子检测技术综述

鼻子是人脸中一个突出的器官,其特征不易受面部表情变化的影响。鼻子检测是在图像或图像序列中搜索人鼻的位置及其轮廓线特征,其研究在人脸检测和定位、人脸识别、人脸姿态估计、3D人脸重构等方面具有重要的意义。近年来,研究者们在该领域做了大量研究,提出了很多有效的算法。对相关文献进行了综述,将现有的鼻子检测方法分为基于2D图像的方法和基于3D信息的方法,分析了这两类方法的优缺点。     

基于局部形变模型三维人脸快速建模

针对传统三维形变模型计算量大,难以满足实时性的不足,提出一种实用、高效的三维形变模型,进行个性化三维人脸重建。首先,对三维人脸关键特征点定位,分割划片;然后,对每个分片分别建立形变模型并进行匹配恢复其局部形状信息;最后,把每部分进行无缝拼接,生成逼真的三维人脸。实验结果表明,该方法能够获得较好的建模精度,在短时间内可以通过单幅真实图像重建出逼真的三维人脸模型。     

基于改进二次误差测度及网格参数化的三维人脸本征属性描述方法

3维人脸特征描述是3维人脸配准及识别的关键技术。该文针对3维人脸高分辨率模型特征分布不均匀且存在信息冗余的问题,提出一种基于模型简化和网格参数化的3维人脸特征描述方法。采用半边折叠及自适应收缩代价加权等手段对基于二次误差测度的网格简化方法进行改进,克服原算法中存在重叠三角形和丢失细节特征的问题。同时,基于多分辨分析思想,利用特征约束的保形同构映射对简化后的3维人脸模型在2维平面进行保形展开,并由此构造多分辨2维本征属性图。该方法将3维空间运算问题简化为简单的2维图像运算,显著降低了计算复杂度。对GavabDB 3维人脸库的识别实验表明,该文方法能有效描述3维人脸的本征属性,同时对数据缺失具有较强的鲁棒性。     

Unsupervised face recognition in the wild using high-dimensional features under super-resolution and 3D alignment effect

Face recognition algorithms customarily utilize query faces captured from uncontrolled, in the wild, environments. The quality of these facial images is affected by various internal factors, including the quality of sensors used in outdoor cameras as well as external ones, such as the quality and direction of light. These factors adversely affect the overall quality of the captured images often causing blurring and/or low resolution, a phenomena commonly referred to as image degradation. Super-resolution algorithms are highly effective in improving the resolution of degraded images, more so if the captured face is small requiring scaling up. With this motivation, this research aims at demonstrating the effect of one of the state-of-the-art image super-resolution algorithms on the labeled faces in the wild (lfw) dataset. In this regard, several cases are analyzed to demonstrate the effectiveness of the super-resolution algorithm. Each case is then investigated independently comparing the order of execution before or after the 3D face alignment step. Following this, resulting images are tested on a closed set face recognition protocol using unsupervised algorithms with high-dimensional extracted features. The inclusion of super-resolution resulted in improvement in the recognition rate compared to unsupervised algorithm results reported in the literature.     

特定人脸的3D模型生成与应用的研究

计算机3D人脸对象的应用范围十分广泛,包括虚拟现实、辅助教学、远程会议、人机交互、游戏娱乐、电影制作等诸多方面。提出了利用区域差值法的算法实现基于两幅人脸图像生成具有特定人特征的3D人脸模型,并在生成的特定人3D人脸模型的基础上实现了基于MPEG 4的特定人的人脸动画功能,实验取得了良好的效果。     

Robust facial landmarking for registration

Finding landmark positions on facial images is an important step in face registration and normalization, for both 2D and 3D face recognition. In this paper, we inspect shortcomings of existing approaches in the literature and compare several methods for performing automatic landmarking on near-frontal faces in different scales. Two novel methods have been employed to analyze facial features in coarse and fine scales successively. The first method uses a mixture of factor analyzers to learn Gabor filter outputs on a coarse scale. The second method is a template matching of block-based Discrete Cosine Transform (DCT) features. In addition, a structural analysis subsystem is proposed that can determine false matches, and correct their positions.     

Research on a face recognition algorithm based on 3D face data and 2D face image matching

Under the condition of weak light or no light, the recognition accuracy of the mature 2D face recognition technology decreases sharply. In this paper, a face recognition algorithm based on the matching of 3D face data and 2D face images is proposed. Firstly, 3D face data is reconstructed from the 2D face in the database based on the 3DMM algorithm, and the face depth image is obtained through orthogonal projection. Then, the average curvature map of the face depth image is used to enhance the data of the depth image. Finally, an improved residual neural network based on the depth image and curvature is designed to compare the scanned face with the face in the database. The method proposed in this paper is tested on the 3D face data in three public face datasets (Texas 3DFRD, FRGC v2.0, and Lock3DFace), and the recognition accuracy is 84.25%, 83.39%, and 78.24%, respectively.