Real-Time Pose-Invariant Face Recognition Using the Efficient Second-Order Minimization and the Pose Transforming Matrix

We propose a real-time pose-invariant face recognition algorithm from a gallery of frontal images only. (i) We modified the second-order minimization method for the active appearance model (AAM). This allows the AAM to have the ability of correct convergence with little loss of frame rate. (ii) We proposed a pose transforming matrix that can eliminate warping artifacts of the warped face image from AAM fitting. This makes it possible to train a neural network as the face recognizer with one frontal face image of each person in the gallery set. (iii) We propose a simple method for pose recognition by using neural networks to select the proper pose transforming matrix. The proposed algorithm was evaluated on a set of 2000 facial images of 10 people (200 images for each person obtained at various poses), achieving a great improvement in recognition.     

Pose-invariant face recognition using a 3D deformable model

The paper proposes a novel, pose-invariant face recognition system based on a deformable, generic 3D face model, that is a composite of: (1) an edge model, (2) a color region model and (3) a wireframe model for jointly describing the shape and important features of the face. The first two submodels are used for image analysis and the third mainly for face synthesis. In order to match the model to face images in arbitrary poses, the 3D model can be projected onto different 2D viewplanes based on rotation, translation and scale parameters, thereby generating multiple face-image templates (in different sizes and orientations). Face shape variations among people are taken into account by the deformation parameters of the model. Given an unknown face, its pose is estimated by model matching and the system synthesizes face images of known subjects in the same pose. The face is then classified as the subject whose synthesized image is most similar. The synthesized images are generated using a 3D face representation scheme which encodes the 3D shape and texture characteristics of the faces. This face representation is automatically derived from training face images of the subject. Experimental results show that the method is capable of determining pose and recognizing faces accurately over a wide range of poses and with naturally varying lighting conditions. Recognition rates of 92.3% have been achieved by the method with 10 training face images per person.     

Face recognition across pose: A review

One of the major challenges encountered by current face recognition techniques lies in the difficulties of handling varying poses, i.e., recognition of faces in arbitrary in-depth rotations. The face image differences caused by rotations are often larger than the inter-person differences used in distinguishing identities. Face recognition across pose, on the other hand, has great potentials in many applications dealing with uncooperative subjects, in which the full power of face recognition being a passive biometric technique can be implemented and utilised. Extensive efforts have been put into the research toward pose-invariant face recognition in recent years and many prominent approaches have been proposed. However, several issues in face recognition across pose still remain open, such as lack of understanding about subspaces of pose variant images, problem intractability in 3D face modelling, complex face surface reflection mechanism, etc. This paper provides a critical survey of researches on image-based face recognition across pose. The existing techniques are comprehensively reviewed and discussed. They are classified into different categories according to their methodologies in handling pose variations. Their strategies, advantages/disadvantages and performances are elaborated. By generalising different tactics in handling pose variations and evaluating their performances, several promising directions for future research have been suggested.     

Optimal Local Basis: A Reinforcement Learning Approach for Face Recognition

This paper presents a novel learning approach for Face Recognition by introducing Optimal Local Basis. Optimal local bases are a set of basis derived by reinforcement learning to represent the face space locally. The reinforcement signal is designed to be correlated to the recognition accuracy. The optimal local bases are derived then by finding the most discriminant features for different parts of the face space, which represents either different individuals or different expressions, orientations, poses, illuminations, and other variants of the same individual. Therefore, unlike most of the existing approaches that solve the recognition problem by using a single basis for all individuals, our proposed method benefits from local information by incorporating different bases for its decision. We also introduce a novel classification scheme that uses reinforcement signal to build a similarity measure in a non-metric space. Experiments on AR, PIE, ORL and YALE databases indicate that the proposed method facilitates robust face recognition under pose, illumination and expression variations. The performance of our method is compared with that of Eigenface, Fisherface, Subclass Discriminant Analysis, and Random Subspace LDA methods as well.     

结合形变模型与图像修复的人脸姿态矫正

目的 人脸姿态偏转是影响人脸识别准确率的一个重要因素,本文利用3维人脸重建中常用的3维形变模型以及深度卷积神经网络,提出一种用于多姿态人脸识别的人脸姿态矫正算法,在一定程度上提高了大姿态下人脸识别的准确率。方法 对传统的3维形变模型拟合方法进行改进,利用人脸形状参数和表情参数对3维形变模型进行建模,针对面部不同区域的关键点赋予不同的权值,加权拟合3维形变模型,使得具有不同姿态和面部表情的人脸图像拟合效果更好。然后,对3维人脸模型进行姿态矫正并利用深度学习对人脸图像进行修复,修复不规则的人脸空洞区域,并使用最新的局部卷积技术同时在新的数据集上重新训练卷积神经网络,使得网络参数达到最优。结果 在LFW（labeled faces in the wild）人脸数据库和StirlingESRC（Economic Social Research Council）3维人脸数据库上,将本文算法与其他方法进行比较,实验结果表明,本文算法的人脸识别精度有一定程度的提高。在LFW数据库上,通过对具有任意姿态的人脸图像进行姿态矫正和修复后,本文方法达到了96.57%的人脸识别精确度。在StirlingESRC数据库上,本文方法在人脸姿态为±22°的情况下,人脸识别准确率分别提高5.195%和2.265%;在人脸姿态为±45°情况下,人脸识别准确率分别提高5.875%和11.095%;平均人脸识别率分别提高5.53%和7.13%。对比实验结果表明,本文提出的人脸姿态矫正算法有效提高了人脸识别的准确率。结论 本文提出的人脸姿态矫正算法,综合了3维形变模型和深度学习模型的优点,在各个人脸姿态角度下,均能使人脸识别准确率在一定程度上有所提高。     

Tied factors analysis for high-dimensional image feature extraction and recognition application

Feature extraction from images, which are typical of high dimensionality, is crucial to the recognition performance. To explore the discriminative information while depressing the intra-class variations due to variable illumination and view conditions, we propose a factor analysis framework for separate “content” from “style,” identifying a familiar face seen under unfamiliar viewing conditions, classifying familiar poses presented in an unfamiliar face, estimating age across unfamiliar faces. The framework applies efficient algorithms derived from objective factor separating functions and space mapping functions, which can produce sufficiently expressive representations of feature extraction and dimensionality reduction. We report promising results on three different tasks in the high-dimensional image perceptual domains: face identification with two benchmark face databases, facial pose classification with a benchmark facial pose database, extrapolation of age to unseen facial image. Experimental results show that our approach produced higher classification performance when compared to classical LDA, WLDA, LPP, MFA, and DLA algorithms.     

Harmonic 1-form based skeleton extraction from examples

This paper presents a method to extract skeletons using examples. Our method is based on the observation that many deformations in real-world applications are isometric or near isometric. By taking advantage of the intrinsic property of harmonic 1-form, i.e., it is determined by the metric and independent of the resolution and embedding, our method can easily find a consistent mapping between the reference and example poses which can be in different resolutions and triangulations. We first construct the skeleton-like Reeb graph of a harmonic function defined on the given poses. Then by examining the changes of mean curvatures, we identify the initial locations of joints. Finally we refine the joint locations by solving a constrained optimization problem. We demonstrate the efficacy of the proposed framework by pose space deformation, skeleton transfer, shape segmentation and pose-invariant shape signature.     

Unseen head pose prediction using dense multivariate label distribution

Accurate head poses are useful for many face-related tasks such as face recognition, gaze estimation, and emotion analysis. Most existing methods estimate head poses that are included in the training data (i.e., previously seen head poses). To predict head poses that are not seen in the training data, some regression-based methods have been proposed. However, they focus on estimating continuous head pose angles, and thus do not systematically evaluate the performance on predicting unseen head poses. In this paper, we use a dense multivariate label distribution (MLD) to represent the pose angle of a face image. By incorporating both seen and unseen pose angles into MLD, the head pose predictor can estimate unseen head poses with an accuracy comparable to that of estimating seen head poses. On the Pointing’04 database, the mean absolute errors of results for yaw and pitch are 4.01° and 2.13°, respectively. In addition, experiments on the CAS-PEAL and CMU Multi-PIE databases show that the proposed dense MLD-based head pose estimation method can obtain the state-of-the-art performance when compared to some existing methods.     

A volumetric/iconic frequency domain representation for objectswith application for pose invariant face recognition

A novel method for representing 3D objects that unifies viewer and model centered object representations is presented. A unified 3D frequency-domain representation, called volumetric frequency representation (VFR), encapsulates both the spatial structure of the object and a continuum of its views in the same data structure. The frequency-domain image of an object viewed from any direction can be directly extracted employing an extension of the projection slice theorem, where each Fourier-transformed view is a planar slice of the volumetric frequency representation. The VFR is employed for pose-invariant recognition of complex objects, such as faces. The recognition and pose estimation is based on an efficient matching algorithm in a four-dimensional Fourier space. Experimental examples of pose estimation and recognition of faces in various poses are also presented     

Coupled Action Recognition and Pose Estimation from Multiple Views

Action recognition and pose estimation are two closely related topics in understanding human body movements; information from one task can be leveraged to assist the other, yet the two are often treated separately. We present here a framework for coupled action recognition and pose estimation by formulating pose estimation as an optimization over a set of action-specific manifolds. The framework allows for integration of a 2D appearance-based action recognition system as a prior for 3D pose estimation and for refinement of the action labels using relational pose features based on the extracted 3D poses. Our experiments show that our pose estimation system is able to estimate body poses with high degrees of freedom using very few particles and can achieve state-of-the-art results on the HumanEva-II benchmark. We also thoroughly investigate the impact of pose estimation and action recognition accuracy on each other on the challenging TUM kitchen dataset. We demonstrate not only the feasibility of using extracted 3D poses for action recognition, but also improved performance in comparison to action recognition using low-level appearance features.     

Orthogonal discriminant vector for face recognition across pose

Recognizing face images across pose is one of the challenging tasks for reliable face recognition. This paper presents a new method to tackle this challenge based on orthogonal discriminant vector (ODV). The result of our theoretical analysis shows that an individual’s probe image captured with a new pose can be represented by a linear combination of his/her gallery images. Based on this observation, in contrast to the conventional methods which model face images of different individuals on a single manifold, we propose to model face images of different individuals on different linear manifolds. The contribution of our approach includes: (1) to prove that the orthogonality to ODVs is a pose-invariant feature.; (2) to categorize each person with a set of ODVs, where his/her face images posses zero projections while other persons’ images are characterized by maximum projections; (3) to define a metric to measure the distance between a face image and an ODV, and classify the face images based on this metric. Our experimental results validate the feasibility of modeling the face images of different individuals on different linear manifolds. The proposed method achieves higher accuracy on face recognition and verification than the existing techniques.     

基于单视图的多姿态人脸识别算法

针对基于多视图的多姿态人脸识别方法的缺陷，即需要对每个人脸拍摄多个视图为前提条件，提出了基于单视图的多姿态人脸识别技术，首先基于二元高次多项式函数最小二乘拟合方法由单视图通过变形生成多姿态人脸图像，然后基于该单视图和生成的多姿态图像进行多姿态人脸识别。实验结果表明该文算法识别的正确率远高于经典算法。     

基于因子分析与稀疏表示的多姿态人脸识别

在非可控环境下,人脸识别面临的最大难题之一是姿态变化与遮挡问题。基于稀疏表示的人脸识别方法将测试人脸表示成训练人脸的稀疏线性组合,根据其组合系数的稀疏性进行人脸识别。该方法对人脸的噪声和遮挡变化具有很好的鲁棒性,但对人脸的姿态变化表现力极差,这是因为当人脸具有姿态变化时,同一个人不同姿态情况下很难对应起来,这违背线性组合的前提条件。为了克服稀疏表示方法对人脸姿态变化表现力极差问题,对人脸进行因子分析,分离出人脸姿态因子,得到合成的正面人脸;利用稀疏表示进行人脸分类识别。实验结果表明,该方法对人脸的遮挡和姿态变化具有很好的鲁棒性。     

基于深度网络的多形态人脸识别

在实际的自动人脸识别系统中,输入的识别图像往往在表情、分辨率大小以及姿态方面呈现出多种变化。现在很多方法尝试通过线性或局部线性的映射来寻找由这些变化共享的统一的特征空间。利用由受限玻尔兹曼机(RBM)堆叠成的深度神经网络来发掘这些变化内在的非线性表达。深度网络能够学习高维数据到低维数据的映射关系,并且有助于提高图像分类和识别的性能。同时,为了实现在一个统一的深度框架下同时进行特征提取和识别,在网络的顶层增加了一个监督的回归层。在预训练阶段,通过训练集中不同姿态、不同表情以及不同分辨率的图像对网络进行初始化。在微调阶段,通过网络的输出与标签之间的差 并利用标准反向传播的方法 对模型的参数空间进行调整。在测试阶段,从测试库中随机选择一幅图像,获得统一空间下的特征向量。通过与参考图像库中的所有特征向量进行对比,利用最近邻域的方法识别人脸身份。在具有丰富表情以及大姿态变化的CMU-PIE人脸数据库上进行了全面的实验,结果表明,提出的方法取得了比最新的局域线性映射(或局部线性)的人脸识别方法更高的识别率。     

Capitalize on dimensionality increasing techniques for improving Face Recognition Grand Challenge performance

This paper presents a novel pattern recognition framework by capitalizing on dimensionality increasing techniques. In particular, the framework integrates Gabor image representation, a novel multiclass Kernel Fisher Analysis (KFA) method, and fractional power polynomial models for improving pattern recognition performance. Gabor image representation, which increases dimensionality by incorporating Gabor filters with different scales and orientations, is characterized by spatial frequency, spatial locality, and orientational selectivity for coping with image variabilities such as illumination variations. The KFA method first performs nonlinear mapping from the input space to a high-dimensional feature space, and then implements the multiclass Fisher discriminant analysis in the feature space. The significance of the nonlinear mapping is that it increases the discriminating power of the KFA method, which is linear in the feature space but nonlinear in the input space. The novelty of the KFA method comes from the fact that 1) it extends the two-class kernel Fisher methods by addressing multiclass pattern classification problems and 2) it improves upon the traditional Generalized Discriminant Analysis (GDA) method by deriving a unique solution (compared to the GDA solution, which is not unique). The fractional power polynomial models further improve performance of the proposed pattern recognition framework. Experiments on face recognition using both the FERET database and the FRGC (Face Recognition Grand Challenge) databases show the feasibility of the proposed framework. In particular, experimental results using the FERET database show that the KFA method performs better than the GDA method and the fractional power polynomial models help both the KFA method and the GDA method improve their face recognition performance. Experimental results using the FRGC databases show that the proposed pattern recognition framework improves face recognition performance upon the BEE baseline algorithm and the LDA-based baseline algorithm by large margins.     

From few to many: illumination cone models for face recognitionunder variable lighting and pose

We present a generative appearance-based method for recognizing human faces under variation in lighting and viewpoint. Our method exploits the fact that the set of images of an object in fixed pose, but under all possible illumination conditions, is a convex cone in the space of images. Using a small number of training images of each face taken with different lighting directions, the shape and albedo of the face can be reconstructed. In turn, this reconstruction serves as a generative model that can be used to render (or synthesize) images of the face under novel poses and illumination conditions. The pose space is then sampled and, for each pose, the corresponding illumination cone is approximated by a low-dimensional linear subspace whose basis vectors are estimated using the generative model. Our recognition algorithm assigns to a test image the identity of the closest approximated illumination cone. Test results show that the method performs almost without error, except on the most extreme lighting directions     

2D face fitting-assisted 3D face reconstruction for pose-robust face recognition

Recent face recognition algorithm can achieve high accuracy when the tested face samples are frontal. However, when the face pose changes largely, the performance of existing methods drop drastically. Efforts on pose-robust face recognition are highly desirable, especially when each face class has only one frontal training sample. In this study, we propose a 2D face fitting-assisted 3D face reconstruction algorithm that aims at recognizing faces of different poses when each face class has only one frontal training sample. For each frontal training sample, a 3D face is reconstructed by optimizing the parameters of 3D morphable model (3DMM). By rotating the reconstructed 3D face to different views, pose virtual face images are generated to enlarge the training set of face recognition. Different from the conventional 3D face reconstruction methods, the proposed algorithm utilizes automatic 2D face fitting to assist 3D face reconstruction. We automatically locate 88 sparse points of the frontal face by 2D face-fitting algorithm. Such 2D face-fitting algorithm is so-called Random Forest Embedded Active Shape Model, which embeds random forest learning into the framework of Active Shape Model. Results of 2D face fitting are added to the 3D face reconstruction objective function as shape constraints. The optimization objective energy function takes not only image intensity, but also 2D fitting results into account. Shape and texture parameters of 3DMM are thus estimated by fitting the 3DMM to the 2D frontal face sample, which is a non-linear optimization problem. We experiment the proposed method on the publicly available CMUPIE database, which includes faces viewed from 11 different poses, and the results show that the proposed method is effective and the face recognition results toward pose variants are promising.     

Kernel-based nonlinear discriminant analysis for face recognition

Linear subspace analysis methods have been successfully applied to extract features for face recognition.But they are inadequate to represent the complex and nonlinear variations of real face images,such as illumination,facial expression and pose variations,because of their linear properties.In this paper,a nonlinear subspace analysis method,Kernel-based Nonlinear Discriminant Analysis (KNDA),is presented for face recognition,which combines the nonlinear kernel trick with the linear subspace analysis method-Fisher Linear Discriminant Analysis (FLDA).First,the kernel trick is used to project the input data into an implicit feature space,then FLDA is performed in this feature space.Thus nonlinear discriminant features of the input data are yielded.In addition,in order to reduce the computational complexity,a geometry-based feature vectors selection scheme is adopted.Another similar nonlinear subspace analysis is Kernel-based Principal Component Analysis (KPCA),which combines the kernel trick with linear Principal Component Analysis (PCA).Experiments are performed with the polynomial kernel,and KNDA is compared with KPCA and FLDA.Extensive experimental results show that KNDA can give a higher recognition rate than KPCA and FLDA.