Hallucinating face by position-patch

A novel face hallucination method is proposed in this paper for the reconstruction of a high-resolution face image from a low-resolution observation based on a set of high- and low-resolution training image pairs. Different from most of the established methods based on probabilistic or manifold learning models, the proposed method hallucinates the high-resolution image patch using the same position image patches of each training image. The optimal weights of the training image position-patches are estimated and the hallucinated patches are reconstructed using the same weights. The final high-resolution facial image is formed by integrating the hallucinated patches. The necessity of two-step framework or residue compensation and the differences between hallucination based on patch and global image are discussed. Experiments show that the proposed method without residue compensation generates higher-quality images and costs less computational time than some recent face image super-resolution (hallucination) techniques.     

Face hallucination based on two-dimensional joint learning

In this paper, a face hallucination method based on two-dimensional joint learning is presented. Unlike the existing works on face super-resolution algorithms that first reshape the image or image patch into 1D vector, in our study the spatial construction of the high resolution (HR) and the low resolution (LR) face image are efficiently maintained in the reconstruction procedure. Enlightened by the 1D joint learning approach for image super-resolution, we propose a 2D joint learning algorithm to map the original 2D LR and HR image patch spaces onto a unified feature subspace. Subsequently, the neighbor-embedding (NE) based super-resolution algorithm can be conducted on the unified feature subspace to estimate the reconstruction weights. With these weights, the initial HR facial image can be generated. To refine further the initial HR estimate, the global reconstruction constraint is exploited to improve the quality of reconstruction result. Experiments on the face databases and real-world face images demonstrate the effectiveness of the proposed algorithm.     

基于组合学习的人脸超分辨率算法

现有的基于深度学习的人脸超分辨算法大部分仅仅利用一种网络分区重建高分辨率输出图像,并未考虑人脸图像中的结构性信息,导致了在人脸的重要器官重建上缺乏足够的细节信息。针对这一问题,提出一种基于组合学习的人脸超分辨率算法。该算法独立采用不同深度学习模型的优势重建感兴趣的区域,由此在训练网络的过程中每个人脸区域的数据分布不同,不同的子网络能够获得更精确的先验信息。首先,对人脸图像采用超像素分割算法生成人脸组件部分和人脸背景图像;然后,采用人脸组件生成对抗网络（C-GAN）独立重建人脸组件图像块,并采用人脸背景重建网络生成人脸背景图像;其次,使用人脸组件融合网络将两种不同模型重建的人脸组件图像块自适应融合;最后,将生成的人脸组件图像块合并至人脸背景图像中,重建出最终的人脸图像。在FEI数据集上的实验结果表明,与人脸图像超分辨率算法通过组件生成和增强学习幻构人脸图像（LCGE）及判决性增强的生成对抗网络（EDGAN）相比,所提算法的峰值信噪比（PSNR）值分别高出1.23 dB和1.11 dB。所提算法能够采用不同深度学习模型的优势组合学习重建更精准的人脸图像,同时拓展了图像重建先验的来源。     

Face Hallucination: Theory and Practice

In this paper, we study face hallucination, or synthesizing a high-resolution face image from an input low-resolution image, with the help of a large collection of other high-resolution face images. Our theoretical contribution is a two-step statistical modeling approach that integrates both a global parametric model and a local nonparametric model. At the first step, we derive a global linear model to learn the relationship between the high-resolution face images and their smoothed and down-sampled lower resolution ones. At the second step, we model the residue between an original high-resolution image and the reconstructed high-resolution image after applying the learned linear model by a patch-based non-parametric Markov network to capture the high-frequency content. By integrating both global and local models, we can generate photorealistic face images. A practical contribution is a robust warping algorithm to align the low-resolution face images to obtain good hallucination results. The effectiveness of our approach is demonstrated by extensive experiments generating high-quality hallucinated face images from low-resolution input with no manual alignment.     

Steerable pyramid-based face hallucination

In this paper we propose a robust learning-based face hallucination algorithm, which predicts a high-resolution face image from an input low-resolution image. It can be utilized for many computer vision tasks, such as face recognition and face tracking. With the help of a database of other high-resolution face images, we use a steerable pyramid to extract multi-orientation and multi-scale information of local low-level facial features both from the input low-resolution face image and other high-resolution ones, and use a pyramid-like parent structure and local best match approach to estimate the best prior; then, this prior is incorporated into a Bayesian maximum a posterior (MAP) framework, and finally the high-resolution version is optimized by a steepest decent algorithm. The experimental results show that we can enhance a 24×32 face image into a 96×128 one while the visual effect is relatively good.     

Face hallucination with imprecise-alignment using iterative sparse representation

Existing face hallucination methods assume that the face images are well-aligned. However, in practice, given a low-resolution face image, it is very difficult to perform precise alignment. As a result, the quality of the super-resolved image is degraded dramatically. In this paper, we propose a near frontal-view face hallucination method which is robust to face image mis-alignment. Based on the discriminative nature of sparse representation, we propose a global face sparse representation model that can reconstruct images with mis-alignment variations. We further propose an iterative method combining the global sparse representation and the local linear regression using the Expectation Maximization (EM) algorithm, in which the face hallucination is converted into a parameter estimation problem with incomplete data. Since the proposed algorithm is independent of the face similarity resulting from precise alignment, the proposed algorithm is robust to mis-alignment. In addition, the proposed iterative manner not only combines the merits of the global and local face hallucination, but also provides a convenient way to integrate different strategies to handle the mis-alignment problem. Experimental results show that the proposed method achieves better performance than existing methods, especially for mis-aligned face images.     

Parts-based face super-resolution via non-negative matrix factorization

Face super-resolution refers to inferring the high-resolution face image from its low-resolution one. In this paper, we propose a parts-based face hallucination framework which consists of global face reconstruction and residue compensation. In the first phase, correlation-constrained non-negative matrix factorization (CCNMF) algorithm combines non-negative matrix factorization and canonical correlation analysis to hallucinate the global high-resolution face. In the second phase, the High-dimensional Coupled NMF (HCNMF) algorithm is used to compensate the error residue in hallucinated images. The proposed CCNMF algorithm can generate global face more similar to the ground truth face by learning a parts-based local representation of facial images; while the HCNMF can learn the relation between high-resolution residue and low-resolution residue to better preserve high frequency details. The experimental results validate the effectiveness of our method.     

Hallucinating Compressed Face Images

A face hallucination algorithm is proposed to generate high-resolution images from JPEG compressed low-resolution inputs by decomposing a deblocked face image into structural regions such as facial components and non-structural regions like the background. For structural regions, landmarks are used to retrieve adequate high-resolution component exemplars in a large dataset based on the estimated head pose and illumination condition. For non-structural regions, an efficient generic super resolution algorithm is applied to generate high-resolution counterparts. Two sets of gradient maps extracted from these two regions are combined to guide an optimization process of generating the hallucination image. Numerous experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art hallucination methods on JPEG compressed face images with different poses, expressions, and illumination conditions.     

An adaptively weighted sub-pattern locality preserving projection for face recognition

In this paper, an adaptively weighted sub-pattern locality preserving projection (Aw-SpLPP) algorithm is proposed for face recognition. Unlike the traditional LPP algorithm which operates directly on the whole face image patterns and obtains a global face features that best detects the essential face manifold structure, the proposed Aw-SpLPP method operates on sub-patterns partitioned from an original whole face image and separately extracts corresponding local sub-features from them. Furthermore, the contribution of each sub-pattern can be adaptively computed by Aw-SpLPP in order to enhance the robustness to facial pose, expression and illumination variations. The efficiency of the proposed algorithm is demonstrated by extensive experiments on three standard face databases (Yale, YaleB and PIE). Experimental results show that Aw-SpLPP outperforms other holistic and sub-pattern based methods.     

Nuclear norm regularized coding with local position-patch and nonlocal similarity for face hallucination

Learning-based face hallucination methods have received much attention and progress in past few decades. Specially, position-patch based approaches have been proposed to replace the probabilistic graph-based or manifold learning-based ones. As opposed to the existing patch based methods, where the input image patch matrix is converted into vectors before combination coefficients calculation, in this paper, we propose to directly use the image matrix based regression model for combination coefficients computation to preserve the essential structural information of the input patch matrix. For each input low-resolution (LR) patch matrix, its combination coefficients over the training image patch matrices at the same position can be computed. Then the corresponding high-resolution (HR) patch matrix can be obtained with the LR training patches replaced by the corresponding HR ones. The nonlocal self-similarities are finally utilized to further improve the hallucination performance. Various experimental results on standard face databases indicate that our proposed method outperforms some state-of-the-art algorithms in terms of both visual quantity and objective metrics.     

基于POEM_SLPP的人脸识别算法

针对方向边缘幅值模式(Patterns of Oriented Edge Magnitudes,POEM)提取的人脸特征维数过高和计算复杂度较大的问题,提出了结合方向边缘幅值模式和有监督的局部保持投影(Patterns of Oriented Edge Magnitudes _Supervised Locality Preserving Projections,POEM_SLPP)的人脸识别算法。首先,采用POEM算子进行特征提取;其次,将高维特征数据投影到SLPP算法求出的低维样本空间进行降维;最后,采用最近邻法对测试样本进行分类。在CAS-PEAL-R1人脸库上的实验结果表明,在姿态、背景、修饰、年龄、距离测试集上,该算法的平均识别率较POEM LPP算法提高了22%,较POEM PCA提高了2%。     

Global consistency,local sparsity and pixel correlation: A unified framework for face hallucination

In this paper, a novel two-phase framework is presented to deal with the face hallucination problem. In the first phase, an initial high-resolution (HR) face image is produced in patch-wise. Each input low-resolution (LR) patch is represented as a linear combination of training patches and the corresponding HR patch is estimated by the same combination coefficients. Realizing that training patches similar with the input may provide more appropriate textures in the reconstruction, we regularize the combination coefficients by a weighted _?2-norm${?}_2\mathrm{-}\mathrm{norm}$ minimization term which enlarges the coefficients for relevant patches. The HR face image is then initialized by integrating all the HR patches. In the second phase, three regularization models are introduced to produce the final HR face image. Different from most previous approaches which consider global and local priors separately, the proposed algorithm incorporates the global reconstruction model, the local sparsity model and the pixel correlation model into a unified regularization framework. Initializing the regularization problem with the HR image obtained in the first phase, the final output HR image can be optimized through an iterative procedure. Experimental results show that the proposed algorithm achieves better performances in both reconstruction error and visual quality.     

无参数无相关最大化判别边界算法

在人脸识别算法中,无参数局部保持投影(PFLPP)是一种有效的特征提取算法, 但忽略了异类近邻样本在分类中所起的作用,并且对于近邻的处理仅利用样本与总体均值的 距离关系来判断,因此并不能有效地确定近邻关系。基于此,提出一种无参数无相关最大化 判别边界算法,有效地利用了样本的类别信息,定义了无参数同类近邻样本的相似权值与异 类近邻样本的惩罚权值,样本邻域大小可根据类内平均余弦距离和类间余弦距离自适应确定, 为了进一步增强算法的性能,给出了具有不相关性的目标函数。UMIST 和 AR 人脸库上的实 验结果表明,该算法相对于不相关保局投影分析算法和 PFLPP 算法,具有运算量低、识别性 能高的优势。     

基于人脸运动信息和改进保局投影的疲劳识别

提出了一种基于人脸运动信息和改进保局投影的疲劳识别方法。利用光流技术计算人脸皮层的运动速度,并以此作为疲劳特征;为了有效地进行疲劳特征降维,提出了改进的保局投影方法,该方法很好地保留了数据的局部流形结构和全局结构;采用加权k近部的方法进行疲劳识别。实验结果表明该方法具有很好的识别效果。     

Multi-resolution feature fusion for face recognition

For face recognition, image features are first extracted and then matched to those features in a gallery set. The amount of information and the effectiveness of the features used will determine the recognition performance. In this paper, we propose a novel face recognition approach using information about face images at higher and lower resolutions so as to enhance the information content of the features that are extracted and combined at different resolutions. As the features from different resolutions should closely correlate with each other, we employ the cascaded generalized canonical correlation analysis (GCCA) to fuse the information to form a single feature vector for face recognition. To improve the performance and efficiency, we also employ “Gabor-feature hallucination”, which predicts the high-resolution (HR) Gabor features from the Gabor features of a face image directly by local linear regression. We also extend the algorithm to low-resolution (LR) face recognition, in which the medium-resolution (MR) and HR Gabor features of a LR input image are estimated directly. The LR Gabor features and the predicted MR and HR Gabor features are then fused using GCCA for LR face recognition. Our algorithm can avoid having to perform the interpolation/super-resolution of face images and having to extract HR Gabor features. Experimental results show that the proposed methods have a superior recognition rate and are more efficient than traditional methods.     

基于Gabor变换和Zernike矩的人脸特征提取方法

针对目前常用的三种人脸特征提取方法中存在的识别率低、抗噪性较弱的问题,提出一种基于Gabor变换和Zernike矩的人脸特征提取方法.该方法首先对人脸进行多分辨的Gabor变换,然后利用Zernike矩获得具有平移、尺度、旋转不变性的特征,并用线性判别分析(LDA)方法进一步进行特征选择,最后采用K最近邻分类方法进行人脸的识别.实验结果表明,在与常用的三种人脸特征提取方法的比较中,该方法具有更高的识别率和更强的抗噪性能.     

A novel kernel-based framework for facial-image hallucination

In this paper, we present a kernel-based eigentransformation framework to hallucinate the high-resolution (HR) facial image of a low-resolution (LR) input. The eigentransformation method is a linear subspace approach, which represents an image as a linear combination of training samples. Consequently, those novel facial appearances not included in the training samples cannot be super-resolved properly. To solve this problem, we devise a kernel-based extension of the eigentransformation method, which takes higher-order statistics of the image data into account. To generate HR face images with higher fidelity, the HR face image reconstructed using this kernel-based eigentransformation method is treated as an initial estimation of the target HR face. The corresponding high-frequency components of this estimation are extracted to form a prior in the maximum a posteriori (MAP) formulation of the SR problem so as to derive the final reconstruction result. We have evaluated our proposed method using different kernels and configurations, and have compared these performances with some current SR algorithms. Experimental results show that our kernel-based framework, along with a proper kernel, can produce good HR facial images in terms of both visual quality and reconstruction errors.     

基于独立分量分析的人脸超分辨率重建

提出一种基于独立分量分析(ICA)的人脸超分辨率重建算法。该算法利用ICA从高分辨率训练图像中提取出独立分量,并对ICA系数进行先验估计。对于给定的低分辨率图像,结合最大后验概率估计求出ICA系数,进行ICA反变换得到高分辨率图像的近似估计,并利用局部结构张量对图像进行精化处理得到重建图像。仿真结果表明,该算法在实现人脸超分辨率重建的同时保持了人脸整体结构特征,且对光照、表情、姿态等具有一定的鲁棒性,将重建结果用于人脸辨识,有效提高了辨识效率。     

Face hallucination based on sparse local-pixel structure

In this paper, we propose a face-hallucination method, namely face hallucination based on sparse local-pixel structure. In our framework, a high resolution (HR) face is estimated from a single frame low resolution (LR) face with the help of the facial dataset. Unlike many existing face-hallucination methods such as the from local-pixel structure to global image super-resolution method (LPS-GIS) and the super-resolution through neighbor embedding, where the prior models are learned by employing the least-square methods, our framework aims to shape the prior model using sparse representation. Then this learned prior model is employed to guide the reconstruction process. Experiments show that our framework is very flexible, and achieves a competitive or even superior performance in terms of both reconstruction error and visual quality. Our method still exhibits an impressive ability to generate plausible HR facial images based on their sparse local structures.     

完备鉴别保局投影人脸识别算法

为了充分利用保局总体散布主元空间内的鉴别信息进行人脸识别,提出了一种完备鉴别保局投影(complete discriminant locality preserving projections,简称CDLPP)人脸识别算法.鉴于Fisher鉴别分析和保局投影已经被广泛的应用于人脸识别,完备鉴别保局投影(locality preserving projections,简称LPP)算法将这两者结合起来,分析了保局类内散布、类间散布和总体散布的主元空间和零空间内包含的鉴别信息.该算法采用奇异值分解(singular value decomposition,简称SVD),去除了不含任何鉴别信息的保局总体散布的零空间;分别在保局类内散布的主元空间和零空间提取规则鉴别特征和不规则鉴别特征;用串联的方式在特征层融合规则鉴别特征和不规则鉴别特征形成完备的鉴别特征进行人脸识别.在ORL库、FERET子库和PIE子库上的大量识别实验充分表明了完备鉴别保局投影算法的性能优于线性鉴别分析、保局投影和鉴别保局投影等现有的子空间人脸识别算法,验证了算法的有 效性.