An example-based face hallucination method for single-frame, low-resolution facial images

This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face models. First, we propose a recursive error back-projection method to compensate for residual errors, and a region-based reconstruction method to preserve characteristics of local facial regions. Then, we define an extended morphable face model, in which an extended face is composed of the interpolated high-resolution face from a given low-resolution face, and its original high-resolution equivalent. Then, the extended face is separated into an extended shape and an extended texture. We performed various hallucination experiments using the MPI, XM2VTS, and KF databases, compared the reconstruction errors, structural similarity index, and recognition rates, and showed the effects of face detection errors and shape estimation errors. The encouraging results demonstrate that the proposed methods can improve the performance of face recognition systems. Especially the proposed method can enhance the resolution of single-frame, low-resolution facial images.     

A super-resolution-based license plate recognition method for remote surveillance

With the continuous development of deep learning, neural networks have made great progress in license plate recognition (LPR). Nevertheless, there is still room to improve the performance of license plate recognition for low-resolution and relatively blurry images in remote surveillance scenarios. When it is difficult to enhance the recognition algorithm, we choose super-resolution (SR) to improve the quality of license plate images and thereby provide clearer input for the subsequent recognition stage. In this paper, we propose an automatic super-resolution license plate recognition (SRLPR) network which consists of four parts separately: license plate detection, character detection, single character super-resolution, and recognition. In the training stage, firstly, LP detection model needs to be trained alone and then its detection results will be used to successively train the three subsequent modules. During the test phase, for each input image, the network can get its LP number automatically. We also collect an applicable and challenging LPR dataset called SRLP, which is collected from real remote traffic surveillance. The experimental results demonstrate that our method achieves comprehensive quality of SR images and higher recognition accuracy compared with state-of-the-art methods. The SRLP dataset and the code for training and testing SRLPR network are available at https://pan.baidu.com/s/1vnhRa-c-dBj6jlfBZV5w4g.     

Hallucinating face by eigentransformation

In video surveillance, the faces of interest are often of small size. Image resolution is an important factor affecting face recognition by human and computer. In this paper, we propose a new face hallucination method using eigentransformation. Different from most of the proposed methods based on probabilistic models, this method views hallucination as a transformation between different image styles. We use Principal Component Analysis (PCA) to fit the input face image as a linear combination of the low-resolution face images in the training set. The high-resolution image is rendered by replacing the low-resolution training images with high-resolution ones, while retaining the same combination coefficients. Experiments show that the hallucinated face images are not only very helpful for recognition by humans, but also make the automatic recognition procedure easier, since they emphasize the face difference by adding more high-frequency details.     

Reference guided image super-resolution via efficient dense warping and adaptive fusion

Due to the limited improvement of single-image based super-resolution (SR) methods in recent years, the reference based image SR (RefSR) methods, which super-resolve the low-resolution (LR) input with the guidance of similar high-resolution (HR) reference images are emerging. There are two main challenges in RefSR, i.e. reference image warping and exploring the guidance information from the warped references. For reference warping, we propose an efficient dense warping method to deal with large displacements, which is much faster than traditional patch (or texture) matching strategy. For the SR process, since different reference images complement each other, and have different similarities with the LR image, we further propose a similarity based feature fusion strategy to take advantage of the most similar reference regions. The SR process is realized by an encoder–decoder network and trained with pixel-level reconstruction loss, degradation loss and feature-level perceptual loss. Extensive experiments on three benchmark datasets demonstrate that the proposed method outperforms state-of-the-art SR methods in both subjective and objective measurements.     

Classifier shared deep network with multi-hierarchy loss for low resolution face recognition

Face images in real Closed-Circuit Television (CCTV) are usually with low resolution, which remarkably deteriorates the performance of existing face recognition algorithms and hinders the application of face recognition. The main technical focus of this issue, matching between high-resolution (HR) and low-resolution (LR) face images has attracted significant attention. In order to better address this problem, we propose a Classifier Shared Deep Network with Multi-Hierarchy Loss (CS-MHL-Net) for low-resolution face recognition (LRFR) in this paper. Firstly, considering that contrastive loss and its variants are not conducive to the convergence of network and the reduction of discrepancy, a shared classifier between HR and LR face images is proposed to further narrow the domain gap between HR and LR by sharing the corresponding weights which can be seen as the class center. Secondly, to fully exploit intermediate features and loss constraints, we embed multi-hierarchy loss into intermediate layers, with the target of reducing the distances between HR and LR intermediate features after max pooling and avoiding the decreasing of accuracy caused by over-utilization of intermediate features. Experimental results on LFW and SCface demonstrate the effectiveness and superiority of the proposed method.     

Generalized face super-resolution.

Existing learning-based face super-resolution (hallucination) techniques generate high-resolution images of a single facial modality (i.e., at a fixed expression, pose and illumination) given one or set of low-resolution face images as probe. Here, we present a generalized approach based on a hierarchical tensor (multilinear) space representation for hallucinating high-resolution face images across multiple modalities, achieving generalization to variations in expression and pose. In particular, we formulate a unified tensor which can be reduced to two parts: a global image-based tensor for modeling the mappings among different facial modalities, and a local patch-based multiresolution tensor for incorporating high-resolution image details. For realistic hallucination of unregistered low-resolution faces contained in raw images, we develop an automatic face alignment algorithm capable of pixel-wise alignment by iteratively warping the probing face to its projection in the space of training face images. Our experiments show not only performance superiority over existing benchmark face super-resolution techniques on single modal face hallucination, but also novelty of our approach in coping with multimodal hallucination and its robustness in automatic alignment under practical imaging conditions.     

Cross-view information interaction and feedback network for face hallucination

Hallucinating a photo-realistic frontal face image from a low-resolution (LR) non-frontal face image is beneficial for a series of face-related applications. However, previous efforts either focus on super-resolving high-resolution (HR) face images from nearly frontal LR counterparts or frontalizing non-frontal HR faces. It is necessary to address all these challenges jointly for real-world face images in unconstrained environment. In this paper, we develop a novel Cross-view Information Interaction and Feedback Network (CVIFNet), which simultaneously handles the non-frontal LR face image super-resolution (SR) and frontalization in a unified framework and interacts them with each other to further improve their performance. Specifically, the CVIFNet is composed of two feedback sub-networks for frontal and profile face images. Considering the reliable correspondence between frontal and non-frontal face images can be crucial and contribute to face hallucination in a different manner, we design a cross-view information interaction module (CVIM) to aggregate HR representations of different views produced by the SR and frontalization processes to generate finer face hallucination results. Besides, since 3D rendered facial priors contain rich hierarchical features, such as low-level (e.g., sharp edge and illumination) and perception level (e.g., identity) information, we design an identity-preserving consistency loss based on 3D rendered facial priors, which can ensure that the high-frequency details of frontal face hallucination result are consistent with the profile. Extensive experiments demonstrate the effectiveness and advancement of CVIFNet.     

松弛耦合非负矩阵分解的低分辨率人脸识别算法

针对实际监控场景中经常遇到的人脸图像分辨率较低的问题,本文提出了一种利用耦合非负矩阵分解并保持系数松弛的低分辨率人脸识别算法（Relaxed Coupled Nonnegative Matrix Factorization,后文简称RCNMF）。首先,对高低分辨率人脸图像进行非负矩阵矩阵分解（nonnegative matrix factorization,后文简称NMF）,在分解的同时保持组合系数近似一致,从而得到高低分辨率图像的基矩阵。然后,通过低分辨率图像的基矩阵提取训练和测试样本的特征。最后进行识别。实验结果验证了与其他几种基于耦合映射的低分辨率人脸识别方法相比,RCNMF算法的识别性能更好。同时通过实验验证了RCNMF算法的收敛性。      

A Fast Face Recognition Method Based on Fractal Coding

Nowadays, many methods for face recognition are proposed and most of them can obtain good results. However, when these methods are simulated on the platform of the PC, it is hard to apply these methods, especially complex ones to practical devices. This paper uses fractal theory to compress face images and improves the encoding speed with the inherent feature of facial symmetry. To improve the performance of Fractal Neighbor Distance (FND), which is a way of ranging, the degree of similarity between encoded images is defined, and a novel method called Fractal Neighbor Distance-based Classification (FNDC) is presented in this paper. The criterion of FNDC is classifying different samples of the same person as a class. Experimental results on Yale, FERET and CMU PIE databases show the effectiveness of FNDC in face recognition. Then we apply the method to i.MX6 which embeds Android operating system. Actual operating results demonstrated the high efficiency of our method in runtime and correct rate.     

Stochastic super-resolution image reconstruction

The objective of super-resolution (SR) imaging is to reconstruct a single higher-resolution image based on a set of lower-resolution images that were acquired from the same scene to overcome the limitations of image acquisition process for facilitating better visualization and content recognition. In this paper, a stochastic Markov chain Monte Carlo (MCMC) SR image reconstruction approach is proposed. First, a Bayesian inference formulation, which is based on the observed low-resolution images and the prior high-resolution image model, is mathematically derived. Second, to exploit the MCMC sample-generation technique for the stochastic SR image reconstruction, three fundamental issues are observed as follows. First, since the hyperparameter value of the prior image model controls the degree of regularization and intimately affects the quality of the reconstructed high-resolution image, how to determine an optimal hyperparameter value for different low-resolution input images becomes a very challenging task. Rather than exploiting the exhaustive search, an iterative updating approach is developed in this paper by allowing the value of hyperparameter being simultaneously updated in each sample-generation iteration. Second, the samples generated during the so-called burn-in period (measured in terms of the number of samples initially generated) of the MCMC-based sample-generation process are considered unreliable and should be discarded. To determine the length of the burn-in period for each set of low-resolution input images, a time-period bound in closed form is mathematically derived. Third, image artifacts could be incurred in the reconstructed high-resolution image, if the number of samples (counting after the burn-in period) generated by the MCMC-based sample-generation process is insufficient. For that, a variation-sensitive bilateral filter is proposed as a ‘complementary’ post-processing scheme, to improve the reconstructed high-resolution image quality, when the number of samples is insufficient. Extensive simulation results have clearly shown that the proposed stochastic SR image reconstruction method consistently yields superior performance.     

融合半监督降维与稀疏表示的人脸识别方法

由于人脸图像数据的维数都较高,将稀疏表示分类用于人脸识别时计算量很大,为了提高人脸识别系统的效率,提出了一种融合半监督降维和稀疏表示的人脸识别方法。首先利用半监督降维算法对图像进行降维处理,在较低的维数空间快速取得较高的识别率,然后利用稀疏表示分类进行人脸识别,取得比传统的最近邻分类器更高的识别率,最后在ORL人脸库上进行实验验证。结果表明,利用该融合算法可快速有效地提高人脸图像的识别效果。     

分块CS-LBP和加权PCA的低分辨率人脸识别

针对局部二值模式(LBP)特征在低分辨率的人脸图 像上识别率较低的问题,提出了一种基于分块中心对称局部二值模式(CS-LBP,center symmetric local binary pattern)和加权主成分分析(PCA)算法的低分辨率人脸识别算法。 首先利用分块CS-LBP算子提取低分辨率人脸图像的特征;然后利用加权PCA算子对特 征进行降维, 从而得到更强的分类特征;最后利用最近邻分类器选出人脸最优分类类别并计算识别率。在 ORL人脸库上的实验表明,在人脸图像分辨率下降到(12×10)时,本 文算法的识别率仍能达 到85.00%,基本满足了实际运用中对识别率的要求,并且降低了运算 时间。     

Text-image super-resolution through anchored neighborhood regression with multiple class-specific dictionaries

In the dictionary-based image super-resolution (SR) methods, the resolution of the input image is enhanced using a dictionary of low-resolution (LR) and high-resolution (HR) image patches. Typically, a single dictionary is learned from all the patches in the training set. Then, the input LR patch is super-resolved using its nearest LR patches and their corresponding HR patches in the dictionary. In this paper, we propose a text-image SR method using multiple class-specific dictionaries. Each dictionary is learned from the patches of images of a specific character in the training set. The input LR image is segmented into text lines and characters, and the characters are preliminarily classified. Likewise, overlapping patches are extracted from the input LR image. Then, each patch is super-resolved through the anchored neighborhood regression, using n class-specific dictionaries corresponding to the top-n classification results of the character containing the patch. The final HR image is generated by aggregating all the super-resolved patches. Our method achieves significant improvements in visual image quality and OCR accuracy, compared to the related dictionary-based SR methods. This confirms the effectiveness of applying the preliminary character classification results and multiple class-specific dictionaries in text-image SR.     

Multiframe super-resolution based on half-quadratic prior with artifacts suppress

The multiframe super-resolution (SR) technique aims to obtain a high-resolution (HR) image by using a set of observed low-resolution (LR) images. In the reconstruction process, artifacts may be possibly produced due to the noise, especially in presence of stronger noise. In order to suppress artifacts while preserving discontinuities of images, in this paper a multiframe SR method is proposed by involving the reconstruction properties of the half-quadratic prior model together with the quadratic prior model using a convex combination. Moreover, by analyzing local features of the underlined HR image, these two prior models are combined by using an automatically calculated weight function, making both smooth and discontinuous pixels handled properly. A variational Bayesian inference (VBF) based algorithm is designed to efficiently and effectively seek the solution of the proposed method. With the VBF framework, motion parameters and hyper-parameters are all determined automatically, leading to an unsupervised SR method. The efficiency of the hybrid prior model is demonstrated theoretically and practically, which shows that our SR method can obtain better results from LR images even with stronger noise. Extensive experiments on several visual data have demonstrated the efficacy and superior performance of the proposed algorithm, which can not only preserve image details but also suppress artifacts.     

Integrating the Missing Information Estimation into Multi-frame Super-Resolution

Multi-frame super-resolution image reconstruction aims to restore a high-resolution image by fusing a set of low-resolution images. The low-resolution images are usually subject to some degradation, such as warping, blurring, down-sampling, or noising, which causes substantial information loss in the low-resolution images, especially in the texture regions. The missing information is not well estimated using existing traditional methods. In this paper, having analyzed the observation model describing the degradation process starting with a high-resolution image and moving to the low-resolution images, we propose a more reasonable observation model that integrates the missing information into the super-resolution reconstruction. Our approach is fully formulated in a Bayesian framework using the Kullback–Leibler divergence. In this way, the missing information is estimated simultaneously with the high-resolution image, motion parameters, and hyper-parameters. Our proposed estimation of the missing information improves the quality of the reconstructed image. Experimental results presented in this paper show improved performance compared with that of existing traditional methods.     

Illumination robust face recognition using random projection and sparse representation

Under uneven illumination, the performances degrade significantly for some existing face recognition methods. It is a challenge for face recognition methods to work effectively under different illumination conditions. In this paper, an illumination robust face recognition method, based on random projection and sparse representation, is proposed. In the proposed method, face images are preliminary illumination normalized by gamma correction and difference of Gaussian filtering, and then several projection spaces are obtained by iterative random projection, followed by constructing an initial sample space using Fisher discrimination analysis. This scheme enriches the discrimination abilities of sample features and achieves the security and completeness for biometric template. Test samples are sparsely decomposed into each subspace, and based on statistical average residual, a modified sparse representation method is proposed to realize face recognition with higher stability and illumination robustness. Experimental results indicate that the proposed method provides competitive performance with acceptable computational efficiency. Specifically, for the five subsets of Yale B database, our approach achieves 99.74% average recognition rate, which performs higher accuracy than that of comparative methods.     

Wavelet-based multispectral face recognition

This paper proposes a novel wavelet-based face recognition method using thermal infrared （IR） and visible-light face images. The method applies the combination of Gabor and the Fisherfaces method to the reconstructed IR and visible images derived from wavelet frequency subbands. Our objective is to search for the subbands that are insensitive to the variation in expression and in illumination. The classification performance is improved by combining the multispectal information coming from the subbands that attain individually low equal error rate. Experimental results on Notre Dame face database show that the proposed wavelet-based algorithm outperforms previous multispectral images fusion method as well as monospectral method.     

核典型相关分析的融合人脸识别算法

为了更有效地映射图像数据样本到可分类特征空间,提高分类正确率,提出了一种新的基于核函数的典型相关分析的融合人脸识别算法.该方法首先把图像矩阵通过核函数影射到核空间,然后从核空间的行和列两个方向进行特征抽取,同时避免分解映射后的数据矩阵,简化了数据运算,获得了更具鉴别力的分类特征.在Ohio州立大学的OTCBVS可见/红外人脸数据库中进行了分类识别实验,实验结果表明:该方法可以获得90%以上的识别正确率,优于其他的典型相关分析的人脸识别方法的分类正确率.此外,对不均匀光照变化,表情变化等人脸识别的常见问题具有很好的抵抗能力.     

Example-based image super-resolution with class-specific predictors

Example-based super-resolution is a promising approach to solving the image super-resolution problem. However, the learning process can be slow and prediction can be inaccurate. In this paper, we present a novel learning-based algorithm for image super-resolution to improve the computational speed and prediction accuracy. Our new method classifies image patches into several classes, for each class, a class-specific predictor is designed. A class-specific predictor takes a low-resolution image patch as input and predicts a corresponding high-resolution patch as output. The performances of the class-specific predictors are evaluated using different datasets formed by face images and natural-scene images. We present experimental results which demonstrate that the new method provides improved performances over existing methods.     

Posterior-mean super-resolution with a causal Gaussian Markov random field prior

We propose a Bayesian image super-resolution (SR) method with a causal Gaussian Markov random field (MRF) prior. SR is a technique to estimate a spatially high-resolution image from given multiple low-resolution images. An MRF model with the line process supplies a preferable prior for natural images with edges. We improve the existing image transformation model, the compound MRF model, and its hyperparameter prior model. We also derive the optimal estimator--not the joint maximum a posteriori (MAP) or the marginalized maximum likelihood (ML) but the posterior mean (PM)--from the objective function of the L2-norm-based (mean square error) peak signal-to-noise ratio. Point estimates such as MAP and ML are generally not stable in ill-posed high-dimensional problems because of overfitting, whereas PM is a stable estimator because all the parameters in the model are evaluated as distributions. The estimator is numerically determined by using the variational Bayesian method. The variational Bayesian method is a widely used method that approximately determines a complicated posterior distribution, but it is generally hard to use because it needs the conjugate prior. We solve this problem with simple Taylor approximations. Experimental results have shown that the proposed method is more accurate or comparable to existing methods.