Human authentication based on fusion of thermal and visible face images

Multimedia Tools and Applications - In recent past, considerable amount of research has been done to increase the performance of a face authentication system in uncontrolled environment such as...     

Gender recognition from visible and thermal infrared facial images

Most present research of gender recognition focuses on visible facial images, which are sensitive to illumination changes. In this paper, we proposed hybrid methods for gender recognition by fusing visible and thermal infrared images. First, the active appearance model is used to extract features from visible images, as well as local binary pattern features and several statistical temperature features are extracted from thermal infrared images. Then, feature selection is performed by using the F-test statistic. Third, we propose using Bayesian Networks to perform explicit and implicit fusion of visible and thermal infrared image features. For explicit fusion, we propose two Bayesian Networks to perform decision-level and feature-level fusion. For implicit fusion, we propose using features from one modality as privileged information to improve gender recognition by another modality. Finally, we evaluate the proposed methods on the Natural Visible and Infrared facial Expression spontaneous database and the Equinox face database. Experimental results show that both feature-level and decision-level fusion improve the gender recognition performance, compared to that achieved from one modality. The proposed implicit fusion methods successfully capture the role of privileged information of one modality, thus enhance the gender recognition from another modality.     

Hierarchical fusion of multi-spectral face images for improved recognition performance

This paper presents a two level hierarchical fusion of face images captured under visible and infrared light spectrum to improve the performance of face recognition. At image level fusion, two face images from different spectrums are fused using DWT based fusion algorithm. At feature level fusion, the amplitude and phase features are extracted from the fused image using 2D log polar Gabor wavelet. An adaptive SVM learning algorithm intelligently selects either the amplitude or phase features to generate a fused feature set for improved face recognition. The recognition performance is observed under the worst case scenario of using single training images. Experimental results on Equinox face database show that the combination of visible light and short-wave IR spectrum face images yielded the best recognition performance with an equal error rate of 2.86%. The proposed image-feature fusion algorithm also performed better than existing fusion algorithms.     

IR and visible light face recognition

This paper presents the results of several large-scale studies of face recognition employing visible-light and infrared (IR) imagery in the context of principal component analysis. We find that in a scenario involving time lapse between gallery and probe, and relatively controlled lighting, (1) PCA-based recognition using visible-light images outperforms PCA-based recognition using infrared images, (2) the combination of PCA-based recognition using visible-light and infrared imagery substantially outperforms either one individually. In a same-session scenario (i.e., near-simultaneous acquisition of gallery and probe images) neither modality is significantly better than the other. These experimental results reinforce prior research that employed a smaller data set, presenting a convincing argument that, even across a broad experimental spectrum, the behaviors enumerated above are valid and consistent.     

Particle swarm optimization based fusion of near infrared and visible images for improved face verification

This paper presents two novel image fusion schemes for combining visible and near infrared face images (NIR), aiming at improving the verification performance. Sub-band decomposition is first performed on the visible and NIR images separately. In both cases, we further employ particle swarm optimization (PSO) to find an optimal strategy for performing fusion of the visible and NIR sub-band coefficients. In the first scheme, PSO is used to calculate the optimum weights of a weighted linear combination of the coefficients. In the second scheme, PSO is used to select an optimal subset of features from visible and near infrared face images. To evaluate and compare the efficacy of the proposed schemes, we have performed extensive verification experiments on the IRVI database. This database was acquired in our laboratory using a new sensor that is capable of acquiring visible and near infrared face images simultaneously thereby avoiding the need for image calibration. The experiments show the strong superiority of our first scheme compared to NIR and score fusion performance, which already showed a good stability to illumination variations.     

A novel deep network architecture for reconstructing RGB facial images from thermal for face recognition

Multimedia Tools and Applications - This work proposes a fully convolutional network architecture for RGB face image generation from a given input thermal face image to be applied in face...     

Multi-resolution feature fusion for face recognition

For face recognition, image features are first extracted and then matched to those features in a gallery set. The amount of information and the effectiveness of the features used will determine the recognition performance. In this paper, we propose a novel face recognition approach using information about face images at higher and lower resolutions so as to enhance the information content of the features that are extracted and combined at different resolutions. As the features from different resolutions should closely correlate with each other, we employ the cascaded generalized canonical correlation analysis (GCCA) to fuse the information to form a single feature vector for face recognition. To improve the performance and efficiency, we also employ “Gabor-feature hallucination”, which predicts the high-resolution (HR) Gabor features from the Gabor features of a face image directly by local linear regression. We also extend the algorithm to low-resolution (LR) face recognition, in which the medium-resolution (MR) and HR Gabor features of a LR input image are estimated directly. The LR Gabor features and the predicted MR and HR Gabor features are then fused using GCCA for LR face recognition. Our algorithm can avoid having to perform the interpolation/super-resolution of face images and having to extract HR Gabor features. Experimental results show that the proposed methods have a superior recognition rate and are more efficient than traditional methods.     

Downsampling sparse representation and discriminant information aided occluded face recognition

In this paper, a strategy is proposed to deal with a challenging research topic, occluded face recog- nition. Our approach relies on sparse representation on downsampled input image to first locate unoccluded face parts, and then exploits the linear discriminant ability of those pixels to identify the input subject. The advantages and novelties of our method include, 1） since the sparse representation based occlusion detection is conducted on dowsampled image, our algorithm is much faster than classic SRC; 2） the discriminant informa- tion learned from training samples is combined with sparse representation to recognize occluded face for the first time. The verification experiments are conducted on both sinmlated block occlusion images and genuine occluded images.     

Illumination invariant face recognition using near-infrared images

Most current face recognition systems are designed for indoor, cooperative-user applications. However, even in thus-constrained applications, most existing systems, academic and commercial, are compromised in accuracy by changes in environmental illumination. In this paper, we present a novel solution for illumination invariant face recognition for indoor, cooperative-user applications. First, we present an active near infrared (NIR) imaging system that is able to produce face images of good condition regardless of visible lights in the environment. Second, we show that the resulting face images encode intrinsic information of the face, subject only to a monotonic transform in the gray tone; based on this, we use local binary pattern (LBP) features to compensate for the monotonic transform, thus deriving an illumination invariant face representation. Then, we present methods for face recognition using NIR images; statistical learning algorithms are used to extract most discriminative features from a large pool of invariant LBP features and construct a highly accurate face matching engine. Finally, we present a system that is able to achieve accurate and fast face recognition in practice, in which a method is provided to deal with specular reflections of active NIR lights on eyeglasses, a critical issue in active NIR image-based face recognition. Extensive, comparative results are provided to evaluate the imaging hardware, the face and eye detection algorithms, and the face recognition algorithms and systems, with respect to various factors, including illumination, eyeglasses, time lapse, and ethnic groups     

Facial action recognition for facial expression analysis from static face images

Automatic recognition of facial gestures (i.e., facial muscle activity) is rapidly becoming an area of intense interest in the research field of machine vision. In this paper, we present an automated system that we developed to recognize facial gestures in static, frontal- and/or profile-view color face images. A multidetector approach to facial feature localization is utilized to spatially sample the profile contour and the contours of the facial components such as the eyes and the mouth. From the extracted contours of the facial features, we extract ten profile-contour fiducial points and 19 fiducial points of the contours of the facial components. Based on these, 32 individual facial muscle actions (AUs) occurring alone or in combination are recognized using rule-based reasoning. With each scored AU, the utilized algorithm associates a factor denoting the certainty with which the pertinent AU has been scored. A recognition rate of 86% is achieved.     

Background-subtraction using contour-based fusion of thermal and visible imagery

We present a new background-subtraction technique fusing contours from thermal and visible imagery for persistent object detection in urban settings. Statistical background-subtraction in the thermal domain is used to identify the initial regions-of-interest. Color and intensity information are used within these areas to obtain the corresponding regions-of-interest in the visible domain. Within each region, input and background gradient information are combined to form a Contour Saliency Map. The binary contour fragments, obtained from corresponding Contour Saliency Maps, are then fused into a single image. An A^* path-constrained search along watershed boundaries of the regions-of-interest is used to complete and close any broken segments in the fused contour image. Lastly, the contour image is flood-filled to produce silhouettes. Results of our approach are evaluated quantitatively and compared with other low- and high-level fusion techniques using manually segmented data.     

Multimodal information fusion application to human emotion recognition from face and speech

A multimedia content is composed of several streams that carry information in audio, video or textual channels. Classification and clustering multimedia contents require extraction and combination of information from these streams. The streams constituting a multimedia content are naturally different in terms of scale, dynamics and temporal patterns. These differences make combining the information sources using classic combination techniques difficult. We propose an asynchronous feature level fusion approach that creates a unified hybrid feature space out of the individual signal measurements. The target space can be used for clustering or classification of the multimedia content. As a representative application, we used the proposed approach to recognize basic affective states from speech prosody and facial expressions. Experimental results over two audiovisual emotion databases with 42 and 12 subjects revealed that the performance of the proposed system is significantly higher than the unimodal face based and speech based systems, as well as synchronous feature level and decision level fusion approaches.     

Emotion recognition from thermal infrared images using deep Boltzmann machine

Facial expression and emotion recognition from thermal infrared images has attracted more and more attentions in recent years. However, the features adopted in current work are either temperature statistical parameters extracted from the facial regions of interest or several hand-crafted features that are commonly used in visible spectrum. Till now there are no image features specially designed for thermal infrared images. In this paper, we propose using the deep Boltzmann machine to learn thermal features for emotion recognition from thermal infrared facial images. First, the face is located and normalized from the thermal infrared im- ages. Then, a deep Boltzmann machine model composed of two layers is trained. The parameters of the deep Boltzmann machine model are further fine-tuned for emotion recognition after pre-tralning of feature learning. Comparative experimental results on the NVIE database demonstrate that our approach outperforms other approaches using temperature statistic features or hand-crafted features borrowed from visible domain. The learned features from the forehead, eye, and mouth are more effective for discriminating valence dimension of emotion than other facial areas. In addition, our study shows that adding unlabeled data from other database during training can also improve feature learning performance.     

Optimal locality preserving projection for face recognition

In face recognition, when the number of images in the training set is much smaller than the number of pixels in each image, Locality Preserving Projections (LPP) often suffers from the singularity problem. To overcome singularity problem, principal component analysis is applied as a preprocessing step. But this procession may discard some important discriminative information. In this paper, a novel algorithm called Optimal Locality Preserving Projections (O-LPP) is proposed. The algorithm transforms the singular eigensystem computation to eigenvalue decomposition problems without losing any discriminative information, which can reduce the computation complexity. And the theoretical analysis related to the algorithm is also obtained. Extensive experiments on face databases demonstrate the proposed algorithm is superior to the traditional LPP algorithm.     

Contactless blood pressure sensing using facial visible and thermal images

Hypertension is one of the leading risk factors for several diseases. Measurement and monitoring of blood pressure anytime and anywhere are important to lower blood pressure and prevent pathogenesis of diseases. Non-contact blood pressure measurement is desired to monitor blood pressure anytime and anywhere. The aim of this study was to develop a non-contact blood pressure sensing system. A previous study reported that amplitude and time differences of facial photoplethysmogram (PPG) components extracted using brightness variation of facial skin color in facial visible images could be useful indices for estimating blood pressure. The maximum error between measured and estimated blood pressure using facial PPG components was 12 mmHg. An additional signal processing algorithm is desired to increase the accuracy for estimating blood pressure using facial PPG components. By contrast, facial skin temperature also reflects changes in the facial blood circulation. High-accuracy estimation of blood pressure could be expected using both facial PPG components and facial skin temperature. In this study, improvement of accuracy for estimating blood pressure using facial PPG components by attempting to apply additional signal processing to facial skin color variation. Furthermore, a correlation analysis between facial skin temperature and measured blood pressure was performed, and individual models for blood pressure estimation were created.     

Expression-robust 3D face recognition based on feature-level fusion and feature-region fusion

3D face shape is essentially a non-rigid free-form surface, which will produce non-rigid deformation under expression variations. In terms of that problem, a promising solution named Coherent Point Drift (CPD) non-rigid registration for the non-rigid region is applied to eliminate the influence from the facial expression while guarantees 3D surface topology. In order to take full advantage of the extracted discriminative feature of the whole face under facial expression variations, the novel expression-robust 3D face recognition method using feature-level fusion and feature-region fusion is proposed. Furthermore, the Principal Component Analysis and Linear Discriminant Analysis in combination with Rotated Sparse Regression (PL-RSR) dimensionality reduction method is presented to promote the computational efficiency and provide a solution to the curse of dimensionality problem, which benefit the performance optimization. The experimental evaluation indicates that the proposed strategy has achieved the rank-1 recognition rate of 97.91 % and 96.71 % based on Face Recognition Grand Challenge (FRGC) v2.0 and Bosphorus respectively, which means the proposed approach outperforms state-of-the-art approach.     

Representation plurality and fusion for 3-D face recognition

In this paper, we present an extensive study of 3-D face recognition algorithms and examine the benefits of various score-, rank-, and decision-level fusion rules. We investigate face recognizers from two perspectives: the data representation techniques used and the feature extraction algorithms that match best each representation type. We also consider novel applications of various feature extraction techniques such as discrete Fourier transform, discrete cosine transform, nonnegative matrix factorization, and principal curvature directions to the shape modality. We discuss and compare various classifier combination methods such as fixed rules voting- and rank-based fusion schemes. We also present a dynamic confidence estimation algorithm to boost fusion performance. In identification experiments performed on FRGC v1.0 and FRGC v2.0 face databases, we tried to find the answers to the following questions: 1) the relative importance of the face representation technique vis-à-vis the types of features extracted; 2) the impact of the gallery size; 3) the conditions, under which subspace methods are preferable, and the compression factor; 4) the most advantageous fusion level and fusion methods; 5) the role confidence votes in improving fusion and the style of selecting experts in the fusion; and 6) the consistency of the conclusions across different databases.     

Reconstruction and recognition of face and digit images using autoencoders

This paper presents techniques for image reconstruction and recognition using autoencoders. Experiments are conducted to compare the performances of three types of autoencoder neural networks based on their efficiency of reconstruction and recognition. Reconstruction error and recognition rate are determined in all the three cases using the same architecture configuration and training algorithm. The results obtained with autoencoders are also compared with those obtained using principal component analysis method. Instead of whole images, image patches are used for training, and this leads to much simpler autoencoder architectures and reduced training time.