Human authentication based on fusion of thermal and visible face images

Multimedia Tools and Applications - In recent past, considerable amount of research has been done to increase the performance of a face authentication system in uncontrolled environment such as...     

Gender recognition from visible and thermal infrared facial images

Most present research of gender recognition focuses on visible facial images, which are sensitive to illumination changes. In this paper, we proposed hybrid methods for gender recognition by fusing visible and thermal infrared images. First, the active appearance model is used to extract features from visible images, as well as local binary pattern features and several statistical temperature features are extracted from thermal infrared images. Then, feature selection is performed by using the F-test statistic. Third, we propose using Bayesian Networks to perform explicit and implicit fusion of visible and thermal infrared image features. For explicit fusion, we propose two Bayesian Networks to perform decision-level and feature-level fusion. For implicit fusion, we propose using features from one modality as privileged information to improve gender recognition by another modality. Finally, we evaluate the proposed methods on the Natural Visible and Infrared facial Expression spontaneous database and the Equinox face database. Experimental results show that both feature-level and decision-level fusion improve the gender recognition performance, compared to that achieved from one modality. The proposed implicit fusion methods successfully capture the role of privileged information of one modality, thus enhance the gender recognition from another modality.     

Fusion of visible and thermal images for facial expression recognition

Most present research into facial expression recognition focuses on the visible spectrum, which is sensitive to illumination change. In this paper, we focus on integrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the active appearance model AAM parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from infrared (IR) images. Second, feature selection is performed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression (NVIE) spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal IR images’ supplementary role for visible facial expression recognition.     

Hierarchical fusion of multi-spectral face images for improved recognition performance

This paper presents a two level hierarchical fusion of face images captured under visible and infrared light spectrum to improve the performance of face recognition. At image level fusion, two face images from different spectrums are fused using DWT based fusion algorithm. At feature level fusion, the amplitude and phase features are extracted from the fused image using 2D log polar Gabor wavelet. An adaptive SVM learning algorithm intelligently selects either the amplitude or phase features to generate a fused feature set for improved face recognition. The recognition performance is observed under the worst case scenario of using single training images. Experimental results on Equinox face database show that the combination of visible light and short-wave IR spectrum face images yielded the best recognition performance with an equal error rate of 2.86%. The proposed image-feature fusion algorithm also performed better than existing fusion algorithms.     

IR and visible light face recognition

This paper presents the results of several large-scale studies of face recognition employing visible-light and infrared (IR) imagery in the context of principal component analysis. We find that in a scenario involving time lapse between gallery and probe, and relatively controlled lighting, (1) PCA-based recognition using visible-light images outperforms PCA-based recognition using infrared images, (2) the combination of PCA-based recognition using visible-light and infrared imagery substantially outperforms either one individually. In a same-session scenario (i.e., near-simultaneous acquisition of gallery and probe images) neither modality is significantly better than the other. These experimental results reinforce prior research that employed a smaller data set, presenting a convincing argument that, even across a broad experimental spectrum, the behaviors enumerated above are valid and consistent.     

一种有效的可见光图像中水坝目标的识别方法

提出了一种在中高分辨率可见光图像中识别水坝的有效方法。根据源图像的分辨率采样图像,利用梯度直方图及区域增长获得水坝上游的大面积水域,用链码表示大面积水域,找寻下游水域,分割可能包含水坝的区域;在初步分割的基础上,提取目标的一组特征参数。算法具有检测率高、扩展性好等特点。实验证明该算法能够有效地在中高分辨率可见光图像上识别水坝。     

Particle swarm optimization based fusion of near infrared and visible images for improved face verification

This paper presents two novel image fusion schemes for combining visible and near infrared face images (NIR), aiming at improving the verification performance. Sub-band decomposition is first performed on the visible and NIR images separately. In both cases, we further employ particle swarm optimization (PSO) to find an optimal strategy for performing fusion of the visible and NIR sub-band coefficients. In the first scheme, PSO is used to calculate the optimum weights of a weighted linear combination of the coefficients. In the second scheme, PSO is used to select an optimal subset of features from visible and near infrared face images. To evaluate and compare the efficacy of the proposed schemes, we have performed extensive verification experiments on the IRVI database. This database was acquired in our laboratory using a new sensor that is capable of acquiring visible and near infrared face images simultaneously thereby avoiding the need for image calibration. The experiments show the strong superiority of our first scheme compared to NIR and score fusion performance, which already showed a good stability to illumination variations.     

A novel deep network architecture for reconstructing RGB facial images from thermal for face recognition

Multimedia Tools and Applications - This work proposes a fully convolutional network architecture for RGB face image generation from a given input thermal face image to be applied in face...     

Multi-resolution feature fusion for face recognition

For face recognition, image features are first extracted and then matched to those features in a gallery set. The amount of information and the effectiveness of the features used will determine the recognition performance. In this paper, we propose a novel face recognition approach using information about face images at higher and lower resolutions so as to enhance the information content of the features that are extracted and combined at different resolutions. As the features from different resolutions should closely correlate with each other, we employ the cascaded generalized canonical correlation analysis (GCCA) to fuse the information to form a single feature vector for face recognition. To improve the performance and efficiency, we also employ “Gabor-feature hallucination”, which predicts the high-resolution (HR) Gabor features from the Gabor features of a face image directly by local linear regression. We also extend the algorithm to low-resolution (LR) face recognition, in which the medium-resolution (MR) and HR Gabor features of a LR input image are estimated directly. The LR Gabor features and the predicted MR and HR Gabor features are then fused using GCCA for LR face recognition. Our algorithm can avoid having to perform the interpolation/super-resolution of face images and having to extract HR Gabor features. Experimental results show that the proposed methods have a superior recognition rate and are more efficient than traditional methods.     

基于近红外和可见光差分特征的图像融合方法

暗光环境拍摄的图像通常存在昏暗、噪声、偏色等退化问题。为此,提出一种基于差分特征的可见光与近红外融合方法。首先,观察到近红外与可见光图像成像不一致性,通过两者结构性差异构造差分特征;其次,利用差分特征映射为融合权重进行权重融合;最后,采用偏色校正方法对融合后易产生偏色区域进行亮度复原,使得融合后偏色区域颜色真实自然。提出的方法能切实有效地利用近红外信息,输出高质量清晰图像。实验结论表明,该方法在主观感受和客观评价上均优于现有算法。     

采用非抽样轮廓波的红外与可见光图像融合

提出了一种基于非抽样轮廓波变换的红外与可见光图像融合算法。该算法首先利用非抽样轮廓波变换对输入图像进行多尺度、多方向稀疏分解,有效地表达了图像的高维奇异信息;然后,为了弥补基于像素的图像融合方法的不足,在变换域通过邻域一致性测度的计算,实现了变换系数的局部自适应融合。实验结果表明,所提出的算法既可保持可见光图像的光谱信息,又可有效获取红外图像的热目标信息。     

Downsampling sparse representation and discriminant information aided occluded face recognition

In this paper, a strategy is proposed to deal with a challenging research topic, occluded face recog- nition. Our approach relies on sparse representation on downsampled input image to first locate unoccluded face parts, and then exploits the linear discriminant ability of those pixels to identify the input subject. The advantages and novelties of our method include, 1） since the sparse representation based occlusion detection is conducted on dowsampled image, our algorithm is much faster than classic SRC; 2） the discriminant informa- tion learned from training samples is combined with sparse representation to recognize occluded face for the first time. The verification experiments are conducted on both sinmlated block occlusion images and genuine occluded images.     

Illumination invariant face recognition using near-infrared images

Most current face recognition systems are designed for indoor, cooperative-user applications. However, even in thus-constrained applications, most existing systems, academic and commercial, are compromised in accuracy by changes in environmental illumination. In this paper, we present a novel solution for illumination invariant face recognition for indoor, cooperative-user applications. First, we present an active near infrared (NIR) imaging system that is able to produce face images of good condition regardless of visible lights in the environment. Second, we show that the resulting face images encode intrinsic information of the face, subject only to a monotonic transform in the gray tone; based on this, we use local binary pattern (LBP) features to compensate for the monotonic transform, thus deriving an illumination invariant face representation. Then, we present methods for face recognition using NIR images; statistical learning algorithms are used to extract most discriminative features from a large pool of invariant LBP features and construct a highly accurate face matching engine. Finally, we present a system that is able to achieve accurate and fast face recognition in practice, in which a method is provided to deal with specular reflections of active NIR lights on eyeglasses, a critical issue in active NIR image-based face recognition. Extensive, comparative results are provided to evaluate the imaging hardware, the face and eye detection algorithms, and the face recognition algorithms and systems, with respect to various factors, including illumination, eyeglasses, time lapse, and ethnic groups     

Facial action recognition for facial expression analysis from static face images

Automatic recognition of facial gestures (i.e., facial muscle activity) is rapidly becoming an area of intense interest in the research field of machine vision. In this paper, we present an automated system that we developed to recognize facial gestures in static, frontal- and/or profile-view color face images. A multidetector approach to facial feature localization is utilized to spatially sample the profile contour and the contours of the facial components such as the eyes and the mouth. From the extracted contours of the facial features, we extract ten profile-contour fiducial points and 19 fiducial points of the contours of the facial components. Based on these, 32 individual facial muscle actions (AUs) occurring alone or in combination are recognized using rule-based reasoning. With each scored AU, the utilized algorithm associates a factor denoting the certainty with which the pertinent AU has been scored. A recognition rate of 86% is achieved.     

Multimodal information fusion application to human emotion recognition from face and speech

A multimedia content is composed of several streams that carry information in audio, video or textual channels. Classification and clustering multimedia contents require extraction and combination of information from these streams. The streams constituting a multimedia content are naturally different in terms of scale, dynamics and temporal patterns. These differences make combining the information sources using classic combination techniques difficult. We propose an asynchronous feature level fusion approach that creates a unified hybrid feature space out of the individual signal measurements. The target space can be used for clustering or classification of the multimedia content. As a representative application, we used the proposed approach to recognize basic affective states from speech prosody and facial expressions. Experimental results over two audiovisual emotion databases with 42 and 12 subjects revealed that the performance of the proposed system is significantly higher than the unimodal face based and speech based systems, as well as synchronous feature level and decision level fusion approaches.     

有效的红外与可见光图像融合方法研究*

针对红外图像可视化程度弱、对比度低的问题, 提出一种基于轮廓小波变换和区域能量的红外与可见光图像融合算法。首先进行多尺度小波分解, 然后进行多方向滤波; 引入循环平移方法来消除伪吉布斯失真;采用基于区域的能量融合规则, 重构变换系数得到最终融合结果;最后用信息熵、信噪比等指标来评价融合的性能。实验表明,该方法不论在客观评价还是在主观评价指标上都优于其他融合方法, 提高了融合图像的视觉效果, 可以得到更加清晰的融合图像。     

Background-subtraction using contour-based fusion of thermal and visible imagery

We present a new background-subtraction technique fusing contours from thermal and visible imagery for persistent object detection in urban settings. Statistical background-subtraction in the thermal domain is used to identify the initial regions-of-interest. Color and intensity information are used within these areas to obtain the corresponding regions-of-interest in the visible domain. Within each region, input and background gradient information are combined to form a Contour Saliency Map. The binary contour fragments, obtained from corresponding Contour Saliency Maps, are then fused into a single image. An A^* path-constrained search along watershed boundaries of the regions-of-interest is used to complete and close any broken segments in the fused contour image. Lastly, the contour image is flood-filled to produce silhouettes. Results of our approach are evaluated quantitatively and compared with other low- and high-level fusion techniques using manually segmented data.     

Emotion recognition from thermal infrared images using deep Boltzmann machine

Facial expression and emotion recognition from thermal infrared images has attracted more and more attentions in recent years. However, the features adopted in current work are either temperature statistical parameters extracted from the facial regions of interest or several hand-crafted features that are commonly used in visible spectrum. Till now there are no image features specially designed for thermal infrared images. In this paper, we propose using the deep Boltzmann machine to learn thermal features for emotion recognition from thermal infrared facial images. First, the face is located and normalized from the thermal infrared images. Then, a deep Boltzmann machine model composed of two layers is trained. The parameters of the deep Boltzmann machine model are further fine-tuned for emotion recognition after pre-training of feature learning. Comparative experimental results on the NVIE database demonstrate that our approach outperforms other approaches using temperature statistic features or hand-crafted features borrowed from visible domain. The learned features from the forehead, eye, and mouth are more effective for discriminating valence dimension of emotion than other facial areas. In addition, our study shows that adding unlabeled data from other database during training can also improve feature learning performance.