Gender recognition from visible and thermal infrared facial images

Most present research of gender recognition focuses on visible facial images, which are sensitive to illumination changes. In this paper, we proposed hybrid methods for gender recognition by fusing visible and thermal infrared images. First, the active appearance model is used to extract features from visible images, as well as local binary pattern features and several statistical temperature features are extracted from thermal infrared images. Then, feature selection is performed by using the F-test statistic. Third, we propose using Bayesian Networks to perform explicit and implicit fusion of visible and thermal infrared image features. For explicit fusion, we propose two Bayesian Networks to perform decision-level and feature-level fusion. For implicit fusion, we propose using features from one modality as privileged information to improve gender recognition by another modality. Finally, we evaluate the proposed methods on the Natural Visible and Infrared facial Expression spontaneous database and the Equinox face database. Experimental results show that both feature-level and decision-level fusion improve the gender recognition performance, compared to that achieved from one modality. The proposed implicit fusion methods successfully capture the role of privileged information of one modality, thus enhance the gender recognition from another modality.     

A new thermal infrared and visible spectrum images-based pedestrian detection system

In this paper, we propose a hybrid system for pedestrian detection, in which both thermal and visible images of the same scene are used. The proposed method is achieved in two basic steps: (1) Hypotheses generation (HG) where the locations of possible pedestrians in an image are determined and (2) hypotheses verification (HV), where tests are done to check the presence of pedestrians in the generated hypotheses. HG step segments the thermal image using a modified version of OTSU thresholding technique. The segmentation results are mapped into the corresponding visible image to obtain the regions of interests (possible pedestrians). A post-processing is done on the resulting regions of interests to keep only significant ones. HV is performed using random forest as classifier and a color-based histogram of oriented gradients (HOG) together with the histograms of oriented optical flow (HOOF) as features. The proposed approach has been tested on OSU Color-Thermal, INO Video Analytics and LITIV data sets and the results justify its effectiveness.

     

Hourly evapotranspiration derived from NOAA-AVHRR visible and GOES-IMAGER thermal infrared data

In previous studies, remotely sensed values of evapotranspiration are generally computed using a simplified surface energy budget model that employs a semi-empirical coefficient with combinations of Sun-synchronous satellite data and ground-based data. This approach has two main limitations, however: Sun-synchronous satellites have low temporal resolution and the estimation is limited only to a local point around the meteorological station because the models require the aid of ground-base measurements, especially air temperature. This study reduced both limitations through the supplemental use of geostationary satellite (GOES-8) data and remotely sensed estimates of all necessary parameters, including net radiation, air temperature and surface temperature. In particular, air temperature, which is an important meteorological parameter in evapotranspiration estimation, was reproduced through third-order polynomial multiple regressions (R² = 0.88; root mean square (rms) error = 2.21 °C). The coefficient needed for the hourly estimate of evapotranspiration was represented through both a Gaussian model and a plane model. The models were constructed using surface roughness length and Sun hour angle, which replaced wind speed – a parameter that is difficult to estimate remotely over land. Assessments show that the models can depict the temporal distributions of empirical coefficients over various land-cover types. The standard error of this coefficient estimate was 0.002 mm h^?1 K^?1 for both time periods. A strong correlation (R² > 0.87; rms. error <0.17 mm h^?1) was found in comparisons between the selected potential and remotely sensed actual evapotranspiration for four land-cover classes.     

Fusion of thermal infrared and visible spectra for robust moving object detection

The detection of moving objects is a crucial step for many video surveillance applications whether using a visible camera (VIS) or an infrared (IR) one. In order to profit from both types, several fusion methods were proposed in the literature: low-level fusion, medium-level fusion and high-level fusion. The first one is the most used for moving objects’ detection in IR and VIS spectra. In this paper, we present an overview of the different moving object detection methods in IR and VIS spectra and a state of the art of the low-level fusion techniques. Moreover, we propose a new method for moving object detection using low-level fusion of IR and VIS spectra. In order to evaluate quantitatively and qualitatively our proposed method, three series of experiments were carried out using two well-known datasets namely “OSU Color-Thermal Database” and “INO-Database”; the results of these evaluations show promising results and demonstrate the effectiveness of the proposed method.     

Cloud classification from visible and infrared SMS-1 data

The success of a statistical classification-design sample model in discriminating cloud-type samples of visible and infrared meteorological satellite data depends on the selection of the design parameters for the system and the ability of the labelled design samples to characterize and discriminate class patterns within the given geographical region. In a companion study by Parikh (1977), pattern recognition design parameters were examined for a four-class problem and a three-class problem for NOAA-1 cloud data. The purpose of this study was to evaluate pattern recognition systems designed in the previous study on SMS-1 design and test sets. Experiments were conducted for both a four-class problem (separation of “low”, “mix”, “cirrus”, and “cumulonimbus” samples) and a three-class problem (separation of “low”, “cirrus”, and “cumulonimbus” samples). For the four-class problem, decreases in classification accuracy ranging from 4% to 11% occurred when the pattern recognition systems were designed and tested on two different data sets selected from the same satellite orbit. A similar decrease was not observed for the well-defined three-class problem.     

Fusion of infrared and visible images using neutrosophic fuzzy sets

Fusion of infrared and visible image is a technology which combines information from two different sensors for the same scene. It also gives extremely effective information complementation, which is widely used for the monitoring systems and military fields. Due to limited field depth in an imaging device, visible images can’t identify some targets that may not be apparent due to poor lighting conditions or because that the background color is similar to the target. To deal with this problem, a simple and efficient image fusion approach of infrared and visible images is proposed to extract target’s details from infrared images and enhance the vision in order to improve the performance of monitoring systems. This method depends on maximum and minimum operations in neutrosophic fuzzy sets. Firstly, the image is transformed from its spatial domain to the neutrosophic domain which is described by three membership sets: truth membership, indeterminacy membership, and falsity membership. The indeterminacy in the input data is handled to provide a comprehensive fusion result. Finally, deneutrosophicised process is made which means that the membership values are retransformed into a normal image space. At the end of the study, experimental results are applied to evaluate the performance of this approach and compare it to the recent image fusion methods using several objective evaluation criteria. These experiments demonstrate that the proposed method achieves outstanding visual performance and excellent objective indicators.

     

Background-subtraction using contour-based fusion of thermal and visible imagery

We present a new background-subtraction technique fusing contours from thermal and visible imagery for persistent object detection in urban settings. Statistical background-subtraction in the thermal domain is used to identify the initial regions-of-interest. Color and intensity information are used within these areas to obtain the corresponding regions-of-interest in the visible domain. Within each region, input and background gradient information are combined to form a Contour Saliency Map. The binary contour fragments, obtained from corresponding Contour Saliency Maps, are then fused into a single image. An A^* path-constrained search along watershed boundaries of the regions-of-interest is used to complete and close any broken segments in the fused contour image. Lastly, the contour image is flood-filled to produce silhouettes. Results of our approach are evaluated quantitatively and compared with other low- and high-level fusion techniques using manually segmented data.     

使用二维EMD的红外和彩色可见光图像融合

针对红外和彩色可见光图像的融合,提出了一种基于二维经验模式分解（BEMD）的新的融合方法。源图像由BEMD分解成为本征模式函数集（IMFs）和残余;再将红外图像的IMFs和残余分别与对应可见光图像的IMFs和残余进行灰度范围匹配后,用加权平均方法进行融合;最后由BEMD重构成融合图像。用此法所得的融合图像增强了红外图像的细节并具有与可见光图像相似的自然色彩。实验中将此法与传统的小波变换方法和主成分分析方法进行了比较,还与经验模式分解（EMD）和复经验模式分解（CEMD）的方法进行了比较,实验结果都证明了该方法的融合效果最优。     

Object matching between visible and infrared images using a Siamese network

Applied Intelligence - In this study, we propose a method for object matching between visible and infrared images. We consider object matching between visible and infrared images as a computational...     

Contactless blood pressure sensing using facial visible and thermal images

Hypertension is one of the leading risk factors for several diseases. Measurement and monitoring of blood pressure anytime and anywhere are important to lower blood pressure and prevent pathogenesis of diseases. Non-contact blood pressure measurement is desired to monitor blood pressure anytime and anywhere. The aim of this study was to develop a non-contact blood pressure sensing system. A previous study reported that amplitude and time differences of facial photoplethysmogram (PPG) components extracted using brightness variation of facial skin color in facial visible images could be useful indices for estimating blood pressure. The maximum error between measured and estimated blood pressure using facial PPG components was 12 mmHg. An additional signal processing algorithm is desired to increase the accuracy for estimating blood pressure using facial PPG components. By contrast, facial skin temperature also reflects changes in the facial blood circulation. High-accuracy estimation of blood pressure could be expected using both facial PPG components and facial skin temperature. In this study, improvement of accuracy for estimating blood pressure using facial PPG components by attempting to apply additional signal processing to facial skin color variation. Furthermore, a correlation analysis between facial skin temperature and measured blood pressure was performed, and individual models for blood pressure estimation were created.     

采用目标提取和NSCT的红外与可见光图像融合

针对目前红外图像和可见光图像融合中,融合图像信息量不足的问题,将目标提取和NSCT方法相结合,对其中的高频目标区域提出了基于局部信息熵的融合规则。将其与小波变换法、拉普拉斯法、NSCT法、提升方向波变换法作比较,并通过熵、标准差、相关系数等参数对融合后的图像进行定量分析。实验结果表明,该方法不但较好地提高了融合图像信息量,而且能够更加有效、准确地提取源图像中的特征,在主观视觉效果与客观评价指标上均取得了较好的融合效果。     

Audio enhancement using local SNR-based sparse binary mask estimation and spectral imputation

This paper proposes a method for enhancing speech and/or audio quality under noisy conditions. The proposed method first estimates the local signal-to-noise ratio (SNR) of the noisy input signal via sparse non-negative matrix factorization (SNMF). Next, a sparse binary mask (SBM) is proposed that separates the audio signal from the noise by measuring the sparsity of the pool of local SNRs from the adjacent frequency bands of the current and several previous frames. However, some spectral gaps remain across frequency bands after applying the binary masks, which distorts the separated audio signal due to spectral discontinuity. Thus, a spectral imputation technique is used to fill the empty spectrum of the frequency band where it is removed by the SBM. Spectral imputation is conducted by online learning NMF with the spectra of the neighboring non-overlapped frequency bands and their local sparsity. The effectiveness of the proposed enhancement method is demonstrated on two different tasks use speech and musical content, respectively. Consequently, objective measurements and subjective listening tests show that the proposed method outperforms conventional speech and audio enhancement methods, such as SNMF-based alternatives and deep recurrent neural networks for speech enhancement, block thresholding, and a commercially available software tool for audio enhancement.     

Determination and correction of the relative shift between the visible and thermal infrared GOES sensor images

A simple method for the correction of the relative shift between the visible and thermal infrared GOES sensor images is introduced. It makes use of the variance operator and the cross-correlation between two patterns. Results indicate that the proposed method is very promising.     

3D face reconstructions from photometric stereo using near infrared and visible light

This paper seeks to advance the state-of-the-art in 3D face capture and processing via novel Photometric Stereo (PS) hardware and algorithms. The first contribution is a new high-speed 3D data capture system, which is capable of acquiring four raw images in approximately 20 ms. The results presented in this paper demonstrate the feasibility of deploying the device in commercial settings. We show how the device can operate with either visible light or near infrared (NIR) light. The NIR light sources offer the advantages of being less intrusive and more covert than most existing face recognition methods allow. Furthermore, our experiments show that the accuracy of the reconstructions is also better using NIR light. The paper also presents a modified four-source PS algorithm which enhances the surface normal estimates by assigning a likelihood measure for each pixel being in a shadowed region. This likelihood measure is determined by the discrepancies between measured pixel brightnesses and expected values. Where the likelihood of shadow is high, then one light source is omitted from the computation for that pixel, otherwise a weighted combination of pixels is used to determine the surface normal. This means that the precise shadow boundary is not required by our method. The results section of the paper provides a detailed analysis of the methods presented and a comparison to ground truth. We also analyse the reflectance properties of a small number of skin samples to test the validity of the Lambertian model and point towards potential improvements to our method using the Oren–Nayar model.     

基于可见光-热红外视觉监控的车间工人跌倒检测算法

为了解决工厂车间视觉监控存在噪声干扰、光线变化、目标遮挡等问题,提出一种基于多模态视觉监控的工人跌倒检测算法.首先,采用热像仪和可见光相机获取车间内全天候监控图像,结合自适应滤波模型对图像进行降噪处理,以抑制环境噪声对监控图像的干扰;然后,构建一种改进的人体姿态特征提取网络,通过融合串联时间帧合并模块和位姿残差模块,以简化目标检测的特征图尺度,实现监控图像中工人区域被部分遮挡时姿态的实时、可靠预测;最后,设计人体轴线倾角、人体外接矩形框长宽比以及双膝盖点移动速度作为工人跌倒判别性特征,进而实现车间内工人的跌倒判别.在自建数据集和公开数据集上对所提出方法进行验证,实验结果表明,所提出算法的跌倒检测精度分别为95.6%和96.3%,与对比算法相比具有更好的准确性和实时性.     

Multispectral particle filter tracking using adaptive decision-based fusion of visible and thermal sequences

Multimedia Tools and Applications - One of the main challenges of detection and tracking of objects in video monitoring is the lighting conditions of the scene under surveillance and its...     

使用统计模型的动态红外和可见光图像融合

针对动态红外和可见光图像融合,提出了一种新的基于统计模型的融合方法,即将图像的小波分解系数用广义高斯分布来建模。首先,源图像分别用双树复小波进行分解;然后,采用加权平均融合规则来进行小波系数的融合,其中加权系数由估计的广义高斯分布参数来计算;最后,将融合后的系数重构为一幅图像。融合图像采用熵、互信息和边缘保持度Q^AB/F来进行质量评价,实验结果表明方法的性能优于其他两种动态图像融合方法。     

Fully convolutional network-based infrared and visible image fusion

Multimedia Tools and Applications - This study proposes a novel fusion framework for infrared and visual images based on a full convolutional network (FCN) in the local non-subsampled shearlet...     

Integrating visible, near-infrared and short-wave infrared hyperspectral and multispectral thermal imagery for geological mapping at Cuprite, Nevada

This study investigated the potential value of integrating hyperspectral visible, near-infrared, and short-wave infrared imagery with multispectral thermal data for geological mapping. Two coregistered aerial data sets of Cuprite, Nevada were used: Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) hyperspectral data, and MODIS/ASTER Airborne Simulator (MASTER) multispectral thermal data. Four classification methods were each applied to AVIRIS, MASTER, and a combined set. Confusion matrices were used to assess the classification accuracy. The assessment showed, in terms of kappa coefficient, that most classification methods applied to the combined data achieved a marked improvement compared to the results using either AVIRIS or MASTER thermal infrared (TIR) data alone. Spectral angle mapper (SAM) showed the best overall classification performance. Minimum distance classification had the second best accuracy, followed by spectral feature fitting (SFF) and maximum likelihood classification. The results of the study showed that SFF applied to the combination of AVIRIS with MASTER TIR data are especially valuable for identification of silicified alteration and quartzite, both of which exhibit distinctive features in the TIR region. SAM showed some advantages over SFF in dealing with multispectral TIR data, obtaining higher accuracy in discriminating low albedo volcanic rocks and limestone which do not have unique, distinguishing features in the TIR region.     

Intelligent fault diagnosis of rotating machinery using infrared thermal image

This study presents a new intelligent diagnosis system for classification of different machine conditions using data obtained from infrared thermography. In the first stage of this proposed system, two-dimensional discrete wavelet transform is used to decompose the thermal image. However, the data attained from this stage are ordinarily high dimensionality which leads to the reduction of performance. To surmount this problem, feature selection tool based on Mahalanobis distance and relief algorithm is employed in the second stage to select the salient features which can characterize the machine conditions for enhancing the classification accuracy. The data received from the second stage are subsequently utilized to intelligent diagnosis system in which support vector machines and linear discriminant analysis methods are used as classifiers. The results of the proposed system are able to assist in diagnosing of different machine conditions.