Facial expression recognition of a speaker using vowel judgment and thermal image processing

We have previously developed a method for the recognition of the facial expression of a speaker. For facial expression recognition, we previously selected three images: (i) just before speaking, (ii) speaking the first vowel, and (iii) speaking the last vowel in an utterance. By using the speech recognition system named Julius, thermal static images are saved at the timed positions of just before speaking, and when just speaking the phonemes of the first and last vowels. To implement our method, we recorded three subjects who spoke 25 Japanese first names which provided all combinations of the first and last vowels. These recordings were used to prepare first the training data and then the test data. Julius sometimes makes a mistake in recognizing the first and/or last vowel (s). For example, /a/ for the first vowel is sometimes misrecognized as /i/. In the training data, we corrected this misrecognition. However, the correction cannot be carried out in the test data. In the implementation of our method, the facial expressions of the three subjects were distinguished with a mean accuracy of 79.8% when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” The mean accuracy of the speech recognition of vowels by Julius was 84.1%.     

Facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

For facial expression recognition, we selected three images: (i) just before speaking, (ii) speaking the first vowel, and (iii) speaking the last vowel in an utterance. In this study, as a pre-processing module, we added a judgment function to distinguish a front-view face for facial expression recognition. A frame of the front-view face in a dynamic image is selected by estimating the face direction. The judgment function measures four feature parameters using thermal image processing, and selects the thermal images that have all the values of the feature parameters within limited ranges which were decided on the basis of training thermal images of front-view faces. As an initial investigation, we adopted the utterance of the Japanese name “Taro,” which is semantically neutral. The mean judgment accuracy of the front-view face was 99.5% for six subjects who changed their face direction freely. Using the proposed method, the facial expressions of six subjects were distinguishable with 84.0% accuracy when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” We expect the proposed method to be applicable for recognizing facial expressions in daily conversation.     

Robust facial expression recognition of a speaker using thermal image processing and updating of fundamental training data

We previously developed a method for the facial expression recognition of a speaker. For facial expression recognition, we selected three static images at the timing positions of just before speaking and while speaking the phonemes of the first and last vowels. Then, only the static image of the front-view face was used for facial expression recognition. However, frequent updates of the training data were time-consuming. To reduce the time for updates, we found that the classifications of “neutral”, “happy”, and “others” were efficient and accurate for facial expression recognition. Using the proposed method with updated training data of “happy” and “neutral” after an interval such as approximately three and a half years, the facial expressions of two subjects were discriminable with 87.0 % accuracy for the facial expressions of “happy”, “neutral”, and “others” when exhibiting the intentional facial expressions of “angry”, “happy”, “neutral”, “sad”, and “surprised”.     

A system for facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

For facial expression recognition, we previously selected three images: (1) just before speaking, and speaking (2) the first vowel and (3) the last vowel in an utterance. A frame of the front-view face in a dynamic image was selected by estimating the face direction. Based on our method, we have been developing an on-line system for recognizing the facial expression of a speaker using front-view face judgment, vowel judgment, and thermal image processing. In the proposed system, we used three personal computers connected by cables to form a local area network. As an initial investigation, we adopted the utterance of the Japanese name ??Taro,?? which is semantically neutral. Using the proposed system, the facial expressions of one male subject were discriminable with 76?% accuracy when he exhibited one of the intentional facial expressions of ??angry,?? ??happy,?? ??neutral,?? ??sad,?? and ??surprised.??     

基于VGG-NET的特征融合面部表情识别

为了解决在面部表情特征提取过程中卷积神经网络CNN和局部二值模式LBP只能提取面部表情图像的单一特征,难以提取与面部变化高度相关的精确特征的问题,提出了一种基于深度学习的特征融合的表情识别方法。该方法将LBP特征和CNN卷积层提取的特征通过加权的方式结合在改进的VGG-16网络连接层中,最后将融合特征送入Softmax分类器获取各类特征的概率,完成基本的6种表情分类。实验结果表明,所提方法在CK+和JAFFE数据集上的平均识别准确率分别达到了97.5%和97.62%,利用融合特征得到的识别结果明显优于利用单一特征识别的效果。与其他方法相比较,该方法能有效提高表情识别准确率,对光照变化更加鲁棒。     

Facial expression recognition using iterative universum twin support vector machine

Facial expressions are one of the most important characteristics of human behaviour. They are very useful in applications on human computer interaction. To classify facial emotions, different feature extraction methods are used with machine learning techniques. In supervised learning, information about the distribution of data is given by data points not belonging to any of the classes. These data points are known as universum data. In this work, we use universum data to perform multiclass classification of facial emotions from human facial images. Moreover, the existing universum based models suffer from the drawback of high training cost, so we propose an iterative universum twin support vector machine (IUTWSVM) using Newton method. Our IUTWSVM gives good generalization performance with less computation cost. To solve the optimization problem of proposed IUTWSVM, no optimization toolbox is required. Further, improper selection of universum points always leads to degraded performance of the model. For generating better universum, a novel scheme is proposed in this work based on information entropy of data. To check the effectiveness of proposed IUTWSVM, several numerical experiments are performed on benchmark real world datasets. For multiclass classification of facial emotions, the performance of IUTWSVM is compared with existing algorithms using different feature extraction techniques. Our proposed algorithm shows better generalization performance with less training cost in both binary as well as multiclass classification problems.     

音素层特征超矢量的说话人识别性能及优化

音素层特征等高层信息的参数由于完全不受信道的影响,被认为可对基于声学参数的低层信息系统进行有益的补充,但高层信息存在数据稀少的缺点。建立了基于音素特征超矢量的识别方法,并采用BUT的音素层语音识别器对其识别性能进行分析,进而尝试通过数据裁剪和KPCA映射的方法来提升该识别方法的性能。结果表明,采用裁剪并不能有效提升其识别性能,但融合KPCA映射的识别算法的性能得到了显著提升。进一步与主流的GMM-UBM系统融合后,相对于GMM-UBM系统,EER从8.4%降至6.7%。     

Classification in a normalized feature space using support vector machines

This paper discusses classification using support vector machines in a normalized feature space. We consider both normalization in input space and in feature space. Exploiting the fact that in this setting all points lie on the surface of a unit hypersphere we replace the optimal separating hyperplane by one that is symmetric in its angles, leading to an improved estimator. Evaluation of these considerations is done in numerical experiments on two real-world datasets. The stability to noise of this offset correction is subsequently investigated as well as its optimality.     

融合独立分量分析与支持向量聚类的人脸表情识别方法

针对人脸表情特征提取及自动聚类问题,提出了融合独立分量分析（ICA）与支持向量聚类（SVC）的人脸表情识别方法。采用ICA方法进行人脸表情的特征提取,然后采用混合因子分析（MFA）的交互参数调整方法得到局部约束支持向量聚类（LCSVC）的半径,有效降低了表情类别聚类边缘的部分干扰,这比单独采用支持向量聚类（SVC）方法效果要好。测试样本时通过比较新旧半径的值进行判决,实验结果表明该方法是有效的。     

Facial expression recognition using bag of distances

The automatic recognition of facial expressions is critical to applications that are required to recognize human emotions, such as multimodal user interfaces. A novel framework for recognizing facial expressions is presented in this paper. First, distance-based features are introduced and are integrated to yield an improved discriminative power. Second, a bag of distances model is applied to comprehend training images and to construct codebooks automatically. Third, the combined distance-based features are transformed into mid-level features using the trained codebooks. Finally, a support vector machine (SVM) classifier for recognizing facial expressions can be trained. The results of this study show that the proposed approach outperforms the state-of-the-art methods regarding the recognition rate, using a CK+ dataset.     

一种基于空间特征矢量的图像分割方法

针对灰度图像,提出了一种基于空间特征矢量的图像分割方法,构建了以像素的灰度、梯度、像素r的邻域均值为特征的三维特征空间,并将图像像素点对应空间特征点。通过计算像素特征矢量与阈值特征矢量的矢量差,求出矢量差与阈值特征矢量的夹角,然后比较夹角与动态分割参数的关系,判定像素所在区域(目标或背景)。实验表明,该方法可以较快速地实现图像分割,分割的效果也比较好。     

Pattern recognition in image processing using interpixel correlation

When within-object interpixel correlation varies appreciably from object to object, it may be important for the classifier to utilize this correlation, as well as the mean and variance of pixel intensities. In this correspondence interpixel correlation is brought into the classification scheme by means of a two-dimensional Markov model.     

改进LDP结合几何特征融合的人脸表情识别

通过局部方向模式(LDP)提取的局部外观特征常用于人脸表情识别中,但其存在容易受噪声像素影响和伪编码的问题.为此,提出一种改进型的LDP编码方案(ILDP).利用Sobel算子代替Kirsch掩模来提取图像中的梯度信息,将梯度信息进行对数变换后进行累加,避免噪声像素点的影响;通过采用一个梯度幅度阈值判断是否为平坦区域,...     

基于生物启发特征和SVM的人脸表情识别

将C1特征应用于静态图像人脸表情识别,提出了一种新的基于生物启发特征和SVM的表情识别算法。提取人脸图像的C1特征,利用PCA+LDA方法对特征进行降维,用SVM进行分类。在JAFFE和Extended Cohn-Kanade（CK+）人脸表情数据库上的实验结果表明,该算法具有较高的识别率,是一种有效的人脸表情识别方法。     

基于细菌觅食算法和支持向量机的表情识别

为正确选择应用于人脸表情识别的支持向量机相关参数,提高表情识别准确率,提出一种应用于表情识别的基于细菌觅食算法的支持向量机参数选择方法。利用细菌觅食算法,通过模拟细菌觅食行为的趋向性操作、复制操作和迁移操作对应用于表情识别的支持向量机的参数进行寻优,避免寻优陷入局部最优,实现参数优化。实验结果表明,采用该方法能够使人脸表情识别分类结果具有更高的准确率。     

Features classification using geometrical deformation feature vector of support vector machine and active appearance algorithm for automatic facial expression recognition

This paper proposes a method for facial expression recognition in image sequences. Face is detected from the scene and then facial features are detected using image normalization, and thresholding techniques. Using an optimization algorithm the Candide wire frame model is adapted properly on the first frame of face image sequence. In the subsequent frames of image sequence facial features are tracked using active appearance algorithm. Once the model fits on the first frame, animation parameters of model are set to zero, to obtain the shape of model for the neutral facial expression of the same face. The last frame of the image sequence corresponds to greatest facial expression intensity. The geometrical displacement of the Candide wire frame nodes, between the neutral expression frame and the last frame, is used as an input to the multiclass support vector machine, which classifies facial expression into one of the class such as happy, surprise, sadness, anger, disgust, fear and neutral. This method is applicable for frontal as well as tilted faces with angle $\pm 30\,^{\circ }, \pm 45\,^{\circ }, \pm 60\,^{\circ }$ with respect to y axis.     

Pose-specific non-linear mappings in feature space towards multiview facial expression recognition

We introduce a novel approach to recognizing facial expressions over a large range of head poses. Like previous approaches, we map the features extracted from the input image to the corresponding features of the face with the same facial expression but seen in a frontal view. This allows us to collect all training data into a common referential and therefore benefit from more data to learn to recognize the expressions. However, by contrast with such previous work, our mapping depends on the pose of the input image: We first estimate the pose of the head in the input image, and then apply the mapping specifically learned for this pose. The features after mapping are therefore much more reliable for recognition purposes. In addition, we introduce a non-linear form for the mapping of the features, and we show that it is robust to occasional mistakes made by the pose estimation stage. We evaluate our approach with extensive experiments on two protocols of the BU3DFE and Multi-PIE datasets, and show that it outperforms the state-of-the-art on both datasets.     

Improved support vector classification using PCA and ICA feature space modification

An approach that unifies subspace feature selection and optimal classification is presented. Independent component analysis (ICA) and principal component analysis (PCA) provide a maximally variant or statistically independent basis for pattern recognition. A support vector classifier (SVC) provides information about the significance of each feature vector. The feature vectors and the principal and independent component bases are modified to obtain classification results which provide lower classification error and better generalization than can be obtained by the SVC on the raw data and its PCA or ICA subspace representation. The performance of the approach is demonstrated with artificial data sets and an example of face recognition from an image database.     

类间学习神经网络的人脸表情识别*

针对目前表情识别类间信息无关状态,提出了一种表情类间学习的神经网络分类识别算法。该算法首先构建一个BP网络学习对和一个距离判据单元,该距离判据单元仅用来计算类间的实际距离,类间期望距离是根据大量实验结果获得的;然后通过类内实际输出和类间期望距离来修正该网络;最后给出一组实例样本进行表情分类识别。实验结果表明,该算法能有效地识别人脸表情,能紧密地将各类表情间的信息联系起来,效率和准确性均有明显提高。