A system for facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

For facial expression recognition, we previously selected three images: (1) just before speaking, and speaking (2) the first vowel and (3) the last vowel in an utterance. A frame of the front-view face in a dynamic image was selected by estimating the face direction. Based on our method, we have been developing an on-line system for recognizing the facial expression of a speaker using front-view face judgment, vowel judgment, and thermal image processing. In the proposed system, we used three personal computers connected by cables to form a local area network. As an initial investigation, we adopted the utterance of the Japanese name ??Taro,?? which is semantically neutral. Using the proposed system, the facial expressions of one male subject were discriminable with 76?% accuracy when he exhibited one of the intentional facial expressions of ??angry,?? ??happy,?? ??neutral,?? ??sad,?? and ??surprised.??     

Facial expression recognition of a speaker using vowel judgment and thermal image processing

We have previously developed a method for the recognition of the facial expression of a speaker. For facial expression recognition, we previously selected three images: (i) just before speaking, (ii) speaking the first vowel, and (iii) speaking the last vowel in an utterance. By using the speech recognition system named Julius, thermal static images are saved at the timed positions of just before speaking, and when just speaking the phonemes of the first and last vowels. To implement our method, we recorded three subjects who spoke 25 Japanese first names which provided all combinations of the first and last vowels. These recordings were used to prepare first the training data and then the test data. Julius sometimes makes a mistake in recognizing the first and/or last vowel (s). For example, /a/ for the first vowel is sometimes misrecognized as /i/. In the training data, we corrected this misrecognition. However, the correction cannot be carried out in the test data. In the implementation of our method, the facial expressions of the three subjects were distinguished with a mean accuracy of 79.8% when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” The mean accuracy of the speech recognition of vowels by Julius was 84.1%.     

Facial expression recognition of a speaker using thermal image processing and reject criteria in feature vector space

In our previously developed method for the facial expression recognition of a speaker, the positions of feature vectors in the feature vector space in image processing were generated with imperfections. The imperfections, which caused misrecognition of the facial expression, tended to be far from the center of gravity of the class to which the feature vectors belonged. In the present study, to omit the feature vectors generated with imperfections, a method using reject criteria in the feature vector space was applied to facial expression recognition. Using the proposed method, the facial expressions of two subjects were discriminable with 86.8 % accuracy for the three facial expressions of “happy”, “neutral”, and “others” when they exhibited one of the five intentional facial expressions of “angry”, “happy”, “neutral”, “sad”, and “surprised”, whereas these expressions were discriminable with 78.0 % accuracy by the conventional method. Moreover, the proposed method effectively judged whether the training data were acceptable for facial expression recognition at the moment.     

Robust facial expression recognition of a speaker using thermal image processing and updating of fundamental training data

We previously developed a method for the facial expression recognition of a speaker. For facial expression recognition, we selected three static images at the timing positions of just before speaking and while speaking the phonemes of the first and last vowels. Then, only the static image of the front-view face was used for facial expression recognition. However, frequent updates of the training data were time-consuming. To reduce the time for updates, we found that the classifications of “neutral”, “happy”, and “others” were efficient and accurate for facial expression recognition. Using the proposed method with updated training data of “happy” and “neutral” after an interval such as approximately three and a half years, the facial expressions of two subjects were discriminable with 87.0 % accuracy for the facial expressions of “happy”, “neutral”, and “others” when exhibiting the intentional facial expressions of “angry”, “happy”, “neutral”, “sad”, and “surprised”.     

A methodology for rapid illumination-invariant face recognition using image processing filters

Achieving illumination invariance in the presence of large pose changes remains one of the most challenging aspects of automatic face recognition from low resolution imagery. In this paper, we propose a novel recognition methodology for their robust and efficient matching. The framework is based on outputs of simple image processing filters that compete with unprocessed greyscale input to yield a single matching score between two individuals. Specifically, we show how the discrepancy of the illumination conditions between query input and training (gallery) data set can be estimated implicitly and used to weight the contributions of the two competing representations. The weighting parameters are representation-specific (i.e. filter-specific), but not gallery-specific. Thus, the computationally demanding, learning stage of our algorithm is offline-based and needs to be performed only once, making the added online overhead minimal. Finally, we describe an extensive empirical evaluation of the proposed method in both a video and still image-based setup performed on five databases, totalling 333 individuals, over 1660 video sequences and 650 still images, containing extreme variation in illumination, pose and head motion. On this challenging data set our algorithm consistently demonstrated a dramatic performance improvement over traditional filtering approaches. We demonstrate a reduction of 50–75% in recognition error rates, the best performing method-filter combination correctly recognizing 97% of the individuals.     

Speech synthesis of emotions using vowel features of a speaker

Recently, methods for adding emotion to synthetic speech have received considerable attention in the field of speech synthesis research. We previously proposed a case-based method for generating emotional synthetic speech by exploiting the characteristics of the maximum amplitude and the utterance time of vowels, and the fundamental frequency of emotional speech. In the present study, we propose a method in which our reported method is further improved by controlling the fundamental frequency of emotional synthetic speech. As an initial investigation, we adopted the utterance of a Japanese name that is semantically neutral. By using the proposed method, emotional synthetic speech made from the emotional speech of one male subject was discriminable with a mean accuracy of 83.9 % when 18 subjects listened to the emotional synthetic utterances of “angry,” “happy,” “neutral,” “sad,” or “surprised” when the utterance was the Japanese name “Taro,” or “Hiroko.” Further adjustment of fundamental frequency in the proposed method made a much clearer impression on the subjects for emotional synthetic speech.     

利用图像处理技术解决人脸识别中小样本问题

通过应用PCA及2DPCA算法进行人脸识别,得到了在取不同特征值门限情况下的特征提取维数和识别率,给出了以上两种算法最优特征提取向量的维数和最大特征值门限,并在此基础上应用双线性差值图像旋转处理技术,增加了同一个人较少训练样本情况下的训练样本数量,提高了识别率,从一定程度上解决了小样本问题。如果能从小样本图像中生成出一些新的预测信息,例如,增加同一个训练样本的不同的表情,或改变样本表情的深度,实验的效果可能更加明显。     

Facial expression analysis and expression-invariant face recognition by manifold-based synthesis

Over the last decades, expression classification and face recognition have received substantial attention in computer vision and pattern recognition with more recent efforts focusing on understanding and modelling expression variations. In this paper, we present an expression classification and expression-invariant face recognition method by synthesising photorealistic expression manifolds to expand the gallery set. By means of synthesising expression images from neutral faces, more within-subject variability can be obtained. Eigentransformation is utilised to generate both shape and expression details for novel subjects. Expression classification and face recognition are then performed on the extended training set with synthesised expressions. Experimental results on various datasets show that the proposed method is robust for recognising various expressions and faces with varying degrees of expression. Comprehensive experiments conducted and comparisons with the existing methods are reported. Cross-database synthesis and effect of landmark quality are also studied.     

Facial expression recognition using bag of distances

The automatic recognition of facial expressions is critical to applications that are required to recognize human emotions, such as multimodal user interfaces. A novel framework for recognizing facial expressions is presented in this paper. First, distance-based features are introduced and are integrated to yield an improved discriminative power. Second, a bag of distances model is applied to comprehend training images and to construct codebooks automatically. Third, the combined distance-based features are transformed into mid-level features using the trained codebooks. Finally, a support vector machine (SVM) classifier for recognizing facial expressions can be trained. The results of this study show that the proposed approach outperforms the state-of-the-art methods regarding the recognition rate, using a CK+ dataset.     

Facial expression recognition using ASM-based post-processing technique

Facial expression recognition is a challenging field in numerous researches, and impacts important applications in many areas such as human-computer interaction and data-driven animation, etc. Therefore, this paper proposes a facial expression recognition system using active shape model (ASM) landmark information and appearance-based classification algorithm, i.e., embedded hidden Markov model (EHMM). First, we use ASM landmark information for facial image normalization and weight factors of probability resulted from EHMM. The weight factor is calculated through investigating Kullback-Leibler (KL) divergence of best feature with high discrimination power. Next, we introduce the appearance-based recognition algorithm for classification of emotion states. Here, appearance-based recognition means the EHMM algorithm using two-dimensional discrete cosine transform (2D-DCT) feature vector. The performance evaluation of proposed method was performed with the CK facial expression database and the JAFFE database. As a result, the method using ASM information showed performance improvements of 6.5 and 2.5% compared to previous method using ASM-based face alignment for CK database and JAFFE database, respectively.     

基于子区域加权的不同年龄段人脸表情识别

人脸图像中不同子区域对表情识别的贡献度不同,而且同一子区域对不同年龄段人(如中老年、青年、儿童)的表情识别贡献度也不同。因此,若采用单一固定的子区域加权模式进行人脸表情识别,无法达到最佳识别率。为了提高识别率,提出一种可变加权值的表情识别方法。对中老年人、青年人和儿童分别建立表情数据库,分割出纯人脸区域、眼睛区域和嘴巴区域。对这些区域提取特征后将其进行加权融合,通过设置不同的权值研究其对不同年龄段人脸表情识别的影响。实验结果表明,采用可变加权值比采用固定加权值方法的识别率明显更高。对中老年人的表情识别率提高了8.6%,对青年人的表情识别率提高了4.8%,对儿童的表情识别率提高了1.4%。     

Application of two-dimensional canonical correlation analysis for face image processing and recognition

Paper presents the method of two-dimensional canonical correlation analysis (2DCCA) applied to image processing and biometrics. Method is based on representing the image as the sets of its rows (r) and columns (c) and implementation of CCA using these sets (for this reason we named the method as CCArc). CCArc features simple implementation and lower complexity than other known approaches. In applications to biometrics CCArc is suitable to solving the problems when dimension of images (dimension of feature space) is greater than number of images, i.e. when Small Sample Size (SSS) problem exists.     

Facial action recognition for facial expression analysis from static face images

Automatic recognition of facial gestures (i.e., facial muscle activity) is rapidly becoming an area of intense interest in the research field of machine vision. In this paper, we present an automated system that we developed to recognize facial gestures in static, frontal- and/or profile-view color face images. A multidetector approach to facial feature localization is utilized to spatially sample the profile contour and the contours of the facial components such as the eyes and the mouth. From the extracted contours of the facial features, we extract ten profile-contour fiducial points and 19 fiducial points of the contours of the facial components. Based on these, 32 individual facial muscle actions (AUs) occurring alone or in combination are recognized using rule-based reasoning. With each scored AU, the utilized algorithm associates a factor denoting the certainty with which the pertinent AU has been scored. A recognition rate of 86% is achieved.     

基于生物启发特征和SVM的人脸表情识别

将C1特征应用于静态图像人脸表情识别,提出了一种新的基于生物启发特征和SVM的表情识别算法。提取人脸图像的C1特征,利用PCA+LDA方法对特征进行降维,用SVM进行分类。在JAFFE和Extended Cohn-Kanade（CK+）人脸表情数据库上的实验结果表明,该算法具有较高的识别率,是一种有效的人脸表情识别方法。     

Facial expression recognition using constructive feedforward neural networks

A new technique for facial expression recognition is proposed, which uses the two-dimensional (2D) discrete cosine transform (DCT) over the entire face image as a feature detector and a constructive one-hidden-layer feedforward neural network as a facial expression classifier. An input-side pruning technique, proposed previously by the authors, is also incorporated into the constructive learning process to reduce the network size without sacrificing the performance of the resulting network. The proposed technique is applied to a database consisting of images of 60 men, each having five facial expression images (neutral, smile, anger, sadness, and surprise). Images of 40 men are used for network training, and the remaining images of 20 men are used for generalization and testing. Confusion matrices calculated in both network training and generalization for four facial expressions (smile, anger, sadness, and surprise) are used to evaluate the performance of the trained network. It is demonstrated that the best recognition rates are 100% and 93.75% (without rejection), for the training and generalizing images, respectively. Furthermore, the input-side weights of the constructed network are reduced by approximately 30% using our pruning method. In comparison with the fixed structure back propagation-based recognition methods in the literature, the proposed technique constructs one-hidden-layer feedforward neural network with fewer number of hidden units and weights, while simultaneously provide improved generalization and recognition performance capabilities.     

Improved vowel region detection from a continuous speech using post processing of vowel onset points and vowel end-points

Multimedia Tools and Applications - Vowels are produced with an open configuration of the vocal tract, without any audible friction. The acoustic signal is relatively loud with varying strength of...     

类间学习神经网络的人脸表情识别*

针对目前表情识别类间信息无关状态,提出了一种表情类间学习的神经网络分类识别算法。该算法首先构建一个BP网络学习对和一个距离判据单元,该距离判据单元仅用来计算类间的实际距离,类间期望距离是根据大量实验结果获得的;然后通过类内实际输出和类间期望距离来修正该网络;最后给出一组实例样本进行表情分类识别。实验结果表明,该算法能有效地识别人脸表情,能紧密地将各类表情间的信息联系起来,效率和准确性均有明显提高。     

Pattern recognition in image processing using interpixel correlation

When within-object interpixel correlation varies appreciably from object to object, it may be important for the classifier to utilize this correlation, as well as the mean and variance of pixel intensities. In this correspondence interpixel correlation is brought into the classification scheme by means of a two-dimensional Markov model.     

Face recognition using elastic local reconstruction based on a single face image

In this paper, we propose a new face recognition algorithm based on a single frontal-view image for each face subject, which considers the effect of the face manifold structure. To compare two near-frontal face images, each face is considered a combination of a sequence of local image blocks. Each of the image blocks of one image can be reconstructed according to the corresponding local image block of the other face image. Then an elastic local reconstruction (ELR) method is proposed to measure the similarities between the image block pairs in order to measure the difference between the two face images. Our algorithm not only benefits from the face manifold structure, in terms of being robust to various image variations, but also is computationally simple because there is no need to build the face manifold. We evaluate the performance of our proposed face recognition algorithm with the use of different databases, which are produced under various conditions, e.g. lightings, expressions, perspectives, with/without glasses and occlusions. Consistent and promising experimental results were obtained, which show that our algorithm can greatly improve the recognition rates under all the different conditions.