An efficient deep learning technique for facial emotion recognition

Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion recognition from facial images but face the issue of performance degradation due to poor selection of layers in the convolutional neural network model.To address this issue, we propose an efficient deep learning technique using a convolutional neural network model for classifying emotions from facial images and detecting age and gender from the facial expressions efficiently. Experimental results show that the proposed model outperformed baseline works by achieving an accuracy of 95.65% for emotion recognition, 98.5% for age recognition, and 99.14% for gender recognition.

     

混合多尺度卷积结合双层LSTM语音情感识别

针对深度学习算法在语音情感特征提取方面的不足以及识别准确率不高的问题,本文通过提取语音数据中有效的情感特征,并将特征进行多尺度拼接融合,构造语音情感特征,提高深度学习模型对特征的表现能力。传统递归神经网络无法解决语音情感识别长时依赖问题,本文采用双层LSTM模型来改进语音情感识别效果,提出一种混合多尺度卷积与双层LSTM模型相结合的模型。实验结果表明,在中科院自动化所汉语情感数据库（CASIA）和德国柏林情感公开数据集（Emo-DB）下,本文所提语音情感识别模型相较于其他情感识别模型在准确率方面有较大提高。     

基于人脸图像和脑电的连续情绪识别方法

基于多模态生理数据的连续情绪识别技术在多个领域有重要用途,但碍于被试数据的缺乏和情绪的主观性,情绪识别模型的训练仍需更多的生理模态数据,且依赖于同源被试数据.本文基于人脸图像和脑电提出了多种连续情绪识别方法.在人脸图像模态,为解决人脸图像数据集少而造成的过拟合问题,本文提出了利用迁移学习技术训练的多任务卷积神经网络模型.在脑电信号模态,本文提出了两种情绪识别模型:第一个是基于支持向量机的被试依赖型模型,当测试数据与训练数据同源时有较高准确率;第二个是为降低脑电信号的个体差异性和非平稳特性对情绪识别的影响而提出的跨被试型模型,该模型基于长短时记忆网络,在测试数据和训练数据不同源的情况下也具有稳定的情绪识别性能.为提高对同源数据的情绪识别准确率,本文提出两种融合多模态决策层情绪信息的方法:枚举权重方法和自适应增强方法.实验表明:当测试数据与训练数据同源时,在最佳情况下,双模态情绪识别模型在情绪唤醒度维度和效价维度的平均准确率分别达74.23%和80.30%;而当测试数据与训练数据不同源时,长短时记忆网络跨被试型模型在情绪唤醒度维度和效价维度的准确率分别为58.65%和51.70%.     

Face recognition Face2vec based on deep learning: Small database case

Object classification is a common problem in artificial intelligence and now it is usually approached by deep learning. In the paper the artificial neural network (ANN) architecture is considered. According to described ANN architecture, the ANN models are trained and tested on a relatively small Color-FERET facial image database under different conditions. The best fine-tuned ANN model provides 94% face recognition accuracy on Color-FERET frontal images and 98% face recognition accuracy within 3 attempts. However, for improving recognition system accuracy large data sets are still necessary preferably consisting of millions of images.     

基于ConvLSTM网络的维度情感识别模型研究

学业情绪能够影响和调节学习者的注意、记忆、思维等认知活动,情绪自动识别是智慧学习环境中情感交互和教学决策的基础。目前情绪识别研究主要集中在离散情绪的识别,其在时间轴上是非连续的,无法精准刻画学生学业情绪演变过程,为解决这个问题,基于众包方法建立真实在线学习情境中的中学生学习维度情感数据集,设计基于连续维度情感预测的深度学习分析模型。实验中根据学生学习风格确定触发学生学业情绪的学习材料,并招募32位实验人员进行自主在线学习,实时采集被试面部图像,获取157个学生学业情绪视频;对每个视频进行情感Arousal和Valence二维化,建立包含2 178张学生面部表情的维度数据库;建立基于ConvLSTM网络的维度情感模型,并在面向中学生的维度情感数据库上进行实验,得到一致性相关系数（Concordance Correlation Coefficient,CCC）均值为0.581,同时在Aff-Wild公开数据集上进行实验,得到的一致相关系数均值为0.222。实验表明,提出的基于维度情感模型在Aff-Wild公开数据集维度情绪识别中CCC相关度系数指标提升了7.6%~43.0%。     

A novel fuzzy facial expression recognition system based on facial feature extraction from color face images

Emotion recognition plays an effective and important role in Human-Computer Interaction (HCI). Recently, various approaches to emotion recognition have been proposed in the literature, but they do not provide a powerful approach to recognize emotions from Partially Occluded Facial Images.In this paper, we propose a new method for Emotion Recognition from Facial Expression using Fuzzy Inference System (FIS). This novel method is even able to recognize emotions from Partially Occluded Facial Images. Moreover, this research describes new algorithms for facial feature extraction that demonstrate satisfactory performance and precision. In addition, one of the main factors that have an important influence on the final precision of fuzzy inference systems is the membership function parameters. Therefore, we use a Genetic Algorithm for parameter-tuning of the membership functions. Experimental results report an average precision rate of 93.96% for Emotion Recognition of six basic emotions, which is so promising.     

情感计算与理解研究发展概述

情感在感知、决策、逻辑推理和社交等一系列智能活动中起到核心作用,是实现人机交互和机器智能的重要元素。近年来,随着多媒体数据爆发式增长及人工智能的快速发展,情感计算与理解引发了广泛关注。情感计算与理解旨在赋予计算机系统识别、理解、表达和适应人的情感的能力来建立和谐人机环境,并使计算机具有更高、更全面的智能。根据输入信号的不同,情感计算与理解包含不同的研究方向。本文全面回顾了多模态情感识别、孤独症情感识别、情感图像内容分析以及面部表情识别等不同情感计算与理解方向在过去几十年的研究进展并对未来的发展趋势进行展望。对于每个研究方向,首先介绍了研究背景、问题定义和研究意义;其次从不同角度分别介绍了国际和国内研究现状,包括情感数据标注、特征提取、学习算法、部分代表性方法的性能比较和分析以及代表性研究团队等;然后对国内外研究进行了系统比较,分析了国内研究的优势和不足;最后讨论了目前研究存在的问题及未来的发展趋势与展望,例如考虑个体情感表达差异问题和用户隐私问题等。     

触觉手势情感识别的超限学习方法

为了解决声音和图像情感识别的不足,提出一种新的情感识别方式:触觉情感识别。对CoST（corpus of social touch）数据集进行了一系列触觉情感识别研究,对CoST数据集进行数据预处理,提出一些关于触觉情感识别的特征。利用极限学习机分类器探究不同手势下的情感识别,对14种手势下的3种情感（温柔、正常、暴躁）进行识别,准确度较高,且识别速度快识别时间短。结果表明,手势的不同会影响情感识别的准确率,其中手势“stroke”的识别效果在不同分类器下的分类精度均为最高,且有较好的分类精度,达到72.07%;极限学习机作为触觉情感识别的分类器,具有较好的分类效果,识别速度快;有的手势本身对应着某种情感,从而影响分类结果。     

Emotion recognition from thermal infrared images using deep Boltzmann machine

Facial expression and emotion recognition from thermal infrared images has attracted more and more attentions in recent years. However, the features adopted in current work are either temperature statistical parameters extracted from the facial regions of interest or several hand-crafted features that are commonly used in visible spectrum. Till now there are no image features specially designed for thermal infrared images. In this paper, we propose using the deep Boltzmann machine to learn thermal features for emotion recognition from thermal infrared facial images. First, the face is located and normalized from the thermal infrared images. Then, a deep Boltzmann machine model composed of two layers is trained. The parameters of the deep Boltzmann machine model are further fine-tuned for emotion recognition after pre-training of feature learning. Comparative experimental results on the NVIE database demonstrate that our approach outperforms other approaches using temperature statistic features or hand-crafted features borrowed from visible domain. The learned features from the forehead, eye, and mouth are more effective for discriminating valence dimension of emotion than other facial areas. In addition, our study shows that adding unlabeled data from other database during training can also improve feature learning performance.     

结合人脸图像和脑电的情绪识别技术

本文通过人脸图像和脑电两个输入信号对情绪识别技术展开研究. 采用对应不同情绪的电影片段对被实验者进行情绪刺激的方法采集输入信号. 通过表情识别出八种基本表情的分类, 通过脑电识别出情绪的三种强弱波动. 通过决策层面的信息融合, 进行情绪分类. 最终的识别准确率达到89.5%, 高于采用单模态进行识别的准确率, 分别为: 表情识别: 81.35%, 脑电识别: 71.53%.     

Emotion recognition from geometric facial features using self-organizing map

This paper presents a novel emotion recognition model using the system identification approach. A comprehensive data driven model using an extended Kohonen self-organizing map (KSOM) has been developed whose input is a 26 dimensional facial geometric feature vector comprising eye, lip and eyebrow feature points. The analytical face model using this 26 dimensional geometric feature vector has been effectively used to describe the facial changes due to different expressions. This paper thus includes an automated generation scheme of this geometric facial feature vector. The proposed non-heuristic model has been developed using training data from MMI facial expression database. The emotion recognition accuracy of the proposed scheme has been compared with radial basis function network, multi-layered perceptron model and support vector machine based recognition schemes. The experimental results show that the proposed model is very efficient in recognizing six basic emotions while ensuring significant increase in average classification accuracy over radial basis function and multi-layered perceptron. It also shows that the average recognition rate of the proposed method is comparatively better than multi-class support vector machine.     

Deep learning enhanced attributes conditional random forest for robust facial expression recognition

Automated Facial Expression Recognition (FER) has remained challenging because of the high inter-subject (e.g. the variations of age, gender and ethnic backgrounds) and intra-subject variations (e.g. the variations of low image resolution, occlusion and illumination). To reduce the variations of age, gender and ethnic backgrounds, we have introduced a conditional random forest architecture. Moreover, a deep multi-instance learning model has been proposed for reducing the variations of low image resolution, occlusion and illumination. Unlike most existing models are trained with facial expression labels only, other attributes related to facial expressions such as age and gender are also considered in our proposed model. A large number of experiments were conducted on the public CK+, ExpW, RAF-DB and AffectNet datasets, and the recognition rates reached 99% and 69.72% on the normalized CK+ face database and the challenging natural scene database respectively. The experimental results shows that our proposed method outperforms the state-of-the-art methods and it is robust to occlusion, noise and resolution variation in the wild.

     

面向类内差距表情的深度学习识别

目的 为解决真实环境中由类内差距引起的面部表情识别率低及室内外复杂环境对类内差距较大的面部表情识别难度大等问题,提出一种利用生成对抗网络（generative adversarial network,GAN）识别面部表情的方法。方法 在GAN生成对抗的思想下,构建一种IC-GAN（intra-class gap GAN）网络结构,使用卷积组建编码器、解码器对自制混合表情图像进行更深层次的特征提取,使用基于动量的Adam（adaptive moment estimation）优化算法进行网络权重更新,重点针对真实环境面部表情识别过程中的类内差距较大的表情进行识别,使其更好地适应类内差异较大的任务。结果 基于Pytorch环境,在自制的面部表情数据集上进行训练,在面部表情验证集上进行测试,并与深度置信网络（deep belief network,DBN）和GoogLeNet网络进行对比实验,最终IC-GAN网络的识别结果比DBN网络和GoogLeNet网络分别提高11%和8.3%。结论 实验验证了IC-GAN在类内差距较大的面部表情识别中的精度,降低了面部表情在类内差距较大情况下的误识率,提高了系统鲁棒性,为面部表情的生成工作打下了坚实的基础。     

结合改进卷积神经网络与通道加权的轻量级表情识别

目的 表情是人机交互过程中重要的信息传递方式,因此表情识别具有重要的研究意义。针对目前表情识别方法存在背景干扰大、网络模型参数复杂、泛化性差等问题,本文提出了一种结合改进卷积神经网络(convolutional neural network,CNN)与通道加权的轻量级表情识别方法。方法 首先,采用标准卷积和深度可分离卷积组合神经网络结构,再利用全局平均池化层作为输出层,简化网络的复杂程度,有效降低网络参数;其次,网络引入SE(squeeze-and-excitation)模块进行通道加权,通过在不同卷积层后设置不同的压缩率增强表情特征提取能力,提升网络模型精度;最后,用softmax分类函数实现各类表情的准确分类。结果 本文网络参数量为6 108 519,相较于识别性能较好的Xception神经网络参数减少了63%,并且通过对网络模型的实时性测试,平均识别速度可达128 帧/s。在5个公开的表情数据集上验证网络模型对7种表情的识别效果,与7种卷积神经网络方法相比,在FER2013 (Facial Expression Recognition 2013)、CK+ (the extended Cohn-Kanade) 和JAFFE (Japanses Female Facial Expression) 3个表情数据集的识别精确度提高了5.72%、0.51%和0.28%,在RAF-DB (Real-world Affective Faces Database)、AffectNet这两个in-the-wild表情数据库的识别精确度分别提高了2.04%和0.68%。结论 本文提出的轻量级表情识别方法在不同通道具有不同的加权能力,获取更多表情关键特征信息,提高了模型的泛化性。实验结果表明,本文方法在简化网络的复杂程度、减少计算量的同时能够准确识别人脸表情,能够有效提升网络的识别能力。     

Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources

Emotion recognition is a crucial application in human–computer interaction. It is usually conducted using facial expressions as the main modality, which might not be reliable. In this study, we proposed a multimodal approach that uses 2-channel electroencephalography (EEG) signals and eye modality in addition to the face modality to enhance the recognition performance. We also studied the use of facial images versus facial depth as the face modality and adapted the common arousal–valence model of emotions and the convolutional neural network, which can model the spatiotemporal information from the modality data for emotion recognition. Extensive experiments were conducted on the modality and emotion data, the results of which showed that our system has high accuracies of 67.8% and 77.0% in valence recognition and arousal recognition, respectively. The proposed method outperformed most state-of-the-art systems that use similar but fewer modalities. Moreover, the use of facial depth has outperformed the use of facial images. The proposed method of emotion recognition has significant potential for integration into various educational applications.     

Deep learning analysis of mobile physiological,environmental and location sensor data for emotion detection

The detection and monitoring of emotions are important in various applications, e.g., to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g., EEG and GSR), environmental data (e.g., audio and weather), videos (e.g., for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)     

持续学习改进的人脸表情识别

目的 大量标注数据和深度学习方法极大地提升了图像识别性能。然而,表情识别的标注数据缺乏,训练出的深度模型极易过拟合,研究表明使用人脸识别的预训练网络可以缓解这一问题。但是预训练的人脸网络可能会保留大量身份信息,不利于表情识别。本文探究如何有效利用人脸识别的预训练网络来提升表情识别的性能。方法 本文引入持续学习的思想,利用人脸识别和表情识别之间的联系来指导表情识别。方法指出网络中对人脸识别整体损失函数的下降贡献最大的参数与捕获人脸公共特征相关,对表情识别来说为重要参数,能够帮助感知面部特征。该方法由两个阶段组成：首先训练一个人脸识别网络,同时计算并记录网络中每个参数的重要性;然后利用预训练的模型进行表情识别的训练,同时通过限制重要参数的变化来保留模型对于面部特征的强大感知能力,另外非重要参数能够以较大的幅度变化,从而学习更多表情特有的信息。这种方法称之为参数重要性正则。结果 该方法在RAF-DB（real-world affective faces database）,CK+（the extended Cohn-Kanade database）和Oulu-CASIA这3个数据集上进行了实验评估。在主流数据集RAF-DB上,该方法达到了88.04%的精度,相比于直接用预训练网络微调的方法提升了1.83%。其他数据集的实验结果也表明了该方法的有效性。结论 提出的参数重要性正则,通过利用人脸识别和表情识别之间的联系,充分发挥人脸识别预训练模型的作用,使得表情识别模型更加鲁棒。     

Natural facial expression recognition using differential-AAM and manifold learning

This paper proposes a novel natural facial expression recognition method that recognizes a sequence of dynamic facial expression images using the differential active appearance model (AAM) and manifold learning as follows. First, the differential-AAM features (DAFs) are computed by the difference of the AAM parameters between an input face image and a reference (neutral expression) face image. Second, manifold learning embeds the DAFs on the smooth and continuous feature space. Third, the input facial expression is recognized through two steps: (1) computing the distances between the input image sequence and gallery image sequences using directed Hausdorff distance (DHD) and (2) selecting the expression by a majority voting of k-nearest neighbors (k-NN) sequences in the gallery. The DAFs are robust and efficient for the facial expression analysis due to the elimination of the inter-person, camera, and illumination variations. Since the DAFs treat the neutral expression image as the reference image, the neutral expression image must be found effectively. This is done via the differential facial expression probability density model (DFEPDM) using the kernel density approximation of the positively directional DAFs changing from neutral to angry (happy, surprised) and negatively directional DAFs changing from angry (happy, surprised) to neutral. Then, a face image is considered to be the neutral expression if it has the maximum DFEPDM in the input sequences. Experimental results show that (1) the DAFs improve the facial expression recognition performance over conventional AAM features by 20% and (2) the sequence-based k-NN classifier provides a 95% facial expression recognition performance on the facial expression database (FED06).     

Respiration-based emotion recognition with deep learning

Different physiological signals are of different origins and may describe different functions of the human body. This paper studied respiration (RSP) signals alone to figure out its ability in detecting psychological activity. A deep learning framework is proposed to extract and recognize emotional information of respiration. An arousal-valence theory helps recognize emotions by mapping emotions into a two-dimension space. The deep learning framework includes a sparse auto-encoder (SAE) to extract emotion-related features, and two logistic regression with one for arousal classification and the other for valence classification. For the development of this work an international database for emotion classification known as Dataset for Emotion Analysis using Physiological signals (DEAP) is adopted for model establishment. To further evaluate the proposed method on other people, after model establishment, we used the affection database established by Augsburg University in Germany. The accuracies for valence and arousal classification on DEAP are 73.06% and 80.78% respectively, and the mean accuracy on Augsburg dataset is 80.22%. This study demonstrates the potential to use respiration collected from wearable deices to recognize human emotions.