遮挡人脸表情识别深度学习方法研究进展

在真实环境下遮挡是准确分析识别人脸表情的主要障碍之一。近年来研究者采用深度学习技术解决遮挡条件下表情误识别率高的问题。针对遮挡表情识别的深度学习算法和遮挡相关的问题进行归纳总结。首先,概括局部遮挡条件下表情识别的发展现状、表情的表示方式以及研究遮挡表情用到的数据集;其次,回顾遮挡表情识别深度学习方法的最新进展和分析遮挡对表情的影响;最后,总结主要技术挑战,研究难点及其可能的应对策略。目的是为将来的遮挡表情识别研究提供更有益的参考依据和基准。     

基于CNN集成学习的人脸表情识别系统的设计

随着计算机计算资源的提升以及深度学习理论的不断丰富,自动的人脸表情识别技术已经得到了进一步的发展。但由于表情存在复杂性以及微妙性,实现实时的人脸表情识别仍是一大难题。文章设计了一种基于CNN集成学习的人脸表情识别系统,该系统在FER2013数据集上表情的识别准确率达到70.84%,能够实现实时的、高精度的表情识别。     

Recognition of facial expressions based on salient geometric features and support vector machines

Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.     

基于Mixer Layer的人脸表情识别

目前人脸表情识别研究多数采用卷积神经网络(CNN)提取人脸特征并分类, CNN的缺点是网络结构复杂, 消耗计算资源. 针对以上缺点, 本文采用基于多层感知机(MLP)的Mixer Layer网络结构用于人脸表情识别. 采用数据增强和迁移学习方法解决数据集样本不足的问题, 搭建了不同层数的Mixer Layer网络. 经过实验比较, 4层Mixer Layer网络在CK+和JAFFE 数据集上的识别准确率分别达到了98.71%和95.93%, 8层Mixer Layer网络在Fer2013数据集上的识别准确率达到了63.06%. 实验结果表明, 无卷积结构的Mixer Layer网络在人脸表情识别任务上表现出良好的学习能力和泛化能力.     

基于深度学习和证据理论的表情识别模型

表情识别是在人脸检测基础之上的更进一步研究,是计算机视觉领域的一个重要研究方向。将研究的目标定位于基于微视频的表情自动识别,研究在大数据环境下,如何使用深度学习技术来辅助和促进表情识别技术的发展。针对表情智能识别过程中存在的一些关键性技术难题,设计了一个全自动表情识别模型。该模型结合深度自编码网络和自注意力机制,构建了一个人脸表情特征自动提取子模型,然后结合证据理论对多特征分类结果进行有效融合。实验结果表明,该模型能显著提升表情识别的准确度,具有重要的理论意义和研究价值。     

面部表情的深度卷积级联森林识别

面部表情识别广泛应用于各种研究领域,针对面部表情识别使用深度神经网络方法结构复杂、可解释性差和传统机器学习方法特征提取缺乏多样性、识别率低的问题.提出了一种新的深度卷积级联森林(Deep Convolution Cascade Forest,DCCF)方法用于人脸面部表情识别,该方法通过卷积神经网络深度学习人脸面部显著特征,并采用基于随机森林的级联结构森林逐层学习识别出不同的面部表情特征,提高了人脸表情的识别准确率.DCCF在JAFFE、CK+和Fer2013 3个公开面部表情数据集进行了实验,并对面部表情提取的5种特征和7种分类方法进行了比较分析,结果显示DCCF在对比的算法中人脸表情识别性能最好,3个数据集的准确率分别达到91.4%,98.7%,71.6%.     

Exploring the effect of illumination on automatic expression recognition using the ICT-3DRFE database

One of the main challenges in facial expression recognition is illumination invariance. Our long-term goal is to develop a system for automatic facial expression recognition that is robust to light variations. In this paper, we introduce a novel 3D Relightable Facial Expression (ICT-3DRFE) database that enables experimentation in the fields of both computer graphics and computer vision. The database contains 3D models for 23 subjects and 15 expressions, as well as photometric information that allow for photorealistic rendering. It is also facial action units annotated, using FACS standards. Using the ICT-3DRFE database we create an image set of different expressions/illuminations to study the effect of illumination on automatic expression recognition. We compared the output scores from automatic recognition with expert FACS annotations and found that they agree when the illumination is uniform. Our results show that the output distribution of the automatic recognition can change significantly with light variations and sometimes causes the discrimination of two different expressions to be diminished. We propose a ratio-based light transfer method, to factor out unwanted illuminations from given images and show that it reduces the effect of illumination on expression recognition.     

Learning person-specific models for facial expression and action unit recognition

A key assumption of traditional machine learning approach is that the test data are draw from the same distribution as the training data. However, this assumption does not hold in many real-world scenarios. For example, in facial expression recognition, the appearance of an expression may vary significantly for different people. As a result, previous work has shown that learning from adequate person-specific data can improve the expression recognition performance over the one from generic data. However, person-specific data is typically very sparse in real-world applications due to the difficulties of data collection and labeling, and learning from sparse data may suffer from serious over-fitting. In this paper, we propose to learn a person-specific model through transfer learning. By transferring the informative knowledge from other people, it allows us to learn an accurate model for a new subject with only a small amount of person-specific data. We conduct extensive experiments to compare different person-specific models for facial expression and action unit (AU) recognition, and show that transfer learning significantly improves the recognition performance with a small amount of training data.     

面向类内差距表情的深度学习识别

目的 为解决真实环境中由类内差距引起的面部表情识别率低及室内外复杂环境对类内差距较大的面部表情识别难度大等问题,提出一种利用生成对抗网络（generative adversarial network,GAN）识别面部表情的方法。方法 在GAN生成对抗的思想下,构建一种IC-GAN（intra-class gap GAN）网络结构,使用卷积组建编码器、解码器对自制混合表情图像进行更深层次的特征提取,使用基于动量的Adam（adaptive moment estimation）优化算法进行网络权重更新,重点针对真实环境面部表情识别过程中的类内差距较大的表情进行识别,使其更好地适应类内差异较大的任务。结果 基于Pytorch环境,在自制的面部表情数据集上进行训练,在面部表情验证集上进行测试,并与深度置信网络（deep belief network,DBN）和GoogLeNet网络进行对比实验,最终IC-GAN网络的识别结果比DBN网络和GoogLeNet网络分别提高11%和8.3%。结论 实验验证了IC-GAN在类内差距较大的面部表情识别中的精度,降低了面部表情在类内差距较大情况下的误识率,提高了系统鲁棒性,为面部表情的生成工作打下了坚实的基础。     

Pain monitoring: A dynamic and context-sensitive system

The current paper presents an automatic and context sensitive system for the dynamic recognition of pain expression among the six basic facial expressions and neutral on acted and spontaneous sequences. A machine learning approach based on the Transferable Belief Model, successfully used previously to categorize the six basic facial expressions in static images [2], [61], is extended in the current paper for the automatic and dynamic recognition of pain expression from video sequences in a hospital context application. The originality of the proposed method is the use of the dynamic information for the recognition of pain expression and the combination of different sensors, permanent facial features behavior, transient features behavior, and the context of the study, using the same fusion model. Experimental results, on 2-alternative forced choices and, for the first time, on 8-alternative forced choices (i.e. pain expression is classified among seven other facial expressions), show good classification rates even in the case of spontaneous pain sequences. The mean classification rates on acted and spontaneous data reach 81.2% and 84.5% for the 2-alternative and 8-alternative forced choices, respectively. Moreover, the system performances compare favorably to the human observer rates (76%), and lead to the same doubt states in the case of blend expressions.     

大规模连续中国手语数据集的创建与分析

绝大多数健听人不懂手语导致听障人在找工作、就医、法律咨询等各生活、工作领域中遇到了极大的沟通障碍，而手语翻译员需要提前预约，成本也非常高，所以很多科研工作者都开始利用机器学习来开发手语自动翻译器，但其中的大部分研究都因为受到了数据集规模和质量的影响而效果不佳。为解决上述矛盾和问题，创建了目前全球最大的中国连续手语数据集，并使用了考虑身体关节的位置、面部表情及手指关节的端到端的深度学习模型进行有效训练。结论突显了现代深度学习技术在识别复杂手语方面的巨大优势，针对较小子集的BLEU-4已达到30.8。     

基于混淆交叉支撑向量机树的自动面部表情分类方法

面部表情自动分类是情感信息处理研究中的重要内容,为了提高表情识别的准确率以及鲁棒性,提出了一种基于混淆交叉支撑向量机树的面部表情自动分类方法。该方法依据伪Zernike矩特征,以混淆交叉支撑向量机树对矩特征进行学习,实现面部表情的自动分类。混淆交叉支撑向量机树的结构使模型能够根据教师信号将面部表情识别问题分解,在不同的层次上以相对较低的复杂度解决子问题;在训练阶段,对当前中间节点划分的两个子样本集进行混淆交叉,增强了模型在面部表情识别上的整体泛化性能以及鲁棒性。实验对Cohn-Kanade面部表情数据库中的6类基本表情进行自动分类,准确率达到96.31%;与同样基于该数据库的识别方法相比,该方法在识别正确率和鲁棒性上具有较大的优势。     

基于迁移学习的无监督跨域人脸表情识别

本文主要研究了基于迁移学习的无监督跨域人脸表情识别。在过去的几年里,提出的许多方法在人脸表情识别方面取得了令人满意的识别效果。但这些方法通常认为训练和测试数据来自同一个数据集,因此其具有相同的分布。而在实际应用中,这一假设通常并不成立,特别当训练集和测试集来自不同的数据集时,即跨域人脸表情识别问题。为了解决这一问题,本文提出将一种基于联合分布对齐的迁移学习方法(domain align learning)应用于跨域人脸表情识别,该方法通过找到一个特征变换,将源域和目标域数据映射到一个公共子空间中,在该子空间中联合对齐边缘分布和条件分布来减小域之间的分布差异,然后对变换后的特征进行训练得到一个域适应分类器来预测目标域样本标签。为了验证提出算法的有效性,在CK+、Oulu-CASIA NIR和Oulu-CASIA VIS这3个不同的数据库上做了大量实验,实验结果证明所提算法在跨域表情识别上是有效性的。     

Simultaneous Learning of Nonlinear Manifolds Based on the Bottleneck Neural Network

Manifold learning methods are important techniques for nonlinear extraction of high-dimensional data structures. These methods usually extract a global manifold for data. However, in many real-world problems, there is not only one global manifold, but also additional information about the objects is shared by a large number of manifolds. These manifolds can share information for data reconstruction. To simultaneously extract these data manifolds, this paper proposes a nonlinear method based on the deep neural network (NN) named nonlinear manifold separator NN (NMSNN). Unlike unsupervised learning of bottleneck NN, data labels were used for simultaneous manifold learning. This paper makes use of NMSNN for extracting both expression and identity manifolds for facial images of the CK+ database. These manifolds have been evaluated by different metrics. The identity manifold is used for changing image identity. The result of identity recognition by K-nearest neighbor classifier shows that virtual identities are exactly sanitized. The virtual images for different expressions of test subjects are generated by expression manifold. The facial expression recognition rate of 92.86 % is achieved for virtual expressions of test persons. It is shown that NMSNN can be used to enrich datasets by sanitizing virtual images. As a result, 8 and 19 % improvements are gained in the face recognition task by a single image of each person on CK+ and Bosphorus databases, respectively.     

Higher order orthogonal moments for invariant facial expression recognition

Automatic facial expression recognition (FER) is a sub-area of face analysis research that is based heavily on methods of computer vision, machine learning, and image processing. This study proposes a rotation and noise invariant FER system using an orthogonal invariant moment, namely, Zernike moments (ZM) as a feature extractor and Naive Bayesian (NB) classifier. The system is fully automatic and can recognize seven different expressions. Illumination condition, pose, rotation, noise and others changing in the image are challenging task in pattern recognition system. Simulation results on different databases indicated that higher order ZM features are robust in images that are affected by noise and rotation, whereas the computational rate for feature extraction is lower than other methods.     

深度人脸表情识别研究进展

随着人脸表情识别任务逐渐从实验室受控环境转移至具有挑战性的真实世界环境,在深度学习技术的迅猛发展下,深度神经网络能够学习出具有判别能力的特征,逐渐应用于自动人脸表情识别任务。目前的深度人脸表情识别系统致力于解决以下两个问题：1）由于缺乏足量训练数据导致的过拟合问题;2）真实世界环境下其他与表情无关因素变量（例如光照、头部姿态和身份特征）带来的干扰问题。本文首先对近十年深度人脸表情识别方法的研究现状以及相关人脸表情数据库的发展进行概括。然后,将目前基于深度学习的人脸表情识别方法分为两类：静态人脸表情识别和动态人脸表情识别,并对这两类方法分别进行介绍和综述。针对目前领域内先进的深度表情识别算法,对其在常见表情数据库上的性能进行了对比并详细分析了各类算法的优缺点。最后本文对该领域的未来研究方向和机遇挑战进行了总结和展望：考虑到表情本质上是面部肌肉运动的动态活动,基于动态序列的深度表情识别网络往往能够取得比静态表情识别网络更好的识别效果。此外,结合其他表情模型如面部动作单元模型以及其他多媒体模态,如音频模态和人体生理信息能够将表情识别拓展到更具有实际应用价值的场景。     

Automatic coding of facial expressions displayed during posed and genuine pain

We present initial results from the application of an automated facial expression recognition system to spontaneous facial expressions of pain. In this study, 26 participants were videotaped under three experimental conditions: baseline, posed pain, and real pain. The real pain condition consisted of cold pressor pain induced by submerging the arm in ice water. Our goal was to (1) assess whether the automated measurements were consistent with expression measurements obtained by human experts, and (2) develop a classifier to automatically differentiate real from faked pain in a subject-independent manner from the automated measurements. We employed a machine learning approach in a two-stage system. In the first stage, a set of 20 detectors for facial actions from the Facial Action Coding System operated on the continuous video stream. These data were then passed to a second machine learning stage, in which a classifier was trained to detect the difference between expressions of real pain and fake pain. Naïve human subjects tested on the same videos were at chance for differentiating faked from real pain, obtaining only 49% accuracy. The automated system was successfully able to differentiate faked from real pain. In an analysis of 26 subjects with faked pain before real pain, the system obtained 88% correct for subject independent discrimination of real versus fake pain on a 2-alternative forced choice. Moreover, the most discriminative facial actions in the automated system were consistent with findings using human expert FACS codes.     

Expression recognition based on residual rectification convolution neural network

In order to solve the problem of low face recognition rate in controlled scene, an expression recognition algorithm based on residual rectification intensive convolutional neural network is proposed. This method takes convolutional neural network as the prototype. In the process of training model, the idea of residual network is introduced to correct the difference between the effect of test set and the effect of training set. The linear rectification operation of the residual block by the excitation function embedded in the convolution layer helps to express complex features. At the same time, the data intensive method is used to suppress the fast fitting of the deep neural network model during the training process, to improve its generalization performance on a given recognition task, and then to improve the robustness of the model learning effect. In the experiment, the method is applied to simulate the online teaching environment, and get effective facial expression recognition result in controlled scene. According to the experimental data, this method can effectively classify the facial image input under controlled conditions, and the highest accuracy is up to 91.7%. This research is helpful to the development of facial expression recognition and human-computer interaction.

     

心理学视角下的自动表情识别

自动表情识别是心理学与计算机科学等深度交叉的前沿领域。情绪心理学、模式识别、情感计算等领域的研究者发展表情识别相关的理论、数据库和算法,极大地推动了自动表情识别技术的进步。文章基于心理学视角,结合我们前期开展的相关工作,首先梳理自动表情识别的心理学基础、情绪的面部表达方式、表情数据的演化、表情样本的标注等方面的理论观点与实践进展,然后分析指出自动表情识别面临的主要问题,最后基于预测加工理论的建构观点,提出注重交互过程中的表情“理解”,有望进一步提高自动表情识别的有效性,并预期这可能是自动表情识别研究的未来发展方向。