基于生成式对抗网络的鲁棒人脸表情识别

人们在自然情感交流中经常伴随着头部旋转和肢体动作,它们往往导致较大范围的人脸遮挡,使得人脸图像损失部分表情信息.现有的表情识别方法大多基于通用的人脸特征和识别算法,未考虑表情和身份的差异,导致对新用户的识别不够鲁棒.本文提出了一种对人脸局部遮挡图像进行用户无关表情识别的方法.该方法包括一个基于Wasserstein生成式对抗网络（Wasserstein generative adversarial net,WGAN）的人脸图像生成网络,能够为图像中的遮挡区域生成上下文一致的补全图像;以及一个表情识别网络,能够通过在表情识别任务和身份识别任务之间建立对抗关系来提取用户无关的表情特征并推断表情类别.实验结果表明,我们的方法在由CK+,Multi-PIE和JAFFE构成的混合数据集上用户无关的平均识别准确率超过了90%.在CK+上用户无关的识别准确率达到了96%,其中4.5%的性能提升得益于本文提出的对抗式表情特征提取方法.此外,在45°头部旋转范围内,本文方法还能够用于提高非正面表情的识别准确率.     

基于局部特征聚类损失和多类特征融合的面部表情识别

在真实世界中,每个个体对表情的表现方式不同.基于上述事实,文中提出局部特征聚类(LFA)损失函数,能够在深度神经网络的训练过程中减小相同类图像之间的差异,扩大不同类图像之间的差异,从而削弱表情的多态性对深度学习方式提取特征的影响.同时,具有丰富表情的局部区域可以更好地表现面部表情特征,所以提出融入LFA损失函数的深度学习网络框架,提取的面部图像的局部特征用于面部表情识别.实验结果表明文中方法在真实世界的RAF数据集及实验室条件下的CK+数据集上的有效性.     

深度空时婴幼儿表情识别模型下的ASD自动筛查

目的 高危孤独症谱系障碍（high-risk autism spectrum disorder,HR-ASD）筛查依赖于医师的临床评估和问卷量表,传统筛查方式效率低,亟需一种高效的自动筛查工具。为了满足自动筛查的需求,本文提出一种基于婴幼儿表情分析的孤独症谱系障碍自动筛查方法。方法 首先入组30例8~18个月的婴幼儿,包括10例ASD疑似患儿（HR-ASD）和20例正常发育婴幼儿;引入静止脸范式,并利用该范式诱发婴幼儿在社交压力条件下的情绪调节行为;提出一种面向婴幼儿视频表情识别的深度空时特征学习网络,首先在大规模公开数据集AffectNet预训练空域特征学习模型,然后在自建婴幼儿面部表情视频数据集RCLS&NBH+（Research Center of Learning Science&Nanjing Brain Hospital dataset+）上训练时空特征学习模型,从而建立一个较精准的婴幼儿表情识别模型;基于该模型深度特征序列的一阶统计量,构建婴幼儿社交压力环境下的表情行为症状与精神健康状态之间的关联,采用机器学习方法实现自动筛查。结果 1）基于婴幼儿表情人工标注的结果,发现：在1 min静止期,高危组的婴幼儿中性表情持续时长相对正常对照组偏高（p<0.01）,而其他表情未发现有统计学意义的差异;2）提出的深度空时特征学习网络在本研究的30例婴幼儿面部表情视频数据集上的总体平均识别率达到了87.1%,3类表情预测结果与人工标注结果具有较高的一致性,其中Kappa一致性系数达到0.63,Pearson相关系数达到0.67;3）基于面部表情深度特征序列一阶统计量的精神健康状态预测性能达到灵敏度70%,特异性90%,分类正确率83.3%（置换检验p<0.05）。结论 本文提出的基于婴幼儿面部表情深度特征序列一阶统计量的精神健康状态自动预测模型是有效的,有助于实现高危孤独症谱系障碍自动筛查。     

Recognition of 3D facial expression dynamics

In this paper we propose a method that exploits 3D motion-based features between frames of 3D facial geometry sequences for dynamic facial expression recognition. An expressive sequence is modelled to contain an onset followed by an apex and an offset. Feature selection methods are applied in order to extract features for each of the onset and offset segments of the expression. These features are then used to train GentleBoost classifiers and build a Hidden Markov Model in order to model the full temporal dynamics of the expression. The proposed fully automatic system was employed on the BU-4DFE database for distinguishing between the six universal expressions: Happy, Sad, Angry, Disgust, Surprise and Fear. Comparisons with a similar 2D system based on the motion extracted from facial intensity images was also performed. The attained results suggest that the use of the 3D information does indeed improve the recognition accuracy when compared to the 2D data in a fully automatic manner.     

基于Gabor变换与改进SLLE的人脸表情识别

本文通过Gabor变换进行人脸表情图像的特征提取,并利用局部线性嵌入（LLE）系列算法进行数据降维操作.LLE算法是一种非线性降维算法,它可以使得降维后的数据保持原有的拓扑结构,在人脸表情识别中有广泛的应用.因为LLE算法没有考虑样本的类别信息,因此有了监督的局部线性嵌入（SLLE）算法.但是SLLE算法仅仅考虑了样本的类别信息却没有考虑到各种表情之间的关系,因此本文提出一种改进的SLLE算法,该算法认为中性表情是其他各种表情的中心.在JAFFE库上进行人脸表情识别实验结果表明,相比LLE算法和SLLE算法,该算法获得了更好的人脸表情识别率,是一种有效算法.     

基于CNN集成学习的人脸表情识别系统的设计

随着计算机计算资源的提升以及深度学习理论的不断丰富,自动的人脸表情识别技术已经得到了进一步的发展。但由于表情存在复杂性以及微妙性,实现实时的人脸表情识别仍是一大难题。文章设计了一种基于CNN集成学习的人脸表情识别系统,该系统在FER2013数据集上表情的识别准确率达到70.84%,能够实现实时的、高精度的表情识别。     

State-of-the art in component technology for an animated face robot-its component technology development for interactive communication with humans

We are attempting to introduce a 3D, realistic human-like animated face robot to human-robot communication. The face robot can recognize human facial expressions as well as produce realistic facial expressions in real time. For the animated face robot to communicate interactively, we propose a new concept of 'active human interface', and we investigate the performance of real time recognition of facial expressions by neural networks (NN) and the expressionability of facial messages on the face robot. We find that the NN recognition of facial expressions and the face robot's performance in generating facial expressions are of almost same level as that in humans. We also construct an artificial emotion model able to generate six basic emotions in accordance with the recognition of a given facial expression and the situational context. This implies a high potential for the animated face robot to undertake interactive communication with humans, when integrating these three component technologies into the face robot.     

An efficient deep learning technique for facial emotion recognition

Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion recognition from facial images but face the issue of performance degradation due to poor selection of layers in the convolutional neural network model.To address this issue, we propose an efficient deep learning technique using a convolutional neural network model for classifying emotions from facial images and detecting age and gender from the facial expressions efficiently. Experimental results show that the proposed model outperformed baseline works by achieving an accuracy of 95.65% for emotion recognition, 98.5% for age recognition, and 99.14% for gender recognition.

     

11K Hands: Gender recognition and biometric identification using a large dataset of hand images

Human hand not only possesses distinctive feature for gender information, it is also considered one of the primary biometric traits used to identify a person. Unlike face images, which are usually unconstrained, an advantage of hand images is they are usually captured under a controlled position. Most state-of-the-art methods, that rely on hand images for gender recognition or biometric identification, employ handcrafted features to train an off-the-shelf classifier or be used by a similarity metric for biometric identification. In this work, we propose a deep learning-based method to tackle the gender recognition and biometric identification problems. Specifically, we design a two-stream convolutional neural network (CNN) which accepts hand images as input and predicts gender information from these hand images. This trained model is then used as a feature extractor to feed a set of support vector machine classifiers for biometric identification. As part of this effort, we propose a large dataset of human hand images, 11K Hands, which contains dorsal and palmar sides of human hand images with detailed ground-truth information for different problems including gender recognition and biometric identification. By leveraging thousands of hand images, we could effectively train our CNN-based model achieving promising results. One of our findings is that the dorsal side of human hands is found to have effective distinctive features similar to, if not better than, those available in the palmar side of human hand images. To facilitate access to our 11K Hands dataset, the dataset, the trained CNN models, and our Matlab source code are available at (https://goo.gl/rQJndd).

     

Facial expression recognition using bag of distances

The automatic recognition of facial expressions is critical to applications that are required to recognize human emotions, such as multimodal user interfaces. A novel framework for recognizing facial expressions is presented in this paper. First, distance-based features are introduced and are integrated to yield an improved discriminative power. Second, a bag of distances model is applied to comprehend training images and to construct codebooks automatically. Third, the combined distance-based features are transformed into mid-level features using the trained codebooks. Finally, a support vector machine (SVM) classifier for recognizing facial expressions can be trained. The results of this study show that the proposed approach outperforms the state-of-the-art methods regarding the recognition rate, using a CK+ dataset.     

Face alive icon

In this paper, we propose a methodology to synthesize facial expressions from photographs for devices with limited processing power, network bandwidth and display area, which is referred as “LLL” environment. The facial images are reduced to small-sized face alive icons (FAI). Expressions are decomposed into the expression-unrelated facial features and the expression-related expressional features. As a result, the common features can be identified and reused across expressions using a discrete model constructed from the statistical analysis on training dataset. Semantic synthesis rules are introduced to reveal the inner relations of expressions. Verified by the experimental prototype system and usability study, the approach can produce acceptable facial expression images utilizing much less computing, network and storage resource than the traditional approaches.     

Intelligent facial emotion recognition using a layered encoding cascade optimization model

In this research, we propose a facial expression recognition system with a layered encoding cascade optimization model. Since generating an effective facial representation is a vital step to the success of facial emotion recognition, a modified Local Gabor Binary Pattern operator is first employed to derive a refined initial face representation and we then propose two evolutionary algorithms for feature optimization including (i) direct similarity and (ii) Pareto-based feature selection, under the layered cascade model. The direct similarity feature selection considers characteristics within the same emotion category that give the minimum within-class variation while the Pareto-based feature optimization focuses on features that best represent each expression category and at the same time provide the most distinctions to other expressions. Both a neural network and an ensemble classifier with weighted majority vote are implemented for the recognition of seven expressions based on the selected optimized features. The ensemble model also automatically updates itself with the most recent concepts in the data. Evaluated with the Cohn–Kanade database, our system achieves the best accuracies when the ensemble classifier is applied, and outperforms other research reported in the literature with 96.8% for direct similarity based optimization and 97.4% for the Pareto-based feature selection. Cross-database evaluation with frontal images from the MMI database has also been conducted to further prove system efficiency where it achieves 97.5% for Pareto-based approach and 90.7% for direct similarity-based feature selection and outperforms related research for MMI. When evaluated with 90° side-view images extracted from the videos of the MMI database, the system achieves superior performances with >80% accuracies for both optimization algorithms. Experiments with other weighting and meta-learning combination methods for the construction of ensembles are also explored with our proposed ensemble showing great adpativity to new test data stream for cross-database evaluation. In future work, we aim to incorporate other filtering techniques and evolutionary algorithms into the optimization models to further enhance the recognition performance.     

基于深度注意力网络的课堂教学视频中学生表情识别与智能教学评估

为了解决复杂课堂场景下学生表情识别的遮挡的问题,同时发挥深度学习在智能教学评估应用上的优势,提出了一种基于深度注意力网络的课堂教学视频中学生表情识别模型与智能教学评估算法.构建了课堂教学视频库、表情库和行为库,利用裁剪和遮挡策略生成多路人脸图像,在此基础上构建了多路深度注意力网络,并通过自注意力机制为多路网络分配不同权...     

Deep peak-neutral difference feature for facial expression recognition

Facial expression recognition (FER) is important in vision-related applications. Deep neural networks demonstrate impressive performance for face recognition; however, it should be noted that this method relies heavily on a great deal of manually labeled training data, which is not available for facial expressions in real-world applications. Hence, we propose a powerful facial feature called deep peak–neutral difference (DPND) for FER. DPND is defined as the difference between two deep representations of the fully expressive (peak) and neutral facial expression frames. The difference tends to emphasize the facial parts that are changed in the transition from the neutral to the expressive face and to eliminate the face identity information retained in the fine-tuned deep neural network for facial expression, the network has been trained on large-scale face recognition dataset. Furthermore, unsupervised clustering and semi-supervised classification methods are presented to automatically acquire the neutral and peak frames from the expression sequence. The proposed facial expression feature achieved encouraging results on public databases, which suggests that it has strong potential to recognize facial expressions in real-world applications.

     

A novel lossless compression framework for facial depth images in expression recognition

With the development of AR and VR, depth images are widely used for facial expression analysis and recognition. To reduce the storage size and save bandwidth, an efficient compression framework is desired. In this paper, we propose a novel lossless compression framework for facial depth images in expression recognition. In the proposed framework, two steps are designed to remove the redundancy in the facial depth images, which are data preparing and bitstream encoding operations. In the data preparing operation, the original image is represented by the same and different parts between the left and right sides. In the bitstream encoding operation, these parts are compressed to get the final bitstream. The proposed framework is implemented and examined on the BU-3DFE Database. Experimental result shows that the proposed technique outperforms existing lossless compression frameworks in terms of compression efficiency, and the average data size is reduced to 25.27% by the proposed framework.

     

基于Boost改进的嵌入式隐马尔可夫模型的实时表情识别

在人机交互过程中，理解人类的情绪是计算机和人进行交流必备的技能之一。最能表达人类情绪的就是面部表情。设计任何现实情景中的人机界面，面部表情识别是必不可少的。在本文中，我们提出了交互式计算环境中的一种新的实时面部表情识别框架。文章对这个领域的研究主要有两大贡献：第一，提出了一种新的网络结构和基于AdaBoost的嵌入式HMM的参数学习算法。第二，将这种优化的嵌入式HMM用于实时面部表情识别。本文中，嵌入式HMM把二维离散余弦变形后的系数作为观测向量，这和以前利用像素深度来构建观测向量的嵌入式HMM方法不同。因为算法同时修正了嵌入式HMM的网络结构和参数，大大提高了分类的精确度。该系统减少了训练和识别系统的复杂程度，提供了更加灵活的框架，且能应用于实时人机交互应用软件中。实验结果显示该方法是一种高效的面部表情识别方法。     

基于深度可分离卷积结构的人脸表情识别研究

人脸表情识别是计算机视觉领域中人脸识别的一个重要分支。由于人脸表情多样性,头部姿态变化以及表情主 体所处环境等诸多因素的影响,给人脸表情识别的工作带来了很大的挑战。针对采用传统卷积神经网络,由于其模型参数数 量多,且比传统机器学习算法的人脸表情识别精度的提高有限,给出了一种基于深度可分离卷积结构的改进卷积神经网络模 型。基于该模型对Fer2013灰度表情识别数据集进行实验,结果表明,在保证了68.31% 的较高准确率情况下,与传统卷积神经 网络相比,模型的网络结构得到了优化,模型参数数量大大减少,且模型参数的利用效率较高。     

构建并行卷积神经网络的表情识别算法

目的 表情识别在商业、安全、医学等领域有着广泛的应用前景,能够快速准确地识别出面部表情对其研究与应用具有重要意义。传统的机器学习方法需要手工提取特征且准确率难以保证。近年来,卷积神经网络因其良好的自学习和泛化能力得到广泛应用,但还存在表情特征提取困难、网络训练时间过长等问题,针对以上问题,提出一种基于并行卷积神经网络的表情识别方法。方法 首先对面部表情图像进行人脸定位、灰度统一以及角度调整等预处理,去除了复杂的背景、光照、角度等影响,得到了精确的人脸部分。然后针对表情图像设计一个具有两个并行卷积池化单元的卷积神经网络,可以提取细微的表情部分。该并行结构具有3个不同的通道,分别提取不同的图像特征并进行融合,最后送入SoftMax层进行分类。结果 实验使用提出的并行卷积神经网络在CK+、FER2013两个表情数据集上进行了10倍交叉验证,最终的结果取10次验证的平均值,在CK+及FER2013上取得了94.03%与65.6%的准确率。迭代一次的时间分别为0.185 s和0.101 s。结论 为卷积神经网络的设计提供了一种新思路,可以在控制深度的同时扩展广度,提取更多的表情特征。实验结果表明,针对数量、分辨率、大小等差异较大的表情数据集,该网络模型均能够获得较高的识别率并缩短训练时间。