面向学前教育对话机器人的多模态情感识别实现关键技术

为进一步提高学前教育对话机器人交互过程的准确性，结合多模态融合思想，提出一种基于面部表情情感和语音情感融合的识别技术。其中，为解决面部表情异常视频帧的问题，采用卷积神经网络对人脸进行检测，然后基于Gabor小波变换对人脸表情进行特征提取，最后通过残差网络对面部表情情感进行识别；为提高情感识别的准确性，协助学前教育机器人更好地理解儿童情感，在采用MFCC对连续语音特征进行提取后，通过残差网络对连续语音情感进行识别；利用多元线性回归算法对面部和语音情感识别结果进行融合。在AVEC2019数据集上的验证结果表明，表情情感识别和连续语音情感识别均具有较高识别精度；与传统的单一情感识别相比，多模态融合识别的一致性相关系数最高，达0.77。由此得出，将多模态情感识别的方法将有助于提高学前教育对话机器人交互过程中的情感识别水平，提高对话机器人的智能化。     

基于感知重采样和多模态融合的连续情感识别

情感识别在人机交互中发挥着重要的作用,连续情感识别因其能检测到更广泛更细微的情感而备受关注。在多模态连续情感识别中,针对现有方法获取的时序信息包含较多冗余以及多模态交互信息捕捉不全面的问题,提出基于感知重采样和多模态融合的连续情感识别方法。首先感知重采样模块通过非对称交叉注意力机制去除模态冗余信息,将包含时序关系的关键特征压缩到隐藏向量中,降低后期融合的计算复杂度。其次多模态融合模块通过交叉注意力机制捕捉模态间的互补信息,并利用自注意力机制获取模态内的隐藏信息,使特征信息更丰富全面。在Ulm-TSST和Aff-Wild2数据集上唤醒度和愉悦度的CCC均值分别为63.62%和50.09%,证明了该模型的有效性。     

融合背景上下文特征的视觉情感识别与预测方法

为解决基于视觉的情感识别无法捕捉人物所处环境和与周围人物互动对情感识别的影响、单一情感种类无法更丰富地描述人物情感、无法对未来情感进行合理预测的问题,提出了融合背景上下文特征的视觉情感识别与预测方法。该方法由融合背景上下文特征的情感识别模型（Context-ER）和基于GRU与Valence-Arousal连续情感维度的情感预测模型（GRU-mapVA）组成。Context-ER同时综合了面部表情、身体姿态和背景上下文（所处环境、与周围人物互动行为）特征,进行26种离散情感类别的多标签分类和3个连续情感维度的回归。GRU-mapVA根据所提映射规则将Valence-Arousal的预测值投影到改进的Valence-Arousal模型上,使得情感预测类间差异更为明显。Context-ER在Emotic数据集上进行了测试,结果表明,识别情感的平均精确率比现有最优方法提高4%以上;GRU-mapVA在三段视频样本上进行了测试,结果表明情感预测效果相较于现有方法有很大提升。     

真实环境下的多模态情感数据集MED

目的 情感识别的研究一直致力于帮助系统在人机交互的环节中以更合适的方式来对用户的需求进行反馈。但它在现实应用中的表现却较差。主要原因是缺乏与现实应用环境类似的大规模多模态数据集。现有的野外多模态情感数据集很少,而且受试者数量有限,使用的语言单一。方法 为了满足深度学习算法对数据量的要求,本文收集、注释并准备公开发布一个全新的自然状态下的视频数据集（multimodal emotion dataset,MED）。首先收集人员从电影、电视剧、综艺节目中手工截取视频片段,之后通过注释人员对截取视频片段的标注最终得到了1 839个视频片段。这些视频片段经过人物检测、人脸检测等操作获得有效的视频帧。该数据集包含7种基础情感和3种模态：人脸表情,身体姿态,情感语音。结果 为了提供情感识别的基准,在本文的实验部分,利用机器学习和深度学习方法对MED数据集进行了评估。首先与CK+数据集进行了对比实验,结果表明使用实验室环境下收集的数据开发算法很难应用到实际中,然后对各个模态进行了基线实验,并给出了各个模态的基线。最后多模态融合的实验结果相对于单模态的人脸表情识别提高了4.03%。结论 多模态情感数据库MED扩充了现有的真实环境下多模态数据库,以推进跨文化（语言）情感识别和对不同情感评估的感知分析等方向的研究,提高自动情感计算系统在现实应用中的表现。     

维度语音情感识别研究综述

情感识别是多学科交叉的研究方向,涉及认知科学、心理学、信号处理、模式识别、人工智能等领域的研究热点,目的是使机器理解人类情感状态,进而实现自然人机交互.本文首先从心理学及认知学角度介绍了语音情感认知研究进展,详细介绍了情感的认知理论、维度理论、脑机制以及基于情感理论的计算模型,旨在为语音情感识别提供科学的情感理论模型.然后,从人工智能角度系统地总结了目前维度情感识别的研究现状和发展,包括语音维度情感数据库、特征提取、识别算法等技术要点.最后,分析了维度情感识别技术目前面临的挑战以及可能的解决思路,对未来研究方向进行了展望.     

基于情感轮和情感词典的文本情感分布标记增强方法

情感分布学习是一种近年提出的用于处理存在情绪模糊性的多情绪分析模型,其核心思路是通过情感分布记录示例在各个情绪上的表达程度.不同于传统的单标记或多标记学习,情感分布学习可以定量地对多个情绪同时建模.目前,情感分布学习面临的一个重要困难是缺乏已标注情感分布的文本数据集.为了利用大量已有的单标记情感数据集,情感分布标记增强方法可以将示例的情绪标签增强为情感分布.基于文本中的情感词蕴含着大量情感信息的特点,本文在引入普鲁契克情感轮心理学模型的基础上,提出基于情感轮和情感词典的情感分布标记增强方法(Emotion Wheel and Lexicon based emotion distribution Label Enhancement,EWLLE).EWLLE方法基于情绪的心理学距离为句子的真实情绪标签和情感词的情绪标签分别生成离散高斯分布,然后通过分布的叠加将两种信息综合为统一的情感分布.在7个常用的中英文文本情感数据集上的对比实验表明,EWLLE方法在情绪识别任务上的性能优于已有的情感分布标记增强方法.     

语音情感的维度特征提取与识别

研究了情绪的维度空间模型与语音声学特征之间的关系以及语音情感的自动识别方法。介绍了基本情绪的维度空间模型,提取了唤醒度和效价度对应的情感特征,采用全局统计特征减小文本差异对情感特征的影响。研究了生气、高兴、悲伤和平静等情感状态的识别,使用高斯混合模型进行4种基本情感的建模,通过实验设定了高斯混合模型的最佳混合度,从而较好地拟合了4种情感在特征空间中的概率分布。实验结果显示,选取的语音特征适合于基本情感类别的识别,高斯混合模型对情感的建模起到了较好的效果,并且验证了二维情绪空间中,效价维度上的情感特征对语音情感识别的重要作用。     

情感语音数据库综述

情感语音数据库是语音情感识别研究的数据基础,为语音情感识别模型的建立提供训练和测试数据.近年来,国内外研究者们以各自的科研任务为背景,建立了若干面向语音情感识别研究的数据库.然而,由于情感的复杂性以及缺乏统一的数据库建立标准,只有少数的高质量的数据库得到了大多数研究者们的认可.通过文献调查与分析,对当前领域内极具代表性的一些情感语音数据库进行了综述,目的是为相关研究者们的数据库建立和选择工作提供可靠的对比和参考依据.     

基于视听觉感知系统的多模态情感识别

情绪识别作为人机交互的热门领域,其技术已经被应用于医学、教育、安全驾驶、电子商务等领域.情绪主要由面部表情、声音、话语等进行表达,不同情绪表达时的面部肌肉、语气、语调等特征也不相同,使用单一模态特征确定的情绪的不准确性偏高,考虑到情绪表达主要通过视觉和听觉进行感知,本文提出了一种基于视听觉感知系统的多模态表情识别算法,分别从语音和图像模态出发,提取两种模态的情感特征,并设计多个分类器为单特征进行情绪分类实验,得到多个基于单特征的表情识别模型.在语音和图像的多模态实验中,提出了晚期融合策略进行特征融合,考虑到不同模型间的弱依赖性,采用加权投票法进行模型融合,得到基于多个单特征模型的融合表情识别模型.本文使用AFEW数据集进行实验,通过对比融合表情识别模型与单特征的表情识别模型的识别结果,验证了基于视听觉感知系统的多模态情感识别效果要优于基于单模态的识别效果.     

基于姿态的情感计算综述

情感计算的理论与算法研究是近年来人机交互领域的热点话题.当前,常见的情感计算集中在基于面部表情、语音、文本、人体姿态等方向,既有单一模态的算法,又有多模态的综合算法.基于面部表情和语音模态的算法占据多数,国内外基于人体姿态的算法相对较少.文中针对基于姿态的情感计算所面临的几个关键科学问题展开了综述,包括情感的心理学模型、人体姿态估计算法、姿态的情感特征提取算法、情感分类与标注算法、姿态情感数据集、基于姿态的情感识别算法等.具体来说,首先介绍了几种常用的情感计算心理学模型,评述了各类模型的适用场景;随后从人体检测和姿态估计2个角度对人体姿态估计的常用算法进行了总结,并讨论了2D和3D姿态估计的应用前景.针对特征提取算法,分析了基于全身和上半身身体动作的姿态特征提取算法.在情感标注方面,介绍了表演数据和非表演数据的情感标注算法,并指出了半自动或自动的标注非表演数据将是未来的重要发展趋势之一.针对姿态情感数据集,列举了近年来常见的14个数据集,并主要从是否是表演数据、数据维度、静态或动态姿势、全身或非全身数据等几个方面进行了总结.在基于姿态的情感识别算法方面,主要介绍了基于人工神经网络的情感识别算法,指出了不同算法的优劣之处和适用的数据集类型.文中的综述研究,总结提炼了国内外该领域经典且前沿的工作,希望为相关的研究者提供研究帮助.     

Recognizing learning emotion based on convolutional neural networks and transfer learning

Learning effectiveness is normally analyzed by data collection through tests or questionnaires. However, instant feedback is usually not available. Learners’ facial emotion and learning motivation has a positive relationship. Therefore, the system identifying learners’ facial emotions can provide feedback that teachers can understand students’ learning situation and provide help or improve teaching strategy. Studies have found that convolutional neural networks provide a good performance in basic facial emotion recognition. Convolutional neural networks do not require manual design features like traditional machine learning, they automatically learn the necessary features of the entire image. This article improves the FaceLiveNet network with low and high accuracy in basic emotion recognition, and proposes the framework of Dense_FaceLiveNet. We use Dense_FaceLiveNet for two-phases of transfer learning. First, from the relatively simple data JAFFE and KDEF basic emotion recognition model transferring to the FER2013 basic emotion dataset and obtained an accuracy of 70.02%. Secondly, using the FER2013 basic emotion recognition model transferring to learning emotion recognition model, the test accuracy rate is as high as 91.93%, which is 12.9% higher than the accuracy rate of 79.03% without using the transfer learning model, which proves that the use of transfer learning can effectively improve the recognition accuracy of learning emotion recognition model. In addition, in order to test the generalization ability of the Learning Emotion Recognition Model, videos recorded by students from a national university in Taiwan during class learning were used as test data. The original database of learning emotions did not consider that students would have exceptions such as over eyebrows, eyes closed and hand hold the chin etc. To improve this situation, after adding the learning emotion database to the images of the exceptions mentioned above, the model was rebuilt, and the recognition accuracy rate of the model was 92.42%. By comparing the output of maps, the rebuilt model does have the characteristics of success in learning images such as eyebrows, chins, and eyes closed. Furthermore, after combining all the students’ image data with the original learning emotion database, the model was rebuilt and obtained the accuracy rate reached 84.59%. The result proves that the Learning Emotion Recognition Model can achieve high recognition accuracy by processing the unlearned image through transfer learning. The main contribution is to design two-phase transfer learning for establishing the learning emotion recognition model and overcome the problem for small amounts of learning emotion data. Our experiment results have shown the performance improvement of two-phase transfer learning.     

A Facial Expression Emotion Recognition Based Human-robot Interaction System

A facial expression emotion recognition based human-robot interaction (FEER-HRI) system is proposed, for which a four-layer system framework is designed. The FEERHRI system enables the robots not only to recognize human emotions, but also to generate facial expression for adapting to human emotions. A facial emotion recognition method based on 2D-Gabor, uniform local binary pattern (LBP) operator, and multiclass extreme learning machine (ELM) classifier is presented, which is applied to real-time facial expression recognition for robots. Facial expressions of robots are represented by simple cartoon symbols and displayed by a LED screen equipped in the robots, which can be easily understood by human. Four scenarios, i.e., guiding, entertainment, home service and scene simulation are performed in the human-robot interaction experiment, in which smooth communication is realized by facial expression recognition of humans and facial expression generation of robots within 2 seconds. As a few prospective applications, the FEERHRI system can be applied in home service, smart home, safe driving, and so on.      

Emotion recognition from thermal infrared images using deep Boltzmann machine

Facial expression and emotion recognition from thermal infrared images has attracted more and more attentions in recent years. However, the features adopted in current work are either temperature statistical parameters extracted from the facial regions of interest or several hand-crafted features that are commonly used in visible spectrum. Till now there are no image features specially designed for thermal infrared images. In this paper, we propose using the deep Boltzmann machine to learn thermal features for emotion recognition from thermal infrared facial images. First, the face is located and normalized from the thermal infrared images. Then, a deep Boltzmann machine model composed of two layers is trained. The parameters of the deep Boltzmann machine model are further fine-tuned for emotion recognition after pre-training of feature learning. Comparative experimental results on the NVIE database demonstrate that our approach outperforms other approaches using temperature statistic features or hand-crafted features borrowed from visible domain. The learned features from the forehead, eye, and mouth are more effective for discriminating valence dimension of emotion than other facial areas. In addition, our study shows that adding unlabeled data from other database during training can also improve feature learning performance.     

基于粗集的表情识别在电力学习中的应用

本文提出了一种基于粗集的自动表情识别系统(RAFERS),该系统首先对人脸表情进行预处理,然后依次进行特征提取、特征选择、训练情感分类模型,并将该系统实际应用于电力知识学习系统,对电力知识学习的用户进行人脸表情的自动识别,进而为用户提供个性化的服务。     

Emotion recognition from geometric facial features using self-organizing map

This paper presents a novel emotion recognition model using the system identification approach. A comprehensive data driven model using an extended Kohonen self-organizing map (KSOM) has been developed whose input is a 26 dimensional facial geometric feature vector comprising eye, lip and eyebrow feature points. The analytical face model using this 26 dimensional geometric feature vector has been effectively used to describe the facial changes due to different expressions. This paper thus includes an automated generation scheme of this geometric facial feature vector. The proposed non-heuristic model has been developed using training data from MMI facial expression database. The emotion recognition accuracy of the proposed scheme has been compared with radial basis function network, multi-layered perceptron model and support vector machine based recognition schemes. The experimental results show that the proposed model is very efficient in recognizing six basic emotions while ensuring significant increase in average classification accuracy over radial basis function and multi-layered perceptron. It also shows that the average recognition rate of the proposed method is comparatively better than multi-class support vector machine.     

An efficient deep learning technique for facial emotion recognition

Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion recognition from facial images but face the issue of performance degradation due to poor selection of layers in the convolutional neural network model.To address this issue, we propose an efficient deep learning technique using a convolutional neural network model for classifying emotions from facial images and detecting age and gender from the facial expressions efficiently. Experimental results show that the proposed model outperformed baseline works by achieving an accuracy of 95.65% for emotion recognition, 98.5% for age recognition, and 99.14% for gender recognition.

     

阅读严肃文学小说对脸部情绪识别的影响——ERP证据

该研究采用事件相关电位(ERP)观察了被试在识别积极、中性和消极的脸部情绪时,在大脑颞枕部电极点上引发的N170效应,来探索阅读严肃文学小说是否会影响人对他人情绪的反应。阅读组被试在两次脸部情绪识别测试之间阅读严肃文学小说,而对照组没有。第二次测试相比第一次测试,N170的幅度增大,但是阅读严肃文学小说会抑制N170幅度增益,且对情绪越积极的刺激图片抑制越大。据此,阅读对他人脸部情绪的识别确有影响。研究推测阅读可能抑制大脑中的脸部情绪特异性,进而可能提高对脸部情绪的感知力。