Emotional facial expression model building

Human face plays a crucial role in interpersonal communication. If we synthesize vivid expressional face in cyberspace, we could make the interaction between computer and human more natural and friendly. In this paper, we present a simple methodology for mimicking realistic face by manipulating emotional states. Compared with traditional methods of facial expression synthesis, our approach takes three advantages at the same time. They are (1) generating facial expressions under quantitative control of emotional states, (2) rendering shape and illumination changes on face simultaneously and (3) synthesizing expressional face for any new person by only utilizing a neutral face image. We have discussed the implementation method in the paper and demonstrated the effects of our approach by using a series of interesting experiments, such as predicting unseen expressions for an unfamiliar person, simulating one’s facial expressions with someone else’s style, extracting pure emotional expressions from the admixtures.     

联合多任务学习的人脸超分辨率重建

目的 人脸超分辨率重建是特定应用领域的超分辨率问题,为了充分利用面部先验知识,提出一种基于多任务联合学习的深度人脸超分辨率重建算法。方法 首先使用残差学习和对称式跨层连接网络提取低分辨率人脸的多层次特征,根据不同任务的学习难易程度设置损失权重和损失阈值,对网络进行多属性联合学习训练。然后使用感知损失函数衡量HR（high-resolution）图像与SR（super-resolution）图像在语义层面的差距,并论证感知损失在提高人脸语义信息重建效果方面的有效性。最后对人脸属性数据集进行增强,在此基础上进行联合多任务学习,以获得视觉感知效果更加真实的超分辨率结果。结果 使用峰值信噪比（PSNR）和结构相似度（SSIM）两个客观评价标准对实验结果进行评价,并与其他主流方法进行对比。实验结果显示,在人脸属性数据集（CelebA）上,在放大8倍时,与通用超分辨率MemNet（persistent memory network）算法和人脸超分辨率FSRNet（end-to-end learning face super-resolution network）算法相比,本文算法的PSNR分别提升约2.15 dB和1.2 dB。结论 实验数据与效果图表明本文算法可以更好地利用人脸先验知识,产生在视觉感知上更加真实和清晰的人脸边缘和纹理细节。     

基于生成式对抗网络的鲁棒人脸表情识别

人们在自然情感交流中经常伴随着头部旋转和肢体动作,它们往往导致较大范围的人脸遮挡,使得人脸图像损失部分表情信息.现有的表情识别方法大多基于通用的人脸特征和识别算法,未考虑表情和身份的差异,导致对新用户的识别不够鲁棒.本文提出了一种对人脸局部遮挡图像进行用户无关表情识别的方法.该方法包括一个基于Wasserstein生成式对抗网络（Wasserstein generative adversarial net,WGAN）的人脸图像生成网络,能够为图像中的遮挡区域生成上下文一致的补全图像;以及一个表情识别网络,能够通过在表情识别任务和身份识别任务之间建立对抗关系来提取用户无关的表情特征并推断表情类别.实验结果表明,我们的方法在由CK+,Multi-PIE和JAFFE构成的混合数据集上用户无关的平均识别准确率超过了90%.在CK+上用户无关的识别准确率达到了96%,其中4.5%的性能提升得益于本文提出的对抗式表情特征提取方法.此外,在45°头部旋转范围内,本文方法还能够用于提高非正面表情的识别准确率.     

Illumination normalization of facial images by reversing the process of image formation

Variations in illumination degrade the performance of appearance based face recognition. We present a novel algorithm for the normalization of color facial images using a single image and its co-registered 3D pointcloud (3D image). The algorithm borrows the physically based Phong’s lighting model from computer graphics which is used for rendering computer images and employs it in a reverse mode for the calculation of face albedo from real facial images. Our algorithm estimates the number of the dominant light sources and their directions from the specularities in the facial image and the corresponding 3D points. The intensities of the light sources and the parameters of the Phong’s model are estimated by fitting the Phong’s model onto the facial skin data. Unlike existing approaches, our algorithm takes into account both Lambertian and specular reflections as well as attached and cast shadows. Moreover, our algorithm is invariant to facial pose and expression and can effectively handle the case of multiple extended light sources. The algorithm was tested on the challenging FRGC v2.0 data and satisfactory results were achieved. The mean fitting error was 6.3% of the maximum color value. Performing face recognition using the normalized images increased both identification and verification rates.     

Natural facial expression recognition using differential-AAM and manifold learning

This paper proposes a novel natural facial expression recognition method that recognizes a sequence of dynamic facial expression images using the differential active appearance model (AAM) and manifold learning as follows. First, the differential-AAM features (DAFs) are computed by the difference of the AAM parameters between an input face image and a reference (neutral expression) face image. Second, manifold learning embeds the DAFs on the smooth and continuous feature space. Third, the input facial expression is recognized through two steps: (1) computing the distances between the input image sequence and gallery image sequences using directed Hausdorff distance (DHD) and (2) selecting the expression by a majority voting of k-nearest neighbors (k-NN) sequences in the gallery. The DAFs are robust and efficient for the facial expression analysis due to the elimination of the inter-person, camera, and illumination variations. Since the DAFs treat the neutral expression image as the reference image, the neutral expression image must be found effectively. This is done via the differential facial expression probability density model (DFEPDM) using the kernel density approximation of the positively directional DAFs changing from neutral to angry (happy, surprised) and negatively directional DAFs changing from angry (happy, surprised) to neutral. Then, a face image is considered to be the neutral expression if it has the maximum DFEPDM in the input sequences. Experimental results show that (1) the DAFs improve the facial expression recognition performance over conventional AAM features by 20% and (2) the sequence-based k-NN classifier provides a 95% facial expression recognition performance on the facial expression database (FED06).     

Wavelets-based facial expression recognition using a bank of support vector machines

A human face does not play its role in the identification of an individual but also communicates useful information about a person’s emotional state at a particular time. No wonder automatic face expression recognition has become an area of great interest within the computer science, psychology, medicine, and human–computer interaction research communities. Various feature extraction techniques based on statistical to geometrical data have been used for recognition of expressions from static images as well as real-time videos. In this paper, we present a method for automatic recognition of facial expressions from face images by providing discrete wavelet transform features to a bank of seven parallel support vector machines (SVMs). Each SVM is trained to recognize a particular facial expression, so that it is most sensitive to that expression. Multi-classification is achieved by combining multiple SVMs performing binary classification using one-against-all approach. The outputs of all SVMs are combined using a maximum function. The classification efficiency is tested on static images from the publicly available Japanese Female Facial Expression database. The experiments using the proposed method demonstrate promising results.     

面向非受控场景的人脸图像正面化重建

目的 人脸正面化重建是当前视觉领域的热点问题。现有方法对于模型的训练数据具有较高的需求,如精确的输入输出图像配准、完备的人脸先验信息等。但该类数据采集成本较高,可应用的数据集规模较小,直接将现有方法应用于真实的非受控场景中往往难以取得理想表现。针对上述问题,提出了一种无图像配准和先验信息依赖的任意视角人脸图像正面化重建方法。方法 首先提出了一种具有双输入路径的人脸编码网络,分别用于学习输入人脸的视觉表征信息以及人脸的语义表征信息,两者联合构造出更加完备的人脸表征模型。随后建立了一种多类别表征融合的解码网络,通过以视觉表征为基础、以语义表征为引导的方式对两种表征信息进行融合,融合后的信息经过图像解码即可得到最终的正面化人脸图像重建结果。结果 首先在Multi-PIE(multi-pose, illumination and expression)数据集上与8种较先进方法进行了性能评估。定量和定性的实验结果表明,所提方法在客观指标以及视觉质量方面均优于对比方法。此外,相较于当前性能先进的基于光流的特征翘曲模型(flow-based feature warping model,FFWM)方法,本文方法能够节省79%的参数量和42%的计算操作数。进一步基于CASIA-WebFace(Institute of Automation, Chinese Academy of Sciences—WebFace)数据集对所提出方法在真实非受控场景中的表现进行了评估,识别精度超过现有方法10%以上。结论 本文提出的双层级表征集成推理网络,能够挖掘并联合人脸图像的底层视觉特征以及高层语义特征,充分利用图像自身信息,不仅以更低的计算复杂度取得了更优的视觉质量和身份识别精度,而且在非受控的场景下同样展现出了出色的泛化性能。     

结合形变模型与图像修复的人脸姿态矫正

目的 人脸姿态偏转是影响人脸识别准确率的一个重要因素,本文利用3维人脸重建中常用的3维形变模型以及深度卷积神经网络,提出一种用于多姿态人脸识别的人脸姿态矫正算法,在一定程度上提高了大姿态下人脸识别的准确率。方法 对传统的3维形变模型拟合方法进行改进,利用人脸形状参数和表情参数对3维形变模型进行建模,针对面部不同区域的关键点赋予不同的权值,加权拟合3维形变模型,使得具有不同姿态和面部表情的人脸图像拟合效果更好。然后,对3维人脸模型进行姿态矫正并利用深度学习对人脸图像进行修复,修复不规则的人脸空洞区域,并使用最新的局部卷积技术同时在新的数据集上重新训练卷积神经网络,使得网络参数达到最优。结果 在LFW（labeled faces in the wild）人脸数据库和StirlingESRC（Economic Social Research Council）3维人脸数据库上,将本文算法与其他方法进行比较,实验结果表明,本文算法的人脸识别精度有一定程度的提高。在LFW数据库上,通过对具有任意姿态的人脸图像进行姿态矫正和修复后,本文方法达到了96.57%的人脸识别精确度。在StirlingESRC数据库上,本文方法在人脸姿态为±22°的情况下,人脸识别准确率分别提高5.195%和2.265%;在人脸姿态为±45°情况下,人脸识别准确率分别提高5.875%和11.095%;平均人脸识别率分别提高5.53%和7.13%。对比实验结果表明,本文提出的人脸姿态矫正算法有效提高了人脸识别的准确率。结论 本文提出的人脸姿态矫正算法,综合了3维形变模型和深度学习模型的优点,在各个人脸姿态角度下,均能使人脸识别准确率在一定程度上有所提高。     

Facial expression recognition and tracking for intelligent human-robot interaction

For effective interaction between humans and socially adept, intelligent service robots, a key capability required by this class of sociable robots is the successful interpretation of visual data. In addition to crucial techniques like human face detection and recognition, an important next step for enabling intelligence and empathy within social robots is that of emotion recognition. In this paper, an automated and interactive computer vision system is investigated for human facial expression recognition and tracking based on the facial structure features and movement information. Twenty facial features are adopted since they are more informative and prominent for reducing the ambiguity during classification. An unsupervised learning algorithm, distributed locally linear embedding (DLLE), is introduced to recover the inherent properties of scattered data lying on a manifold embedded in high-dimensional input facial images. The selected person-dependent facial expression images in a video are classified using the DLLE. In addition, facial expression motion energy is introduced to describe the facial muscle’s tension during the expressions for person-independent tracking for person-independent recognition. This method takes advantage of the optical flow which tracks the feature points’ movement information. Finally, experimental results show that our approach is able to separate different expressions successfully.     

Facial expression recognition of a speaker using front-view face judgment, vowel judgment, and thermal image processing

For facial expression recognition, we selected three images: (i) just before speaking, (ii) speaking the first vowel, and (iii) speaking the last vowel in an utterance. In this study, as a pre-processing module, we added a judgment function to distinguish a front-view face for facial expression recognition. A frame of the front-view face in a dynamic image is selected by estimating the face direction. The judgment function measures four feature parameters using thermal image processing, and selects the thermal images that have all the values of the feature parameters within limited ranges which were decided on the basis of training thermal images of front-view faces. As an initial investigation, we adopted the utterance of the Japanese name “Taro,” which is semantically neutral. The mean judgment accuracy of the front-view face was 99.5% for six subjects who changed their face direction freely. Using the proposed method, the facial expressions of six subjects were distinguishable with 84.0% accuracy when they exhibited one of the intentional facial expressions of “angry,” “happy,” “neutral,” “sad,” and “surprised.” We expect the proposed method to be applicable for recognizing facial expressions in daily conversation.     

人脸局部特征增强的亲属关系验证方法

亲属关系验证是人脸识别的一个重要分支,可以用于寻找失散亲人、搜寻走失儿童、构建家庭图谱、社交媒体分析等重要场景。父母和孩子的人脸图像之间往往存在较大的差异,如何从人脸中提取到有鉴别力的特征是提高亲属关系验证准确率的关键。因此,提出了一种基于深度学习和人脸局部特征增强的亲属关系验证方法,构建了人脸局部特征增强验证网络（Local Facial Feature Enhancement Verification Net,LFFEV Net）,获取用于亲属关系验证的具有强鉴别力的人脸特征表示。LFFEV Net由局部特征注意力网络和残差验证网络两部分组成。局部特征注意力网络提取人脸局部关键特征,将获取的局部关键特征和对应的原始图像一同输入到残差验证网络中获取更具鉴别力的人脸特征,将特征经过融合并结合Family ID信息进行亲属关系验证。算法在公开的亲属关系数据集KinFaceW-I和KinFaceW-II上进行测试,实验结果表明,所设计的方法在亲属关系验证任务中有较高的识别率。     

基于组合学习的人脸超分辨率算法

现有的基于深度学习的人脸超分辨算法大部分仅仅利用一种网络分区重建高分辨率输出图像,并未考虑人脸图像中的结构性信息,导致了在人脸的重要器官重建上缺乏足够的细节信息。针对这一问题,提出一种基于组合学习的人脸超分辨率算法。该算法独立采用不同深度学习模型的优势重建感兴趣的区域,由此在训练网络的过程中每个人脸区域的数据分布不同,不同的子网络能够获得更精确的先验信息。首先,对人脸图像采用超像素分割算法生成人脸组件部分和人脸背景图像;然后,采用人脸组件生成对抗网络（C-GAN）独立重建人脸组件图像块,并采用人脸背景重建网络生成人脸背景图像;其次,使用人脸组件融合网络将两种不同模型重建的人脸组件图像块自适应融合;最后,将生成的人脸组件图像块合并至人脸背景图像中,重建出最终的人脸图像。在FEI数据集上的实验结果表明,与人脸图像超分辨率算法通过组件生成和增强学习幻构人脸图像（LCGE）及判决性增强的生成对抗网络（EDGAN）相比,所提算法的峰值信噪比（PSNR）值分别高出1.23 dB和1.11 dB。所提算法能够采用不同深度学习模型的优势组合学习重建更精准的人脸图像,同时拓展了图像重建先验的来源。     

深度多尺度融合注意力残差人脸表情识别网络

针对人脸表情呈现方式多样化以及人脸表情识别易受光照、姿势、遮挡等非线性因素影响的问题,提出了一种深度多尺度融合注意力残差网络(deep multi-scale fusion attention residual network, DMFA-ResNet)。该模型基于ResNet-50残差网络,设计了新的注意力残差模块,由7个具有三条支路的注意残差学习单元构成,能够对输入图像进行并行多卷积操作,以获得多尺度特征,同时引入注意力机制,突出重点局部区域,有利于遮挡图像的特征学习。通过在注意力残差模块之间增加过渡层以去除冗余信息,简化网络复杂度,在保证感受野的情况下减少计算量,实现网络抗过拟合效果。在3组数据集上的实验结果表明,本文提出的算法均优于对比的其他先进方法。     

融合参考图像的人脸超分辨率重构方法

基于深度学习的图像超分辨率重构方法对低分辨率人脸图像进行超分辨率重构时,通常存在重构图像模糊和重构图像与真实图像差异较大等问题.基于此问题,文中提出融合参考图像的人脸超分辨率重构方法,可以实现对低分辨率人脸图像的有效重构.参考图像特征提取子网提取参考图像的多尺度特征,保留人脸神态和重点部位的细节特征信息,去除人脸轮廓和面部表情等冗余信息.基于提取的参考图像多尺度特征,逐级超分主网络对低分辨率人脸图像特征进行逐次填充,最终重构生成高分辨率的人脸图像.在数据集上的实验表明,文中方法可以实现对低分辨率人脸图像的有效重构,具有良好的鲁棒性.     

A system for synchronizing the nods of a CG character and a human using thermal image processing and a moving-average model

The purpose of this study was to develop a system for communication between a human and a computer generated (CG) character to make him or her more peaceful and cheerful. Nonverbal communication using such things as facial expression, a nod, or a hand gesture is very important for reciprocal communication between humans. In this study, an image registered by infrared rays which describes the thermal distribution of the face and neck has been used to develop a system for communication between a human and a CG character. The CG character can synchronize its nod with a person’s nod by predicting his or her nod angle. The measured feature parameter is input to a fuzzy algorithm system to obtain the nod angle of a person in front of an infrared camera, and then a moving-average model is used to predict the nod angle of the person. The average error of the nod angle obtained by the system has been estimated as about 5°. The CG character nods its head, not only when the human nods his or her head, but also when the human shakes his or her head to the left or right.     

Deep peak-neutral difference feature for facial expression recognition

Facial expression recognition (FER) is important in vision-related applications. Deep neural networks demonstrate impressive performance for face recognition; however, it should be noted that this method relies heavily on a great deal of manually labeled training data, which is not available for facial expressions in real-world applications. Hence, we propose a powerful facial feature called deep peak–neutral difference (DPND) for FER. DPND is defined as the difference between two deep representations of the fully expressive (peak) and neutral facial expression frames. The difference tends to emphasize the facial parts that are changed in the transition from the neutral to the expressive face and to eliminate the face identity information retained in the fine-tuned deep neural network for facial expression, the network has been trained on large-scale face recognition dataset. Furthermore, unsupervised clustering and semi-supervised classification methods are presented to automatically acquire the neutral and peak frames from the expression sequence. The proposed facial expression feature achieved encouraging results on public databases, which suggests that it has strong potential to recognize facial expressions in real-world applications.

     

Extracting spatiotemporal human activity patterns in assisted living using a home sensor network

This paper presents an automated methodology for extracting the spatiotemporal activity model of a person using a wireless sensor network deployed inside a home. The sensor network is modeled as a source of spatiotemporal symbols whose output is triggered by the monitored person’s motion over space and time. Using this stream of symbols, the problem of human activity modeling is formulated as a spatiotemporal pattern-matching problem on top of the sequence of symbolic information the sensor network produces, and is solved using an exhaustive search algorithm. The effectiveness of the proposed methodology is demonstrated on a real 30-day dataset extracted from an ongoing deployment of a sensor network inside a home monitoring an elder. The developed algorithm examines the person’s data over these 30 days and automatically extracts the person’s daily pattern.     

Face alive icon

In this paper, we propose a methodology to synthesize facial expressions from photographs for devices with limited processing power, network bandwidth and display area, which is referred as “LLL” environment. The facial images are reduced to small-sized face alive icons (FAI). Expressions are decomposed into the expression-unrelated facial features and the expression-related expressional features. As a result, the common features can be identified and reused across expressions using a discrete model constructed from the statistical analysis on training dataset. Semantic synthesis rules are introduced to reveal the inner relations of expressions. Verified by the experimental prototype system and usability study, the approach can produce acceptable facial expression images utilizing much less computing, network and storage resource than the traditional approaches.     

Three-dimensional view-invariant face recognition using a hierarchical pose-normalization strategy

Face recognition from three-dimensional (3D) shape data has been proposed as a method of biometric identification as a way of either supplanting or reinforcing a two-dimensional approach. This paper presents a 3D face recognition system capable of recognizing the identity of an individual from a 3D facial scan in any pose across the view-sphere, by suitably comparing it with a set of models (all in frontal pose) stored in a database. The system makes use of only 3D shape data, ignoring textural information completely. Firstly, we propose a generic learning strategy using support vector regression [Burges, Data Mining Knowl Discov 2(2): 121–167, 1998] to estimate the approximate pose of a 3D head. The support vector machine (SVM) is trained on range images in several poses belonging to only a small set of individuals and is able to coarsely estimate the pose of any unseen facial scan. Secondly, we propose a hierarchical two-step strategy to normalize a facial scan to a nearly frontal pose before performing any recognition. The first step consists of either a coarse normalization making use of facial features or the generic learning algorithm using the SVM. This is followed by an iterative technique to refine the alignment to the frontal pose, which is basically an improved form of the Iterated Closest Point Algorithm [Besl and Mckay, IEEE Trans Pattern Anal Mach Intell 14(2):239–256, 1992]. The latter step produces a residual error value, which can be used as a metric to gauge the similarity between two faces. Our two-step approach is experimentally shown to outperform both of the individual normalization methods in terms of recognition rates, over a very wide range of facial poses. Our strategy has been tested on a large database of 3D facial scans in which the training and test images of each individual were acquired at significantly different times, unlike all except two of the existing 3D face recognition methods.