基于NURBS向量肌肉模型的人脸动画及实现

当前,动画及其实现技术受到业界广泛关注,而人脸动画如喜、怒、哀、乐的表达其真实感还不够好.以Waters肌肉模型为基础,提出NURBS弹性肌肉模型,该方法依据解剖学知识,用非均匀有理B样条曲线仿真肌肉.通过改变曲线控制点的权重,可以找到一个动作向量控制肌肉的运动,进而合成人脸的各种表情.控制点数量越多,肌肉就越好控制,那么就可以更加真实地仿真人脸表情.     

Linear subspaces for facial expression recognition

This paper presents a method for the recognition of the six basic facial expressions in images or in image sequences using landmark points. The proposed technique relies on the observation that the vectors formed by the landmark point coordinates belong to a different manifold for each of the expressions. In addition experimental measurements validate the hypothesis that each of these manifolds can be decomposed to a small number of linear subspaces of very low dimension. This yields a parameterization of the manifolds that allows for computing the distance of a feature vector from each subspace and consequently from each one of the six manifolds. Two alternative classifiers are next proposed that use the corresponding distances as input: the first one is based on the minimum distance from the manifolds, while the second one uses SVMs that are trained with the vector of all distances from each subspace. The proposed technique is tested for two scenarios, the subject-independent and the subject-dependent one. Extensive experiments for each scenario have been performed on two publicly available datasets yielding very satisfactory expression recognition accuracy.     

Internet communication using real-time facial expression analysis and synthesis

In this paper, the authors have developed a system that animates 3D facial agents based on real-time facial expression analysis techniques and research on synthesizing facial expressions and text-to-speech capabilities. This system combines visual, auditory, and primary interfaces to communicate one coherent multimodal chat experience. Users can represent themselves using agents they select from a group that we have predefined. When a user shows a particular expression while typing a text, the 3D agent at the receiving end speaks the message aloud while it replays the recognized facial expression sequences and also augments the synthesized voice with appropriate emotional content. Because the visual data exchange is based on the MPEG-4 high-level Facial Animation Parameter for facial expressions (FAP 2), rather than real-time video, the method requires very low bandwidth.     

Entropy-based feature selection for improved 3D facial expression recognition

Facial expressions contain most of the information on human face which is essential for human–computer interaction. Development of robust algorithms for automatic recognition of facial expressions with high recognition rates has been a challenge for the last 10 years. In this paper, we propose a novel feature selection procedure which recognizes basic facial expressions with high recognition rates by utilizing three-Dimensional (3D) geometrical facial feature positions. The paper presents a system of classifying expressions in one of the six basic emotional categories which are anger, disgust, fear, happiness, sadness, and surprise. The paper contributes on feature selections for each expression independently and achieves high recognition rates with the proposed geometric facial features selected for each expression. The novel feature selection procedure is entropy based, and it is employed independently for each of the six basic expressions. The system’s performance is evaluated using the 3D facial expression database, BU-3DFE. Experimental results show that the proposed method outperforms the latest methods reported in the literature.     

自适应卷积神经网络在面部表情识别中的应用

针对无法对面部表情进行精确识别的问题,提出了基于ResNet50网络融合双线性混合注意力机制的网络模型。针对传统池化算法造成图像特征提取残缺、模糊等问题,提出了一种基于Average-Pooling算法的自适应池化权重算法,同时基于粒子群算法对卷积神经网络模型超参数进行自适应调节,从而进一步提升模型识别精度。基于改进的网络模型,设计了一款实时面部表情识别系统。经验证,在Fer2013数据集和CK+数据集上,改进的模型在测试集中的识别精度分别为73.51%和99.86%。     

Expression-invariant representations of faces.

Addressed here is the problem of constructing and analyzing expression-invariant representations of human faces. We demonstrate and justify experimentally a simple geometric model that allows to describe facial expressions as isometric deformations of the facial surface. The main step in the construction of expression-invariant representation of a face involves embedding of the facial intrinsic geometric structure into some low-dimensional space. We study the influence of the embedding space geometry and dimensionality choice on the representation accuracy and argue that compared to its Euclidean counterpart, spherical embedding leads to notably smaller metric distortions. We experimentally support our claim showing that a smaller embedding error leads to better recognition.     

基于人脸相似度加权距离的非特定人表情识别

该文提出了一种用于非特定人表情识别的方法。首先,对测试人的初始表情特征进行高阶奇异值分解,得到测试人与训练集中所有人相关的表情特征。然后,根据相似的人有相似的表情的假设,计算人脸相似度加权距离,作为测试人的表情特征与标准的表情特征之间的相似性测度。通过加权的过程,可以有效地去除由于个体差异而造成的表情特征的差异,提高非特定人表情识别的鲁棒性。该文提出的方法在JAFFE数据库上进行了测试。对非特定人的表情识别实验表明,该文方法比传统的方法在识别率上有了提高。     

Cross-dataset emotion recognition from facial expressions through convolutional neural networks

The face is the window to the soul. This is what the 19th-century French doctor Duchenne de Boulogne thought. Using electric shocks to stimulate muscular contractions and induce bizarre-looking expressions, he wanted to understand how muscles produce facial expressions and reveal the most hidden human emotions. Two centuries later, this research field remains very active. We see automatic systems for recognizing emotion and facial expression being applied in medicine, security and surveillance systems, advertising and marketing, among others. However, there are still fundamental questions that scientists are trying to answer when analyzing a person’s emotional state from their facial expressions. Is it possible to reliably infer someone’s internal state based only on their facial muscles’ movements? Is there a universal facial setting to express basic emotions such as anger, disgust, fear, happiness, sadness, and surprise? In this research, we seek to address some of these questions through convolutional neural networks. Unlike most studies in the prior art, we are particularly interested in examining whether characteristics learned from one group of people can be generalized to predict another’s emotions successfully. In this sense, we adopt a cross-dataset evaluation protocol to assess the performance of the proposed methods. Our baseline is a custom-tailored model initially used in face recognition to categorize emotion. By applying data visualization techniques, we improve our baseline model, deriving two other methods. The first method aims to direct the network’s attention to regions of the face considered important in the literature but ignored by the baseline model, using patches to hide random parts of the facial image so that the network can learn discriminative characteristics in different regions. The second method explores a loss function that generates data representations in high-dimensional spaces so that examples of the same emotion class are close and examples of different classes are distant. Finally, we investigate the complementarity between these two methods, proposing a late-fusion technique that combines their outputs through the multiplication of probabilities. We compare our results to an extensive list of works evaluated in the same adopted datasets. In all of them, when compared to works that followed an intra-dataset protocol, our methods present competitive numbers. Under a cross-dataset protocol, we achieve state-of-the-art results, outperforming even commercial off-the-shelf solutions from well-known tech companies.     

Optimization-based methodology for training set selection to synthesize composite correlation filters for face recognition

Face recognition has been addressed with pattern recognition techniques such as composite correlation filters. These filters are synthesized from training sets which are representative of facial classes. For this reason, the filter performance depends greatly on the appropriate selection of the training set. This set can be selected either by a filter designer or by a conventional method. This paper presents an optimization-based methodology for the automatic selection of the training set. Given an optimization algorithm, the proposed methodology uses its main mechanics to iteratively examine a given set of available images in order to find the best subset for the training set. To this end, three objective functions are proposed as optimization criteria for training set selection. The proposed methodology was evaluated by undertaking face recognition under variable illumination and facial expressions. Four optimization algorithms and three composite correlation filters were used to test the proposed methodology. The Maximum Average Correlation Height filter designed by Grey Wolf Optimizer obtained the best performance under homogeneous illumination and facial expressions, while the Unconstrained Nonlinear Composite Filter designed by either Grey Wolf Optimizer or (1+1)-Evolution Strategy obtained the best performance under variable illumination. The proposed methodology selects training sets for the synthesis of composite filters with competitive results comparable to the results reported in the face recognition literature.     

Expression transfer for facial sketch animation

This paper presents a hierarchical animation method for transferring facial expressions extracted from a performance video to different facial sketches. Without any expression example obtained from target faces, our approach can transfer expressions by motion retargetting to facial sketches. However, in practical applications, the image noise in each frame will reduce the feature extraction accuracy from source faces. And the shape difference between source and target faces will influence the animation quality for representing expressions. To solve these difficulties, we propose a robust neighbor-expression transfer (NET) model, which aims at modeling the spatial relations among sparse facial features. By learning expression behaviors from neighbor face examples, the NET model can reconstruct facial expressions from noisy signals. Based on the NET model, we present a hierarchical method to animate facial sketches. The motion vectors on the source face are adjusted from coarse to fine on the target face. Accordingly, the animation results are generated to replicate source expressions. Experimental results demonstrate that the proposed method can effectively and robustly transfer expressions by noisy animation signals.     

基于面部检测和深度神经网络的面部情绪自动识别算法

面部情绪识别已成为可见光人脸识别应用的重要部 分,是光学模式识别研究中最重要的领域之一。为了进一步实现可见光条件下面部情绪的自 动识别,本文结合Viola-Jones、自适应直方图均衡(AHE)、离散小波变换(DWT)和深度卷 积神经网络(CNN),提出了一种面部情绪自动识别算法。该算法使用Viola-Jones定位脸 部和五官,使用自适应直方图均衡增强面部图像,使用DWT完成面部特征提取;最后,提取 的特征直接用于深度卷积神经网络训练,以实现面部情绪自动识别。仿真实验分别在CK+数 据库和可见光人脸图像中进行,在CK+数据集上收获了97%的平均准确 率,在可见光人脸图像测试中也获得了95%的平均准确率。实验结果 表明,针对不同的面部五官和情绪,本文算法能够对可见光面部特征进行准确定位,对可见 光图像信息进行均衡处理,对情绪类别进行自动识别,并且能够满足同框下多类面部情绪同 时识别的需求,有着较高的识别率和鲁棒性。     

Bilinear factorisation for facial expression analysis and synthesis

The paper addresses the issue of face representations for facial expression recognition and synthesis. In this context, a global active appearance model is used in conjunction with two bilinear factorisation models to separate expression and identity factors from the global appearance parameters. Although active appearance models and bilinear modelling are not new concepts, the paper's main contribution consists in combining both techniques to improve facial expression recognition and synthesis (control). Indeed, facial expression recognition is performed through linear discriminant analysis of the global appearance parameters extracted by active appearance model search. Results are compared to ones obtained for the same training and test images using classification of the expression factors extracted by bilinear factorisation. This experiment highlights the advantages of bilinear factorisation. Finally, it is proposed to exploit bilinear factorisation to synthesise facial expressions through replacement of the extracted expression factors. This yields very interesting synthesis performances in terms of visual quality of the synthetic faces. Indeed, synthetic open mouth reconstruction, either with or without teeth appearing, is of better quality than with classical linear-regression-based synthesis.     

An efficient approach for facial action unit intensity detection using distance metric learning based on cosine similarity

Emotions of human beings are largely represented by facial expressions. Facial expressions, simple as well as complex, are well decoded by facial action units. Any facial expression can be detected and analyzed if facial action units are decoded well. In the presented work, an attempt has been made to detect facial action unit intensity by mapping the features based on their cosine similarity. Distance metric learning based on cosine similarity maps the data by learning a metric that measures orientation rather than magnitude. The motivation behind using cosine similarity is that change in facial expressions can be better represented by changes in orientation as compared to the magnitude. The features are applied to support vector machine for classification of various intensities of action units. Experimental results on the popularly accepted database such as DISFA database and UNBC McMaster shoulder pain database confirm the efficacy of the proposed approach.     

An adaptive weighted fusion model with two subspaces for facial expression recognition

Automatic facial expression recognition has received considerable attention in the research areas of computer vision and pattern recognition. To achieve satisfactory accuracy, deriving a robust facial expression representation is especially important. In this paper, we present an adaptive weighted fusion model (AWFM), aiming to automatically determine optimal weighted values. The AWFM integrates two subspaces, i.e., unsupervised and supervised subspaces, to represent and classify query samples. The unsupervised subspace is formed by differentiated expression samples generated via an auxiliary neutral training set. The supervised subspace is obtained through the reconstruction of intra-class singular value decomposition based on low-rank decomposition from raw training data. Our experiments using three public facial expression datasets confirm that the proposed model can obtain better performance compared to conventional fusion methods as well as state-of-the-art methods from the literature.     

Frontalization and adaptive exponential ensemble rule for deep-learning-based facial expression recognition system

Automatic facial expression recognition (FER) is an important technique in human–computer interfaces and surveillance systems. It classifies the input facial image into one of the basic expressions (anger, sadness, surprise, happiness, disgust, fear, and neutral). There are two types of FER algorithms: feature-based and convolutional neural network (CNN)-based algorithms. The CNN is a powerful classifier, however, without proper auxiliary techniques, its performance may be limited. In this study, we improve the CNN-based FER system by utilizing face frontalization and the hierarchical architecture. The frontalization algorithm aligns the face by in-plane or out-of-plane, rotation, landmark point matching, and removing background noise. The proposed adaptive exponentially weighted average ensemble rule can determine the optimal weight according to the accuracy of classifiers to improve robustness. Experiments on several popular databases are performed and the results show that the proposed system has a very high accuracy and outperforms state-of-the-art FER systems.     

Affine-structure-based facial image encoding

A real-time algorithm for affine-structure-based video compression for facial images is presented. The face undergoing motion is segmented and triangulated to yield a set of control points. The set of control points generated by triangulation are tracked across a few frames using an intensity-based correlation technique. For accurate motion and structure estimation a Kalman-filter-based algorithm is used to track features on the facial image. The structure information of the control points is transmitted only during the bootstrapping stage. After that only the motion information is transmitted to the decoder. This reduces the number of motion parameters associated with control points in each frame. The local motion of the eyes and lips is captured using local 2-D affine transformations. For real time implementation a quad-tree based search technique is adopted to solve local correlation. Any remaining reconstruction error is accounted for using predictive encoding. Results on real image sequences demonstrate the applicability of the method     

PCA-based dictionary building for accurate facial expression recognition via sparse representation

Sparse representation is a new approach that has received significant attention for image classification and recognition. This paper presents a PCA-based dictionary building for sparse representation and classification of universal facial expressions. In our method, expressive facials images of each subject are subtracted from a neutral facial image of the same subject. Then the PCA is applied to these difference images to model the variations within each class of facial expressions. The learned principal components are used as the atoms of the dictionary. In the classification step, a given test image is sparsely represented as a linear combination of the principal components of six basic facial expressions. Our extensive experiments on several publicly available face datasets (CK+, MMI, and Bosphorus datasets) show that our framework outperforms the recognition rate of the state-of-the-art techniques by about 6%. This approach is promising and can further be applied to visual object recognition.     

Inferring facial expressions from videos: Tool and application

In this paper, we propose a novel approach for facial expression analysis and recognition. The main contributions of the paper are as follows. First, we propose a temporal recognition scheme that classifies a given image in an unseen video into one of the universal facial expression categories using an analysis–synthesis scheme. The proposed approach relies on tracked facial actions provided by a real-time face tracker. Second, we propose an efficient recognition scheme based on the detection of keyframes in videos. Third, we use the proposed method for extending the human–machine interaction functionality of the AIBO robot. More precisely, the robot is displaying an emotional state in response to the user's recognized facial expression. Experiments using unseen videos demonstrated the effectiveness of the developed methods.     

A modular approach to facial feature segmentation on real sequences

In this paper a modular approach of gradual confidence for facial feature extraction over real video frames is presented. The problem is being dealt under general imaging conditions and soft presumptions. The proposed methodology copes with large variations in the appearance of diverse subjects, as well as of the same subject in various instances within real video sequences. Areas of the face that statistically seem to be outstanding form an initial set of regions that are likely to include information about the features of interest. Enhancement of these regions produces closed objects, which reveal—through the use of a fuzzy system—a dominant angle, i.e. the facial rotation angle. The object set is restricted using the dominant angle. An exhaustive search is performed among all candidate objects, matching a pattern that models the relative position of the eyes and the mouth. Labeling of the winner features can be used to evaluate the features extracted and provide feedback in an iterative framework. A subset of the MPEG-4 facial definition or facial animation parameter set can be obtained. This gradual feature revelation is performed under optimization for each step, producing a posteriori knowledge about the face and leading to a step-by-step visualization of the features in search.     

An interactive 3D video system for human facial reconstruction and expression modeling

A 3D facial reconstruction and expression modeling system which creates 3D video sequences of test subjects and facilitates interactive generation of novel facial expressions is described. Dynamic 3D video sequences are generated using computational binocular stereo matching with active illumination and are used for interactive expression modeling. An individual’s 3D video set is annotated with control points associated with face subregions. Dragging a control point updates texture and depth in only the associated subregion so that the user generates new composite expressions unseen in the original source video sequences. Such an interactive manipulation of dynamic 3D face reconstructions requires as little preparation on the test subject as possible. Dense depth data combined with video-based texture results in realistic and convincing facial animations, a feature lacking in conventional marker-based motion capture systems.