Robust face alignment and tracking by combining local search and global fitting

When a face in an image is considerably occluded, existing local search and global fitting methods often cannot find the facial features due to failures in the local facial feature detectors or the fitting limitations of appearance modeling. To solve these problems, we propose a new face alignment method that combines the local search and global fitting methods, where local misalignments in the local search method are restricted by holistic appearance fitting in the global fitting method and the divergent or shrinking alignments in the global fitting method are avoided by the restricting local movements in the local search method. The proposed alignment method consists of two stages: the initialization stage detects the face, estimates the facial pose and obtains the initial facial features by locating a pose-specific mean shape on the detected face; the optimization stage then obtains the facial features by updating the parameter set from the combined Hessian matrix and the combined gradient vector. We also extend the proposed face alignment to face tracking by adding a template image that is warped from the facial features obtained in the previous frame. In the experiments, the proposed method yields more accurate and stable face alignment or tracking under heavy occlusion and pose variation than the existing methods.     

多尺度和纹理特征增强的小尺寸人脸检测

针对现有人脸检测算法难以处理多尺度、多姿态的人脸检测,尤其是面对小尺寸时准确性低的问题,提出了多尺度和纹理特征增强的小尺寸人脸检测算法。该算法的多尺度增强模块能够丰富特征的多尺度信息,提高对多尺度人脸的检测能力;纹理特征增强模块能够通过融合低层的纹理信息提升高层语义的表达,从而加强对小尺寸人脸的检测能力;多阶段加权损失函数平衡网络的输出,充分发挥各个模块的增强作用。实验结果表明,该方法不仅在检测速度上可以达到实时,而且对MALF数据集中高度小于60像素的人脸检测精度可达88.69%;在FDDB数据集上相比目前的BBFCN算法精度提高近四个百分点。     

级联回归的多姿态人脸配准

目的 人脸配准是当前计算机视觉领域的研究热点之一,其目的是准确定位出人脸图像中具有语义特征的面部关键点,这也是人脸识别、人脸美化等众多与人脸有关的视觉任务的重要步骤。最近,基于级联回归的人脸配准算法在配准精度和速度上都达到了最先进的水准。级联回归是一种迭代更新的算法,初始脸形将通过多个线性组合的弱回归器逐渐逼近真实的人脸形状。但目前的算法大多致力于改进学习方法或提取具有几何不变性的特征来提升弱回归器的能力,而忽略了初始脸形的质量,这极大的降低了它们在复杂场景下的配准精度,如夸张的面部表情和极端的头部姿态等。因此,在现有的级联回归框架上,提出自动估计初始形状的多姿态人脸配准算法。方法 本文算法首先在脸部区域提取基于高斯滤波一阶导数的梯度差值特征,并使用随机回归森林预测人脸形状;然后针对不同的形状使用独立的级联回归器。结果 验证初始形状估计算法的有效性,结果显示,本文的初始化算法能给现有的级联回归算法带来精度上的提升,同时结果也更加稳定;本文算法产生的初始形状都与实际脸型较为相近,只需很少的初始形状即可取得较高的精度;在COFW、HELEN和300W人脸数据库上,将本文提出的多姿态级联回归算法和现有配准算法进行对比实验,本文算法的配准误差相较现有算法分别下降了29.2%、13.3%和9.2%,结果表明,本文算法能有效消除不同脸型之间的干扰,在多姿态场景下得到更加精确的配准结果,并能达到实时的检测速度。结论 基于级联回归模型的多姿态人脸配准算法可以取得优于现有算法的结果,在应对复杂的脸形时也更加鲁棒。所提出的初始形状估计算法可以自动产生高质量的初始形状,用于提升现有的级联回归算法。     

基于特征点表情变化的3维人脸识别

目的 为克服表情变化对3维人脸识别的影响,提出一种基于特征点提取局部区域特征的3维人脸识别方法。方法 首先,在深度图上应用2维图像的ASM(active shape model)算法粗略定位出人脸特征点,再根据Shape index特征在人脸点云上精确定位出特征点。其次,提取以鼻中为中心的一系列等测地轮廓线来表征人脸形状;然后,提取具有姿态不变性的Procrustean向量特征(距离和角度)作为识别特征;最后,对各条等测地轮廓线特征的分类结果进行了比较,并对分类结果进行决策级融合。结果 在FRGC V2.0人脸数据库分别进行特征点定位实验和识别实验,平均定位误差小于2.36 mm,Rank-1识别率为98.35%。结论 基于特征点的3维人脸识别方法,通过特征点在人脸近似刚性区域提取特征,有效避免了受表情影响较大的嘴部区域。实验证明该方法具有较高的识别精度,同时对姿态、表情变化具有一定的鲁棒性。     

Robust facial feature tracking under varying face pose and facial expression

This paper presents a hierarchical multi-state pose-dependent approach for facial feature detection and tracking under varying facial expression and face pose. For effective and efficient representation of feature points, a hybrid representation that integrates Gabor wavelets and gray-level profiles is proposed. To model the spatial relations among feature points, a hierarchical statistical face shape model is proposed to characterize both the global shape of human face and the local structural details of each facial component. Furthermore, multi-state local shape models are introduced to deal with shape variations of some facial components under different facial expressions. During detection and tracking, both facial component states and feature point positions, constrained by the hierarchical face shape model, are dynamically estimated using a switching hypothesized measurements (SHM) model. Experimental results demonstrate that the proposed method accurately and robustly tracks facial features in real time under different facial expressions and face poses.     

基于相对索引特征回归的人脸对齐算法

针对传统人脸对齐算法对于较大人脸姿态鲁棒性较差,并且对于人脸检测结果十分敏感的问题,提出了一种基于改进的局部二值特征的人脸对齐算法.不同于传统的形状索引特征,在一种空间依赖假设下的前提下设计相对索引特征.同时,在这种空间依赖假设下使用半全局线性学习替代全局线性学习.通过人脸对齐敏感性分析比较了算法在不同人脸检测器下的鲁棒性,在300-W基准数据集上的实验表明:算法优于传统的级联回归算法.     

Face Alignment by Explicit Shape Regression

We present a very efficient, highly accurate, “Explicit Shape Regression” approach for face alignment. Unlike previous regression-based approaches, we directly learn a vectorial regression function to infer the whole facial shape (a set of facial landmarks) from the image and explicitly minimize the alignment errors over the training data. The inherent shape constraint is naturally encoded into the regressor in a cascaded learning framework and applied from coarse to fine during the test, without using a fixed parametric shape model as in most previous methods. To make the regression more effective and efficient, we design a two-level boosted regression, shape indexed features and a correlation-based feature selection method. This combination enables us to learn accurate models from large training data in a short time (20 min for 2,000 training images), and run regression extremely fast in test (15 ms for a 87 landmarks shape). Experiments on challenging data show that our approach significantly outperforms the state-of-the-art in terms of both accuracy and efficiency.     

基于多尺度核特征卷积神经网络的实时人脸表情识别

针对人脸表情识别的泛化能力不足、稳定性差以及速度慢难以满足实时性要求的问题,提出了一种基于多尺度核特征卷积神经网络的实时人脸表情识别方法。首先,提出改进的MobileNet结合单发多盒检测器（MSSD）轻量化人脸检测网络,并利用核相关滤波（KCF）模型对检测到的人脸坐标信息进行跟踪来提高检测速度和稳定性;然后,使用三种不同尺度卷积核的线性瓶颈层构成三条支路,用通道合并的特征融合方式形成多尺度核卷积单元,利用其多样性特征来提高表情识别的精度;最后,为了提升模型泛化能力和防止过拟合,采用不同的线性变换方式进行数据增强来扩充数据集,并将FER-2013人脸表情数据集上训练得到的模型迁移到小样本CK+数据集上进行再训练。实验结果表明,所提方法在FER-2013数据集上的识别率达到73.0%,较Kaggle表情识别挑战赛冠军提高了1.8%,在CK+数据集上的识别率高达99.5%。对于640×480的视频,人脸检测速度达到每秒158帧,是主流人脸检测网络多任务级联卷积神经网络（MTCNN）的6.3倍,同时人脸检测和表情识别整体速度达到每秒78帧。因此所提方法能够实现快速精确的人脸表情识别。     

Face alignment under occlusion based on local and global feature regression

Shape alignment or estimation under occlusion is one of the most challenging tasks in computer vision field. Most previous works treat occlusion as noises or part models, which usually lead to low accuracy or inefficiencies. This paper proposes an efficient and accurate regression-based algorithm for face alignment. In this framework, local and global regressions are iteratively used to train a series of random forests in a cascaded manner. In training and testing process, each step consists of two layers. In the first layer, a set of highly discriminative local features are extracted from local regions according to locality principle. The regression forests are trained for each facial landmark independently using those local features. Then the leaf node of the regression tree is encoded by histogram statistic method and the final shape is estimated by a linear regression matrix. In the second layer, our proposed global features are generated. Then we use those features to train a random fern to keep the global shape constraints. Experiments show that our method has a high speed, but same or slightly lower accuracy than state of the art methods under occlusion condition. In order to gain a higher accuracy we use multi-random shape for initialization, which may slightly reduce the calculation efficiency as a trade-off.

     

基于条件对抗自动编码器的跨年龄人脸合成

跨年龄人脸合成是指通过已知特定年龄的人脸图像合成其他年龄段的人脸图像,在动漫娱乐、公共安全、刑事侦查等领域有广泛的应用。针对跨年龄人脸合成图像容易产生器官变形扭曲、人脸局部特征保持效果不佳等问题,提出一种基于条件对抗自动编码器的合成方法。通过在解码器结构中引入通道关注和空间关注模块,分别从通道域和空间域提取重要信息,使模型在训练过程中忽略背景等无关信息,聚焦人脸图像变化的区域,有效解决合成图像器官扭曲变形等问题。此外,设计一种多尺度特征损失网络,从多个尺度更深层次地约束人脸图像的局部结构特征,从而保持人脸合成过程中局部特征结构的稳定性。在UTKFace跨年龄人脸数据集上的实验结果表明,与CAAE方法相比,该方法有效避免了人脸器官变形扭曲问题,能够更好地保持人脸局部结构特征,具有较佳的人脸合成效果和细节保持能力。     

两层级联卷积神经网络的人脸检测

目的 传统人脸检测方法因人脸多姿态变化和人脸面部特征不完整等问题,导致检测效果不佳。为解决上述问题,提出一种两层级联卷积神经网络（TC_CNN）人脸检测方法。方法 首先,构建两层卷积神经网络模型,利用前端卷积神经网络模型对人脸图像进行特征粗略提取,再利用最大值池化方法对粗提取得到的人脸特征进行降维操作,输出多个疑似人脸窗口;其次,将前端粗提取得到的人脸窗口作为后端卷积神经网络模型的输入进行特征精细提取,并通过池化操作得到新的特征图;最后,通过全连接层判别输出最佳检测窗口,完成人脸检测全过程。结果 实验选取FDDB人脸检测数据集中包含人脸多姿态变化以及人脸面部特征信息不完整等情况的图像进行测试,TC_CNN方法人脸检测率达到96.39%,误检率低至3.78%,相比当前流行方法在保证算法效率的同时检测率均有提高。结论 两层级联卷积神经网络人脸检测方法能够在人脸多姿态变化和面部特征信息不完整等情况下实现精准检测,保证较高的检测率,有效降低误检率,方法具有较好的鲁棒性和泛化能力。     

A robust real-time facial alignment system with facial landmarks detection and rectification for multimedia applications

Face detection often plays the first step in various visual applications. Large variants of facial deformations due to head movements and facial expression make it difficult to identify appropriate face region. In this paper, a robust real-time face alignment system, including facial landmarks detection and face rectification, is proposed. A facial landmarks detection model based on regression tree is utilized in the proposed system. In face rectification framework, 2-D geometrical analysis based on pitch, yaw and roll movements is designed to solve the misalignment problem in face detection. The experiments on the two datasets verify the performance significantly improved by the proposed method in the facial recognition task and outperform than those obtained by other alignment methods. Furthermore, the proposed method can achieve robust recognition results even if the amount of training images is not large.

     

Eye detection in a facial image under pose variation based on multi-scale iris shape feature

The accurate location of eyes in a facial image is important to many human facial recognition-related applications, and has attracted considerable research interest in computer vision. However, most prevalent methods are based on the frontal pose of the face, where applying them to non-frontal poses can yield erroneous results.In this paper, we propose an eye detection method that can locate the eyes in facial images captured at various head poses. Our proposed method consists of two stages: eye candidate detection and eye candidate verification. In eye candidate detection, eye candidates are obtained by using multi-scale iris shape features and integral image. The size of the iris in face images varies as the head pose changes, and the proposed multi-scale iris shape feature method can detect the eyes in such cases. Since it utilizes the integral image, its computational cost is relatively low. The extracted eye candidates are then verified in the eye candidate verification stage using a support vector machine (SVM) based on the feature-level fusion of a histogram of oriented gradients (HOG) and cell mean intensity features.We tested the performance of the proposed method using the Chinese Academy of Sciences' Pose, Expression, Accessories, and Lighting (CAS-PEAL) database and the Pointing'04 database. The results confirmed the superiority of our method over the conventional Haar-like detector and two hybrid eye detectors under relatively extreme head pose variations.     

Non-manual grammatical marker recognition based on multi-scale,spatio-temporal analysis of head pose and facial expressions

Changes in eyebrow configuration, in conjunction with other facial expressions and head gestures, are used to signal essential grammatical information in signed languages. This paper proposes an automatic recognition system for non-manual grammatical markers in American Sign Language (ASL) based on a multi-scale, spatio-temporal analysis of head pose and facial expressions. The analysis takes account of gestural components of these markers, such as raised or lowered eyebrows and different types of periodic head movements. To advance the state of the art in non-manual grammatical marker recognition, we propose a novel multi-scale learning approach that exploits spatio-temporally low-level and high-level facial features. Low-level features are based on information about facial geometry and appearance, as well as head pose, and are obtained through accurate 3D deformable model-based face tracking. High-level features are based on the identification of gestural events, of varying duration, that constitute the components of linguistic non-manual markers. Specifically, we recognize events such as raised and lowered eyebrows, head nods, and head shakes. We also partition these events into temporal phases. We separate the anticipatory transitional movement (the onset) from the linguistically significant portion of the event, and we further separate the core of the event from the transitional movement that occurs as the articulators return to the neutral position towards the end of the event (the offset). This partitioning is essential for the temporally accurate localization of the grammatical markers, which could not be achieved at this level of precision with previous computer vision methods. In addition, we analyze and use the motion patterns of these non-manual events. Those patterns, together with the information about the type of event and its temporal phases, are defined as the high-level features. Using this multi-scale, spatio-temporal combination of low- and high-level features, we employ learning methods for accurate recognition of non-manual grammatical markers in ASL sentences.     

基于层进模型的快速人脸配准方法及其在智能移动设备上的应用

主要研究了移动智能手机上人脸关键点的快速定位问题。在活动形状模型的基础上,提出了一种基于层进模型的快速人脸配准方法:首先,在人脸检测的结果上,采用二值特征快速定位眼角、嘴角等关键点,并对其进行校验修正;然后,通过眼角和嘴角的关键点,并结合边缘约束,对眼睛、嘴巴和人脸外轮廓进行局部配准;最后,对整个人脸形状进行基于加权投影的形状配准。实验结果表明,提出的方法在8~10次迭代后即可收敛,在三星I9300智能手机上,每幅人脸图像的配准时间在40ms以下,满足实时性要求。     

结合形变模型与图像修复的人脸姿态矫正

目的 人脸姿态偏转是影响人脸识别准确率的一个重要因素,本文利用3维人脸重建中常用的3维形变模型以及深度卷积神经网络,提出一种用于多姿态人脸识别的人脸姿态矫正算法,在一定程度上提高了大姿态下人脸识别的准确率。方法 对传统的3维形变模型拟合方法进行改进,利用人脸形状参数和表情参数对3维形变模型进行建模,针对面部不同区域的关键点赋予不同的权值,加权拟合3维形变模型,使得具有不同姿态和面部表情的人脸图像拟合效果更好。然后,对3维人脸模型进行姿态矫正并利用深度学习对人脸图像进行修复,修复不规则的人脸空洞区域,并使用最新的局部卷积技术同时在新的数据集上重新训练卷积神经网络,使得网络参数达到最优。结果 在LFW（labeled faces in the wild）人脸数据库和StirlingESRC（Economic Social Research Council）3维人脸数据库上,将本文算法与其他方法进行比较,实验结果表明,本文算法的人脸识别精度有一定程度的提高。在LFW数据库上,通过对具有任意姿态的人脸图像进行姿态矫正和修复后,本文方法达到了96.57%的人脸识别精确度。在StirlingESRC数据库上,本文方法在人脸姿态为±22°的情况下,人脸识别准确率分别提高5.195%和2.265%;在人脸姿态为±45°情况下,人脸识别准确率分别提高5.875%和11.095%;平均人脸识别率分别提高5.53%和7.13%。对比实验结果表明,本文提出的人脸姿态矫正算法有效提高了人脸识别的准确率。结论 本文提出的人脸姿态矫正算法,综合了3维形变模型和深度学习模型的优点,在各个人脸姿态角度下,均能使人脸识别准确率在一定程度上有所提高。     

基于样例学习的面部特征自动标定算法

面部特征标定是人脸识别中的一个关键问题.提出了一种基于样例学习的面部特征自动标定(人脸形状自动提取)方法.该方法是基于下面假设提出来的:人脸图像差和形状差之间存在一种近似的线性关系--相似的人脸图像在较大程度上蕴涵着相似的形状.因此,给定标注了特征点的人脸图像学习集,则任意新的输入人脸图像的面部形状可以采用如下方法估计:测量该人脸图像和训练集中图像的相似度,并将同样的相似度用于该人脸图像形状的重建.即:如果输入人脸图像可以表示为训练图像的优化的线性组合,那么同样的线性组合系数就可以直接用于训练集对应形状的线性组合从而得到输入人脸图像的形状.实验表明,该算法相对于其他传统的特征标定算法具有可比的精度和较快的速度.并且,还将此算法扩展到了多姿态情况下,实现了多姿态人脸图像形状的自动提取.     

对表情鲁棒的面部轮廓线3维人脸识别

目的表情变化是3维人脸识别面临的主要问题。为克服表情影响,提出了一种基于面部轮廓线对表情鲁棒的3维人脸识别方法。方法首先,对人脸进行预处理,包括人脸区域切割、平滑处理和姿态归一化,将所有的人脸置于姿态坐标系下;然后,从3维人脸模型的半刚性区域提取人脸多条垂直方向的轮廓线来表征人脸面部曲面;最后,利用弹性曲线匹配算法计算不同3维人脸模型间对应的轮廓线在预形状空间(preshape space)中的测地距离,将其作为相似性度量,并且对所有轮廓线的相似度向量加权融合,得到总相似度用于分类。结果在FRGC v2.0数据库上进行识别实验,获得97.1%的Rank-1识别率。结论基于面部轮廓线的3维人脸识别方法,通过从人脸的半刚性区域提取多条面部轮廓线来表征人脸,在一定程度上削弱了表情的影响,同时还提高了人脸匹配速度。实验结果表明,该方法具有较强的识别性能,并且对表情变化具有较好的鲁棒性。