Recognition of facial expressions based on salient geometric features and support vector machines

Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.     

Facial expressions in American sign language: Tracking and recognition

This paper presents work towards recognizing facial expressions that are used in sign language communication. Facial features are tracked to effectively capture temporal visual cues on the signers' face during signing. Face shape constraints are used for robust tracking within a Bayesian framework. The constraints are specified through a set of face shape subspaces learned by Probabilistic Principal Component Analysis (PPCA). An update scheme is also used to adapt to persons with different face shapes. Two tracking algorithms are presented, which differ in the way the face shape constraints are enforced. The results show that the proposed trackers can track facial features with large head motions, substantial facial deformations, and temporary facial occlusions by hand. The tracked results are input to a recognition system comprising Hidden Markov Models (HMM) and a support vector machine (SVM) to recognize six isolated facial expressions representing grammatical markers in American sign language (ASL). Tracking error of less than four pixels (on 640×480 videos) was obtained with probability greater than 90%; in comparison the KLT tracker yielded this accuracy with 76% probability. Recognition accuracy obtained for ASL facial expressions was 91.76% in person dependent tests and 87.71% in person independent tests.     

基于FPGA的KLT特征点多层次归并排序

KLT算法已在多个领域得到成功的应用,其中特征点的排序是用来选择好的特征点跟踪的关键。针对传统排序算法计算耗时、实时性差的缺点,提出一种可并行的多层次归并排序算法并在FPGA中实现了其并行计算,同时分析了其周期精确的计算时间。结果表明该归并排序算法可以[O(N)]的时间复杂度完成特征点的排序,能够满足高清分辨率的图像/视频数据中KLT特征点排序的实时性要求。     

基于人脸识别与光流追踪的移动机器人视觉导航方法

针对移动机器人视觉导航中跟踪目标丢失的问题,提出了基于人脸识别与稀疏光流算法(KLT)结合的移动机器人视觉导航方法(FR-KLT视觉导航方法)。采用OpenCV库中的Haar特征提取人脸识别算法实时检测识别目标人脸,通过Harris角点检测获取目标人体特征点,对目标人体进行精准定位;KLT光流追踪法测算目标移动趋势,并预测目标下一刻大致位置。目标人体位置变动时移动机器人对目标进行实时追踪导航。通过Pioneer-LX机器人在真实环境下试验,验证了该方法准确识别并跟踪目标的实时性和有效性。     

Model-free non-rigid head pose tracking by joint shape and pose estimation

Head pose estimation under non-rigid face movement is particularly useful in applications relating to eye-gaze tracking in less constrained scenarios, where the user is allowed to move naturally during tracking. Existing vision-based head pose estimation methods often require accurate initialisation and tracking of specific facial landmarks, while methods that handle non-rigid face deformations typically necessitate a preliminary training phase prior to head pose estimation. In this paper, we propose a method to estimate the head pose in real-time from the trajectories of a set of feature points spread randomly over the face region, without requiring a training phase or model-fitting of specific facial features. Conversely, our method exploits the 3-dimensional shape of the surface of interest, recovered via shape and motion factorisation, in combination with Kalman and particle filtering to determine the contribution of each feature point to the estimation of head pose based on a variance measure. Quantitative and qualitative results reveal the capability of our method in handling non-rigid face movement without deterioration of the head pose estimation accuracy.     

Efficient covering of target areas using a location prediction-based algorithm

Due to the rapid development of the high-speed wired and wireless Internet, image contents including objects with exposed personal information are being distributed freely, which is a social problem. In this paper, we introduce a method of robustly detecting a target object with facial region exposed from an image that is quickly entered using skin color and a deep learning algorithm and effectively covering the detected target object through prediction. The proposed method in this paper accurately detects the target object containing facial region exposed from the image entered by applying an image adaptive skin color model and a CNN-based deep learning algorithm. Subsequently, the location prediction algorithm is used to quickly track the detected object. A mosaic is overlapped over the target object area to effectively protect the object area where the facial region is exposed. The experimental results show that the proposed approach accurately detects the target object including the facial region exposed from the continuously entered video, and efficiently covers the detected object through mosaic processing while quickly tracking it using a prediction-based tracking algorithm. The tracking-based target covering method proposed in this study is expected to be useful in various practical applications related to pattern recognition and image security, such as content-based image retrieval, real-time surveillance, human–computer interaction, and face detection.

     

Robust face tracking by integration of two separate trackers: Skin color and facial shape

This paper proposes a robust face tracking method based on the condensation algorithm that uses skin color and facial shape as observation measures. Two trackers are used for robust tracking: one tracks the skin color regions and the other tracks the facial shape regions. The two trackers are coupled using an importance sampling technique, where the skin color density obtained from the skin color tracker is used as the importance function to generate samples for the shape tracker. The samples of the skin color tracker within the chosen shape region are updated with higher weights. Also, an adaptive color model is used to avoid the effect of illumination change in the skin color tracker. The proposed face tracker performs more robustly than either the skin-color-based tracker or the facial shape-based tracker, given the presence of background clutter and/or illumination changes.     

Real-time face tracking and recognition by sparse eigentracker with associative mapping to 3D shape

This paper proposes a novel framework of real-time face tracking and recognition by combining two eigen-based methods. The first method is a novel extension of eigenface called augmented eigenface and the second method is a sparse 3D eigentemplate tracker controlled by a particle filter. The augmented eigenface is an eigenface augmented by an associative mapping to 3D shape that is specified by a set of volumetric face models. This paper discusses how to make up the augmented eigenface and how it can be used for inference of 3D shape from partial images. The associative mapping is also generalized to subspace-to-one mappings to cover photometric image changes for a fixed shape. A novel technique, called photometric adjustment, is introduced for simple implementation of associative mapping when an image subspace should be combined to a shape. The sparse 3D eigentemplate tracker is an extension of the 3D template tracker proposed by Oka et al. In combination with the augmented eigenface, the sparse 3D eigentemplate tracker facilitates real-time 3D tracking and recognition when a monocular image sequence is provided. In the tracking, sparse 3D eigentemplate is updated by the augmented eigenface while face pose is estimated by the sparse eigentracker. Since the augmented eigenface is constructed on the conventional eigenfaces, face identification and expression recognition are also accomplished efficiently during the tracking. In the experiment, an augmented eigenface was constructed from 25 faces where 24 images were taken in different lighting conditions for each face. Experimental results show that the augmented eigenface works with the 3D eigentemplate tracker for real-time tracking and recognition.     

多视角约束级联回归的视频人脸特征点跟踪

近年来,静态图像中人脸特征点检测算法得到了极大的改进,然而,由于真实视频中头部姿态、遮挡和光照等因素的变化,人脸特征点检测和跟踪仍然具有挑战性。为了解决这一问题,提出一种多视角约束级联回归的视频人脸特征点跟踪算法。首先,利用三维和二维稀疏点集建立变换关系,并估计初始形状;其次,由于人脸图像存在较大的姿态差异,使用仿射变换对人脸图像进行姿态矫正;在构造形状回归模型时,采用多视角约束级联回归模型减小形状方差,从而使学习到的回归模型对形状方差具有更强的鲁棒性;最后,采用重新初始化机制,并在特征点正确定位时使用归一化互相关（NCC）模板匹配跟踪算法建立连续帧之间的形状关系。在公共数据集上的实验结果表明：该算法的平均误差小于眼间距离的10%。     

基于混合粒子滤波的目标跟踪

为了克服多目标跟踪中估计效果对初始样本选择的强依赖性,首先通过Harris角点检测和KLT算法实现对图像序列中的特征提取和匹配,然后利用Mean-shift算法对匹配的特征点进行聚类和定位,将Mean-shift算法与粒子滤波器相结合,提出了基于Mean-shift算法的混合粒子滤波器,给出了具体算法流程,并就实际图像序列的动态多目标跟踪进行了实验,实验结果证明了该方法的有效性。     

深度学习人脸特征点自动定位综述

人脸特征点定位是根据输入的人脸数据自动定位出预先按人脸生理特征定义的眼角、鼻尖、嘴角和脸部轮廓等面部关键特征点,在人脸识别和分析等系统中起着至关重要的作用。本文对基于深度学习的人脸特征点自动定位进行综述,阐释了人脸特征点自动定位的含义,归纳了目前常用的人脸公开数据集,系统阐述了针对2维和3维数据特征点的自动定位方法,总结了各方法的研究现状及其应用,分析了当前人脸特征点自动定位技术在深度学习应用中的现状、存在问题及发展趋势。在公开的2维和3维人脸数据集上对不同方法进行了比较。通过研究可以看出,基于深度学习的2维人脸特征点的自动定位方法研究相对比较深入,而3维人脸特征点定位方法的研究在模型表示、处理方法和样本数量上都存在挑战。未来基于深度学习的3维人脸特征点定位方法将成为研究趋势。     

以颜色和形状直方图为线索的粒子滤波人脸跟踪

跟踪器的设计和跟踪线索的选择与表达是人脸跟踪中的两大关键因素,针对一般人脸跟踪算法中常用简单椭圆来描述人脸形状线索时易受背景干扰的缺点,以及视频目标跟踪中动态模型和观测模型的非线性非高斯特点,提出了一种以颜色和形状直方图为线索的粒子滤波人脸跟踪算法,该算法在粒子滤波基本框架之下,引入了一种新的用直方图来描述人脸形状的方法,并对其进行了改进,用来作为人脸跟踪的形状线索。同时,为了减轻背景干扰,提出了一种经验有效边缘的检测方法。实验表明,该跟踪方法不仅能有效地处理人脸旋转、背景中的肤色干扰和部分遮掩问题,并且能够在由于大面积遮掩等原因而丢失目标的情况下,及时有效地重新捕获已丢失的目标。     

一种嵌入式环境下的实时人脸跟踪方法

针对复杂背景和光照变化条件下的人脸跟踪问题,提出了一种适于嵌入式环境的实时人脸跟踪方法。该方法使用帧差法检测出运动区域,在运动区域内进行AdaBoost人脸检测并进而提取人脸特征点,运用距离约束的金字塔形Lucas-Kanade方法跟踪人脸特征点。当特征点丢失较严重时,重新选取特征点进行跟踪。实验结果表明,该方法有较鲁棒的跟踪性能,在嵌入式环境下能够实时跟踪人脸。     

基于Mean Shift算法和粒子滤波器的人眼跟踪

基于视觉的驾驶疲劳检测是人脸表情识别技术最有商业前途的应用之一,实时人眼跟踪是其中的关键部分。为了解决跟踪方法对眼睛的部分遮挡、人脸尺度变化等过于敏感的问题,提出了一种综合MeanShift算法和粒子滤波器的跟踪算法。利用粒子滤波器得到样本的观测值后,将MeanShift分析用于每一个粒子,使得粒子集中在测量模型的局部区域内,很好地克服了粒子滤波器的退化现象并有效缩短了计算时间。实验结果表明该算法实时性强,且具有良好的鲁棒性。     

Robust facial feature tracking under varying face pose and facial expression

This paper presents a hierarchical multi-state pose-dependent approach for facial feature detection and tracking under varying facial expression and face pose. For effective and efficient representation of feature points, a hybrid representation that integrates Gabor wavelets and gray-level profiles is proposed. To model the spatial relations among feature points, a hierarchical statistical face shape model is proposed to characterize both the global shape of human face and the local structural details of each facial component. Furthermore, multi-state local shape models are introduced to deal with shape variations of some facial components under different facial expressions. During detection and tracking, both facial component states and feature point positions, constrained by the hierarchical face shape model, are dynamically estimated using a switching hypothesized measurements (SHM) model. Experimental results demonstrate that the proposed method accurately and robustly tracks facial features in real time under different facial expressions and face poses.     

基于Kanade-Lucas-Tomasi 算法的人脸特征点跟踪方法

与传统的在人面部画上标识点的特征点跟踪方法不同，KLT(Kanade-Lucas-Tclmasi)算法可以从未加标识点的正面人像视频系列中通过特征纹理信息直接获取面部某些特征点的位移，在KLT算法中加入了基于人脸统计信息的经验约束，使KLT算法更加合理有效。     

Faster CNN-based vehicle detection and counting strategy for fixed camera scenes

Automatic detection and counting of vehicles in a video is a challenging task and has become a key application area of traffic monitoring and management. In this paper, an efficient real-time approach for the detection and counting of moving vehicles is presented based on YOLOv2 and features point motion analysis. The work is based on synchronous vehicle features detection and tracking to achieve accurate counting results. The proposed strategy works in two phases; the first one is vehicle detection and the second is the counting of moving vehicles. Different convolutional neural networks including pixel by pixel classification networks and regression networks are investigated to improve the detection and counting decisions. For initial object detection, we have utilized state-of-the-art faster deep learning object detection algorithm YOLOv2 before refining them using K-means clustering and KLT tracker. Then an efficient approach is introduced using temporal information of the detection and tracking feature points between the framesets to assign each vehicle label with their corresponding trajectories and truly counted it. Experimental results on twelve challenging videos have shown that the proposed scheme generally outperforms state-of-the-art strategies. Moreover, the proposed approach using YOLOv2 increases the average time performance for the twelve tested sequences by 93.4% and 98.9% from 1.24 frames per second achieved using Faster Region-based Convolutional Neural Network (F R-CNN ) and 0.19 frames per second achieved using the background subtraction based CNN approach (BS-CNN ), respectively to 18.7 frames per second.

     

An efficient face detection based on color-filtering and its application to smart devices

This paper proposes a means of using facial color to enhance conventional face detectors. To detect face rapidly, the proposed approach adopts a color filtering based efficient region scanning method. The scanning method skips over regions that do not contain candidate faces, based on a facial color membership function. Then it adopts a face/non-face classifier using facial color at the preprocessor of the face detector. This classifier has low computational cost and can reject non-face regions at an early stage of face detection. By integrating the proposed face detector with a kernel based object tracker, a real-time face detection and tracking application is implemented for smart devices. The proposed method considerably reduces the overall computation time and reduces the number of false alarms.     

A robust real-time facial alignment system with facial landmarks detection and rectification for multimedia applications

Face detection often plays the first step in various visual applications. Large variants of facial deformations due to head movements and facial expression make it difficult to identify appropriate face region. In this paper, a robust real-time face alignment system, including facial landmarks detection and face rectification, is proposed. A facial landmarks detection model based on regression tree is utilized in the proposed system. In face rectification framework, 2-D geometrical analysis based on pitch, yaw and roll movements is designed to solve the misalignment problem in face detection. The experiments on the two datasets verify the performance significantly improved by the proposed method in the facial recognition task and outperform than those obtained by other alignment methods. Furthermore, the proposed method can achieve robust recognition results even if the amount of training images is not large.

     

Differential optical flow applied to automatic facial expression recognition

This work compares systematically two optical flow-based facial expression recognition methods. The first one is featural and selects a reduced set of highly discriminant facial points while the second one is holistic and uses much more points that are uniformly distributed on the central face region. Both approaches are referred as feature point tracking and holistic face dense flow tracking, respectively. They compute the displacements of different sets of points along the sequence of frames describing each facial expression (i.e. from neutral to apex). First, we evaluate our algorithms on the Cohn-Kanade database for the six prototypic expressions under two different spatial frame resolutions (original and 40%-reduced). Later, our methods were also tested on the MMI database which presents higher variabilities than the Cohn-Kanade one. The results on the first database show that dense flow tracking method at original resolution slightly outperformed, in average, the recognition rates of feature point tracking method (95.45% against 92.42%) but it requires 68.24% more time to track the points. For the patterns of MMI database, using dense flow tracking at the original resolution, we achieved very similar average success rates.