Feature-based detection of facial landmarks from neutral and expressive facial images

Feature-based method for detecting landmarks from facial images was designed. The method was based on extracting oriented edges and constructing edge maps at two resolution levels. Edge regions with characteristic edge pattern formed landmark candidates. The method ensured invariance to expressions while detecting eyes. Nose and mouth detection was deteriorated by happiness and disgust.     

Human shape correspondence with automatically predicted landmarks

We consider the problem of computing accurate point-to-point correspondences among a set of human bodies in similar posture using a landmark-free approach. The approach learns the locations of the anthropometric landmarks present in a database of human models in similar postures and uses this knowledge to automatically predict the locations of these anthropometric landmarks on a newly available scan. The predicted landmarks are then used to compute point-to-point correspondences between a template model and the newly available scan. This study conducts a large-scale evaluation to examine the accuracy of the computed correspondences. Furthermore, we show that the correspondences are accurate enough for the application of motion transfer.     

Robust face tracking by integration of two separate trackers: Skin color and facial shape

This paper proposes a robust face tracking method based on the condensation algorithm that uses skin color and facial shape as observation measures. Two trackers are used for robust tracking: one tracks the skin color regions and the other tracks the facial shape regions. The two trackers are coupled using an importance sampling technique, where the skin color density obtained from the skin color tracker is used as the importance function to generate samples for the shape tracker. The samples of the skin color tracker within the chosen shape region are updated with higher weights. Also, an adaptive color model is used to avoid the effect of illumination change in the skin color tracker. The proposed face tracker performs more robustly than either the skin-color-based tracker or the facial shape-based tracker, given the presence of background clutter and/or illumination changes.     

Real-time face tracking and recognition by sparse eigentracker with associative mapping to 3D shape

This paper proposes a novel framework of real-time face tracking and recognition by combining two eigen-based methods. The first method is a novel extension of eigenface called augmented eigenface and the second method is a sparse 3D eigentemplate tracker controlled by a particle filter. The augmented eigenface is an eigenface augmented by an associative mapping to 3D shape that is specified by a set of volumetric face models. This paper discusses how to make up the augmented eigenface and how it can be used for inference of 3D shape from partial images. The associative mapping is also generalized to subspace-to-one mappings to cover photometric image changes for a fixed shape. A novel technique, called photometric adjustment, is introduced for simple implementation of associative mapping when an image subspace should be combined to a shape. The sparse 3D eigentemplate tracker is an extension of the 3D template tracker proposed by Oka et al. In combination with the augmented eigenface, the sparse 3D eigentemplate tracker facilitates real-time 3D tracking and recognition when a monocular image sequence is provided. In the tracking, sparse 3D eigentemplate is updated by the augmented eigenface while face pose is estimated by the sparse eigentracker. Since the augmented eigenface is constructed on the conventional eigenfaces, face identification and expression recognition are also accomplished efficiently during the tracking. In the experiment, an augmented eigenface was constructed from 25 faces where 24 images were taken in different lighting conditions for each face. Experimental results show that the augmented eigenface works with the 3D eigentemplate tracker for real-time tracking and recognition.     

Real-time tracking using trust-region methods

Optimization methods based on iterative schemes can be divided into two classes: line-search methods and trust-region methods. While line-search techniques are commonly found in various vision applications, not much attention is paid to trust-region ones. Motivated by the fact that line-search methods can be considered as special cases of trust-region methods, we propose to establish a trust-region framework for real-time tracking. Our approach is characterized by three key contributions. First, since a trust-region tracking system is more effective, it often yields better performances than the outcomes of other trackers that rely on iterative optimization to perform tracking, e.g., a line-search-based mean-shift tracker. Second, we have formulated a representation model that uses two coupled weighting schemes derived from the covariance ellipse to integrate an object's color probability distribution and edge density information. As a result, the system can address rotation and nonuniform scaling in a continuous space, rather than working on some presumably possible discrete values of rotation angle and scale. Third, the framework is very flexible in that a variety of distance functions can be adapted easily. Experimental results and comparative studies are provided to demonstrate the efficiency of the proposed method.     

Real-time vision-based tracking and reconstruction

Many of the recent real-time markerless camera tracking systems assume the existence of a complete 3D model of the target scene. Also the system developed in the MATRIS project assumes that a scene model is available. This can be a freeform surface model generated automatically from an image sequence using structure from motion techniques or a textured CAD model built manually using a commercial software. The offline model provides 3D anchors to the tracking. These are stable natural landmarks, which are not updated and thus prevent an accumulating error (drift) in the camera registration by giving an absolute reference. However, sometimes it is not feasible to model the entire target scene in advance, e.g. parts, which are not static, or one would like to employ existing CAD models, which are not complete. In order to allow camera movements beyond the parts of the environment modelled in advance it is desired to derive additional 3D information online. Therefore, a markerless camera tracking system for calibrated perspective cameras has been developed, which employs 3D information about the target scene and complements this knowledge online by reconstruction of 3D points. The proposed algorithm is robust and reduces drift, the most dominant problem of simultaneous localisation and mapping (SLAM), in real-time by a combination of the following crucial points: (1) stable tracking of longterm features on the 2D level; (2) use of robust methods like the well-known Random Sampling Consensus (RANSAC) for all 3D estimation processes; (3) consequent propagation of errors and uncertainties; (4) careful feature selection and map management; (5) incorporation of epipolar constraints into the pose estimation. Validation results on the operation of the system on synthetic and real data are presented.     

Real-time visual tracking of complex structures

Presents a framework for three-dimensional model-based tracking. Graphical rendering technology is combined with constrained active contour tracking to create a robust wire-frame tracking system. It operates in real time at video frame rate (25 Hz) on standard hardware. It is based on an internal CAD model of the object to be tracked which is rendered using a binary space partition tree to perform hidden line removal. A Lie group formalism is used to cast the motion computation problem into simple geometric terms so that tracking becomes a simple optimization problem solved by means of iterative reweighted least squares. A visual servoing system constructed using this framework is presented together with results showing the accuracy of the tracker. The paper then describes how this tracking system has been extended to provide a general framework for tracking in complex configurations. The adjoint representation of the group is used to transform measurements into common coordinate frames. The constraints are then imposed by means of Lagrange multipliers. Results from a number of experiments performed using this framework are presented and discussed     

Deep learning face representation by fixed erasing in facial landmarks

Multimedia Tools and Applications - Face verification (FV) is a challenging problem, because occlusion, posture, illumination, aging will affect the accuracy of FV. Deep convolutional neural...     

Image set-based face recognition using pose estimation with facial landmarks

Multimedia Tools and Applications - Face recognition (FR) based on image set is an important topic in computer vision. There are numerous approaches that apply pose estimation method for single...     

A robust real-time facial alignment system with facial landmarks detection and rectification for multimedia applications

Face detection often plays the first step in various visual applications. Large variants of facial deformations due to head movements and facial expression make it difficult to identify appropriate face region. In this paper, a robust real-time face alignment system, including facial landmarks detection and face rectification, is proposed. A facial landmarks detection model based on regression tree is utilized in the proposed system. In face rectification framework, 2-D geometrical analysis based on pitch, yaw and roll movements is designed to solve the misalignment problem in face detection. The experiments on the two datasets verify the performance significantly improved by the proposed method in the facial recognition task and outperform than those obtained by other alignment methods. Furthermore, the proposed method can achieve robust recognition results even if the amount of training images is not large.

     

Real-time fingertip tracking and gesture recognition

Augmented desk interfaces and other virtual reality systems depend on accurate, real-time hand and fingertip tracking for seamless integration between real objects and associated digital information. We introduce a method for discerning fingertip locations in image frames and measuring fingertip trajectories across image frames. We also propose a mechanism for combining direct manipulation and symbolic gestures based on multiple fingertip motions. Our method uses a filtering technique, in addition to detecting fingertips in each image frame, to predict fingertip locations in successive image frames and to examine the correspondences between the predicted locations and detected fingertips. This lets us obtain multiple complex fingertip trajectories in real time and improves fingertip tracking. This method can track multiple fingertips reliably even on a complex background under changing lighting conditions without invasive devices or color markers.     

基于卷积神经网络的实时跟踪算法

利用卷积神经网络(CNN)强大的特征学习能力,提出了一种基于卷积神经网络的实时跟踪算法.通过对双通道卷积神经网络进行离线训练,学习相邻两帧之间的差异,得到跟踪目标的表观特征与运动之间的普遍规律.在不需要对网络模型在线更新的情况下,直接通过网络回归得到对目标的位置和对应置信度的预测.在VOT2014数据集中进行实验,结果表明:提出的跟踪算法的性能达到了当前领先水平.同时,跟踪算法的运行速度可以达到90帧/s,表现出非常不错的实时性.     

Real-time vehicle tracking for driving assistance

Detecting car taillights at night is a task which can nowadays be accomplished very fast on cheap hardware. We rely on such detections to build a vision-based system that, coupling them in a rule-based fashion, is able to detect and track vehicles. This allows the generation of an interface that informs a driver of the relative distance and velocity of other vehicles in real time and triggers a warning when a potentially dangerous situation arises. We demonstrate the system using sequences shot using a camera mounted behind a car’s windshield.     

Real-time facial animation on mobile devices

We present a performance-based facial animation system capable of running on mobile devices at real-time frame rates. A key component of our system is a novel regression algorithm that accurately infers the facial motion parameters from 2D video frames of an ordinary web camera. Compared with the state-of-the-art facial shape regression algorithm [1], which takes a two-step procedure to track facial animations (i.e., first regressing the 3D positions of facial landmarks, and then computing the head poses and expression coefficients), we directly regress the head poses and expression coefficients. This one-step approach greatly reduces the dimension of the regression target and significantly improves the tracking performance while preserving the tracking accuracy. We further propose to collect the training images of the user under different lighting environments, and make use of the data to learn a user-specific regressor, which can robustly handle lighting changes that frequently occur when using mobile devices.     

Extraction of 3D freeform surfaces as visual landmarks for real-time tracking

This work presents a system for the generation of a free-form surface model from video sequences. Although any single centered camera can be applied in the proposed system the approach is demonstrated using fish-eye lenses because of their good properties for tracking. The system is designed to function automatically and to be flexible with respect to size and shape of the reconstructed scene. To minimize geometric assumptions a statistic fusion of dense depth maps is utilized. Special attention is payed to the necessary rectification of the spherical images and the resulting iso-disparity surfaces, which can be exploited in the fusion approach. Before dense depth estimation can be performed the cameras’ pose parameters are extracted by means of a structure-from-motion (SfM) scheme. In this respect automation of the system is achieved by thorough decision model based on robust statistics and error propagation of projective measurement uncertainties. This leads to a scene-independent set of only a few parameters. All system components are formulated in a general way, making it possible to cope with any single centered projection model, in particular with spherical cameras. In using wide field-of-view cameras the presented system is able to reliably retrieve poses and consistently reconstruct large scenes. A textured triangle mesh constructed on basis of the scene’s reconstructed depth, makes the system’s results suitable to function as reference models in a GPU driven analysis-by-synthesis framework for real-time tracking.     

Robust facial feature tracking under varying face pose and facial expression

This paper presents a hierarchical multi-state pose-dependent approach for facial feature detection and tracking under varying facial expression and face pose. For effective and efficient representation of feature points, a hybrid representation that integrates Gabor wavelets and gray-level profiles is proposed. To model the spatial relations among feature points, a hierarchical statistical face shape model is proposed to characterize both the global shape of human face and the local structural details of each facial component. Furthermore, multi-state local shape models are introduced to deal with shape variations of some facial components under different facial expressions. During detection and tracking, both facial component states and feature point positions, constrained by the hierarchical face shape model, are dynamically estimated using a switching hypothesized measurements (SHM) model. Experimental results demonstrate that the proposed method accurately and robustly tracks facial features in real time under different facial expressions and face poses.     

Facial landmarks detection using improved active shape model on android platform

Detection of facial feature is fundamental for applications such as security, biometrics, 3D face modeling and personal authentication. Active Shape Model (ASM) is one of the most popular local texture models for face detection. This paper presents an issue related to face detection based on ASM, and proposes an efficient extraction algorithm for facial landmarks suitable for use on mobile devices. We modifies the original ASM to improve its performance with three changes; (1) Improving the initialization model using the center of the eyes by using a feature map of color information, (2) Constructing modified model definition and fitting more landmarks than the classical ASM, and (3) Extending and building a 2-D profile model for detecting faces in input image. The proposed method is evaluated on dataset containing over 700 images of faces, and experimental results reveal that the proposed algorithm exhibited a significant improvement of over 10.2 % in average success ratio, compared to the classic ASM, clearly outperforming on success rate and computing time.     

基于CUDA加速的实时团块跟踪算法

为有效解决目标稳定实时跟踪问题,提出一种基于团块的目标跟踪方案.该算法利用目标的颜色和空间特征,通过四叉树分割将目标划分为多个区域,每个区域即为一个团块.根据团块的特征,结合模糊聚类的思想构建了团块目标模型,并定义团块间匹配准则,以此进行目标跟踪.同时,使用CUDA提供的强大并行计算能力提高程序执行速度.经过视频序列测试,表明了该设计方案的有效性和实时性.     

Real-time multiple people tracking using competitive condensation

The CONDENSATION (Conditional Density Propagation) algorithm has a robust tracking performance and suitability for real-time implementation. However, the CONDENSATION tracker has some difficulties with real-time implementation for multiple people tracking since it requires very complicated shape modelling and a large number of samples for precise tracking performance. Further, it shows a poor tracking performance in the case of close or partially occluded people. To overcome these difficulties, we present three improvements: First, we construct effective templates of people's shapes using the SOM (Self-Organizing Map). Second, we take the discrete HMM (Hidden Markov Modelling) for an accurate dynamical model of the people's shape transition. Third, we use the competition rule to separate close or partially occluded people effectively. Simulation results shows that the proposed CONDENSATION algorithm can achieve robust and real-time tracking in the image sequences of a crowd of people.