Video arrays for real-time tracking of person,head, and face in an intelligent room

Real-time three-dimensional tracking of people is an important requirement for a growing number of applications. In this paper we describe two trackers; both of them use a network of video cameras for person tracking. These trackers are called a rectilinear video array tracker (R-VAT) and an omnidirectional video array tracker (O-VAT), indicating the two different ways of video capture. The specific objectives of this paper are twofold: (i) to present a systematic comparison of these two trackers using an extensive series of experiments conducted in an `intelligent' room; (ii) to develop a real-time system for tracking the head and face of a person, as an extension of the O-VAT approach. The comparative research indicates that O-VAT is more robust to the number of people, less complex and runs faster, needs manual camera calibration, and the integrated omnidirectional video network has better reconfigurability. The person head and face tracker study shows that such a system can serve as a most effective input stage for face recognition and facial expression analysis modules.     

一种改进的特征脸方法

在实际应用中,我们发现,已有的一些人脸识别方法对于每人一个样本的识别来说,效果不太理想。鉴于此,本文将在传统的特征脸方法理论基础上提出一种改进的特征脸方法——特征半脸方法。所谓特征半脸方法,就是把人脸图像分成上下两个部分,分别应用特征脸方法,最后在识别计算距离时上部采用较大的权重,下部采用较小的权重,求得综合距离最小的人脸图像序号,从而完成人脸识别的方法。我们把特征脸和特征半脸方法进行了对比实验,结果表明:新的特征半脸方法优于传统的特征脸方法。     

基于样例学习的任意光照下的人脸3D重建方法

待匹配的人脸图像与数据库中的原型图像之间的光照差异是自动人脸识别的主要瓶颈问题之一。提出了一种基于样例学习方式的3D人脸形状重建方法,既可以生成任意光照条件下的数据库中人脸图像,也可以对待识别图像进行重新光照,合成无阴影的图像。该方法在建立人脸数据库时利用光度立体技术分离人脸图像的纹理和形状信息,并用多面体模型在最小二乘意义下恢复其3D信息并更新法向量场以克服阴影误差,从而可以利用计算机图形学的方法合成任意光照条件下和小角度姿态改变时的人脸图像;在识别时采用数据库中3D数据的线性组合形式对输入图像建模,以估计其3D信息,从而可以重新照明。在YaleB人脸数据库上的实验表明,在建立3D人脸数据库后,该方法可以快速恢复输入单幅图像中人脸的3D信息,并生成任意光照条件的该人脸图像。     

2D representation of facial surfaces for multi-pose 3D face recognition

The increasing availability of 3D facial data offers the potential to overcome the intrinsic difficulties faced by conventional face recognition using 2D images. Instead of extending 2D recognition algorithms for 3D purpose, this letter proposes a novel strategy for 3D face recognition from the perspective of representing each 3D facial surface with a 2D attribute image and taking the advantage of the advances in 2D face recognition. In our approach, each 3D facial surface is mapped homeomorphically onto a 2D lattice, where the value at each site is an attribute that represents the local 3D geometrical or textural properties on the surface, therefore invariant to pose changes. This lattice is then interpolated to generate a 2D attribute image. 3D face recognition can be achieved by applying the traditional 2D face recognition techniques to obtained attribute images. In this study, we chose the pose invariant local mean curvature calculated at each vertex on the 3D facial surface to construct the 2D attribute image and adopted the eigenface algorithm for attribute image recognition. We compared our approach to state-of-the-art 3D face recognition algorithms in the FRGC (Version 2.0), GavabDB and NPU3D database. Our results show that the proposed approach has improved the robustness to head pose variation and can produce more accurate 3D multi-pose face recognition.     

实用人脸识别系统的本征脸法实现

本征脸法将图像看做矩阵,计算本征值和对应的本征向量作为代数特征进行识别,具有无需提取眼嘴鼻几何特征的优点,但在单样本时识别率不高,且在人脸模式数较大时计算量大,将人脸模式的多外样本作为子模式,并将较多的人脸模式部分相交地分组,采用基于贝叶斯理论和结合规则,以减小计算量和便于识别系统的扩展,有用ORL和AR图像库的实验表明,本征脸法在采用多样本训练后,识别率和识别时间都较好;识别系统可分布并行计算加快训练,在增加新人脸模式时,系统可以方便地进行扩展,并保持较高的识别率。     

Sparse two-dimensional local discriminant projections for feature extraction

Two-dimensional local graph embedding discriminant analysis (2DLGEDA) and two-dimensional discriminant locality preserving projections (2DDLPP) were recently proposed to directly extract features form 2D face matrices to improve the performance of two-dimensional locality preserving projections (2DLPP). But all of them require a high computational cost and the learned transform matrices lack intuitive and semantic interpretations. In this paper, we propose a novel method called sparse two-dimensional locality discriminant projections (S2DLDP), which is a sparse extension of graph-based image feature extraction method. S2DLDP combines the spectral analysis and L₁-norm regression using the Elastic Net to learn the sparse projections. Differing from the existing 2D methods such as 2DLPP, 2DDLP and 2DLGEDA, S2DLDP can learn the sparse 2D face profile subspaces (also called sparsefaces), which give an intuitive, semantic and interpretable feature subspace for face representation. We point out that using S2DLDP for face feature extraction is, in essence, to project the 2D face images on the semantic face profile subspaces, on which face recognition is also performed. Experiments on Yale, ORL and AR face databases show the efficiency and effectiveness of S2DLDP.     

Face recognition with one training image per person

At present there are many methods that could deal well with frontal view face recognition. However, most of them cannot work well when there is only one training image per person. In this paper, an extension of the eigenface technique, i.e. projection-combined principal component analysis, (PC)²A, is proposed. (PC)²A combines the original face image with its horizontal and vertical projections and then performs principal component analysis on the enriched version of the image. It requires less computational cost than the standard eigenface technique and experimental results show that on a gray-level frontal view face database where each person has only one training image, (PC)²A achieves 3–5% higher accuracy than the standard eigenface technique through using 10–15% fewer eigenfaces.     

Real-time shape tracking of facial landmarks

Detection of facial landmarks and accurate tracking of their shape are essential in real-time applications such as virtual makeup, where users can see the makeup’s effect by moving their face in diverse directions. Typical face tracking techniques detect facial landmarks and track them using a point tracker such as the Kanade-Lucas-Tomasi (KLT) point tracker. Typically, 5 or 64 points are used for tracking a face. Even though these points are enough to track the approximate locations of facial landmarks, they are not sufficient to track the exact shape of facial landmarks. In this paper, we propose a method that can track the exact shape of facial landmarks in real-time by combining a deep learning technique and a point tracker. We detect facial landmarks accurately using SegNet, which performs semantic segmentation based on deep learning. Edge points of detected landmarks are tracked using the KLT point tracker. In spite of its popularity, the KLT point tracker suffers from the point loss problem. We solve this problem by executing SegNet periodically to recalculate the shape of facial landmarks. That is, by combining the two techniques, we can avoid the computational overhead of SegNet and the point loss problem of the KLT point tracker, which leads to accurate real-time shape tracking. We performed several experiments to evaluate the performance of our method and report some of the results herein.

     

Pose-Robust Facial Expression Recognition Using View-Based 2D $+$ 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D 3D active appearance model (AAM) that extends the 2D 3D AAM to the view-based approach, where one independent face model is used for a specific view and an appropriate face model is selected for the input face image. Our extension has been conducted in many aspects. First, we use principal component analysis with missing data to construct the 2D 3D AAM due to the missing data in the posed face images. Second, we develop an effective model selection method that directly uses the estimated pose angle from the 2D 3D AAM, which makes face tracking pose-robust and feature extraction for facial expression recognition accurate. Third, we propose a double-layered generalized discriminant analysis (GDA) for facial expression recognition. Experimental results show the following: 1) The face tracking by the view-based 2D 3D AAM, which uses multiple face models with one face model per each view, is more robust to pose change than that by an integrated 2D 3D AAM, which uses an integrated face model for all three views; 2) the double-layered GDA extracts good features for facial expression recognition; and 3) the view-based 2D 3D AAM outperforms other existing models at pose-varying facial expression recognition.     

Real-time 3D face tracking based on active appearance model constrained by depth data

Active Appearance Model (AAM) is an algorithm for fitting a generative model of object shape and appearance to an input image. AAM allows accurate, real-time tracking of human faces in 2D and can be extended to track faces in 3D by constraining its fitting with a linear 3D morphable model. Unfortunately, this AAM-based 3D tracking does not provide adequate accuracy and robustness, as we show in this paper. We introduce a new constraint into AAM fitting that uses depth data from a commodity RGBD camera (Kinect). This addition significantly reduces 3D tracking errors. We also describe how to initialize the 3D morphable face model used in our tracking algorithm by computing its face shape parameters of the user from a batch of tracked frames. The described face tracking algorithm is used in Microsoft's Kinect system.     

从视频中恢复三维人脸的实时方法

三维人脸恢复是视觉交互的一个难点问题,提出了一种从视频中实时恢复三维人脸的新方法.该方法利用主动形状模型进行人脸特征点提取和跟踪,确保了三维形状恢复和特征跟踪的有效性和一致性;采用非刚体形状和运动估计方法构建三维形变基,有效地适应人脸形状变化的多样性;采用非线性优化算法估算人脸姿态和三维形变基参数,实现了三维人脸形状和姿态的实时恢复.实验结果表明,该方法不仅能从视频中实时恢复三维人脸模型,而且可有效跟踪人脸各种姿态的变化.     

Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.     

The constrained SLAM framework for non-instrumented augmented reality

This paper addresses the challenging issue of marker less tracking for Augmented Reality. It proposes a real-time camera localization in a partially known environment, i.e. for which a geometric 3D model of one static object in the scene is available. We propose to take benefit from this geometric model to improve the localization of keyframe-based SLAM by constraining the local bundle adjustment process with this additional information. We demonstrate the advantages of this solution, called contrained SLAM, on both synthetic and real data and present very convincing augmentation of 3D objects in real-time. Using this tracker, we also propose an interactive augmented reality system for training application. This system, based on a Optical See-Through Head Mounted Display, allows to augment the users vision field with virtual information accurately co-registered with the real world. To keep greatly benefit of the potential of this hand free device, the system combines the tracker module with a simple user-interaction vision-based module to provide overlaid information in response to user requests.     

Three-Dimensional Face Pose Detection and Tracking Using Monocular Videos: Tool and Application

Recently, we have proposed a real-time tracker that simultaneously tracks the 3-D head pose and facial actions in monocular video sequences that can be provided by low quality cameras. This paper has two main contributions. First, we propose an automatic 3-D face pose initialization scheme for the real-time tracker by adopting a 2-D face detector and an eigenface system. Second, we use the proposed methods—the initialization and tracking—for enhancing the human–machine interaction functionality of an AIBO robot. More precisely, we show how the orientation of the robot's camera (or any active vision system) can be controlled through the estimation of the user's head pose. Applications based on head-pose imitation such as telepresence, virtual reality, and video games can directly exploit the proposed techniques. Experiments on real videos confirm the robustness and usefulness of the proposed methods.      

DCT系数重组在特征脸中的应用研究

针对现在图像压缩主要采用DCT变换方法,提出了一种对DCT系数进行重组的方法,获取三级多分辨率子带。通过对各单独子带实施特征脸方法可得到不同的识别效果。提出在子带[S0]和子带[S1],[S2]和[S3]上进行特征脸方法识别,如两者结果不一致,则在满足一定门限条件下输入多张人脸,分析了所提出方法与原方法所需识别时间的关系。在ORL库和YALE库上的实验表明,提出方法的识别率要远高于直接采用特征脸方法。     

Illumination invariant object tracking with adaptive sparse representation

Since the introduction of the sparse representation-based tracking method named ?₁ tracker, there have been further studies into this tracking framework with promised results in challenging video sequences. However, in the situation of large illumination changes and shadow casting, the tracked object cannot be modeled efficiently by sparse representation templates. To overcome this problem, we propose a new illumination invariant tracker based on photometric normalization techniques and the sparse representation framework. With photometric normalization methods, we designed a new illumination invariant template presentation for tracking that eliminates the illumination influences, such as brightness variation and shadow casting. For a higher tracking accuracy, we introduced a strategy that adaptively selects the optimum template presentation at the update step of the tracking process. The experiments show that our approach outperforms the previous ?₁ and some state-of-the-art algorithms in tracking sequences with severe illumination effects.     

特征脸空间中夹角最小法则的人脸识别算法

特征脸方法是人脸识别领域中的一种重要方法。本文在特征脸的基础上提出了一种新的方法,在特征脸空间,用向量之间的夹角来衡量图像之间的相似度,用最近邻法对图像进行分类和识别。我们用ORL提供的标准人脸库进行了测试,并与传统的特征脸方法进行了比较。结果表明,新方法的识别正确率明显高于传统特征脸法。     

基于加权特征脸法的人脸识别

人脸识别是模式识别领域中一个相当困难而又有理论意义和实际价值的研究课题。本文在传统的特征脸方法的理论基础上提出一种改进的特征脸方法,就是把人脸图像分成上中下三个部分,分别应用特征脸方法,在识别计算距离时赋予不同的权值,最后确定综合距离最小的人脸图像。把这种方法和传统特征脸方法进行了对比实验,结果证明了该方法的可行性和良好的抗畸变能力。     

结合形状滤波和几何图像的3D人脸识别算法

表情变化是3维人脸精确识别面临的主要问题,为此提出一种新的对表情鲁棒的匹配方法。通过形状滤波器将人脸空域形状分成不同频率的3个部分：低频部分对应表情变化;高频部分代表白噪声;包含身份区分度最大的中频信息作为表情不变特征。再利用网格平面参数化,将人脸网格映射到边界为正四边形的平面区域内,经过线性插值采样得到3维形状的2维几何图像。最后通过图像匹配识别人脸。FRGC v2人脸数据库上的实验结果表明,使用形状滤波能显著提高算法的精度和鲁棒性。     

Robust visual tracking using information theoretical learning

This paper presents a novel online object tracking algorithm with sparse representation for learning effective appearance models under a particle filtering framework. Compared with the state-of-the-art ? ₁ sparse tracker, which simply assumes that the image pixels are corrupted by independent Gaussian noise, our proposed method is based on information theoretical Learning and is much less sensitive to corruptions; it achieves this by assigning small weights to occluded pixels and outliers. The most appealing aspect of this approach is that it can yield robust estimations without using the trivial templates adopted by the previous sparse tracker. By using a weighted linear least squares with non-negativity constraints at each iteration, a sparse representation of the target candidate is learned; to further improve the tracking performance, target templates are dynamically updated to capture appearance changes. In our template update mechanism, the similarity between the templates and the target candidates is measured by the earth movers’ distance(EMD). Using the largest open benchmark for visual tracking, we empirically compare two ensemble methods constructed from six state-of-the-art trackers, against the individual trackers. The proposed tracking algorithm runs in real-time, and using challenging sequences performs favorably in terms of efficiency, accuracy and robustness against state-of-the-art algorithms.