Global Texture Mapping for Dynamic Objects

We propose a novel framework to generate a global texture atlas for a deforming geometry. Our approach distinguishes from prior arts in two aspects. First, instead of generating a texture map for each timestamp to color a dynamic scene, our framework reconstructs a global texture atlas that can be consistently mapped to a deforming object. Second, our approach is based on a single RGB‐D camera, without the need of a multiple‐camera setup surrounding a scene. In our framework, the input is a 3D template model with an RGB‐D image sequence, and geometric warping fields are found using a state‐of‐the‐art non‐rigid registration method [GXW*15] to align the template mesh to noisy and incomplete input depth images. With these warping fields, our multi‐scale approach for texture coordinate optimization generates a sharp and clear texture atlas that is consistent with multiple color observations over time. Our approach is accelerated by graphical hardware and provides a handy configuration to capture a dynamic geometry along with a clean texture atlas. We demonstrate our approach with practical scenarios, particularly human performance capture. We also show that our approach is resilient on misalignment issues caused by imperfect estimation of warping fields and inaccurate camera parameters.     

单幅人脸图像的全景纹理图生成方法

目的针对从单幅人脸图像中恢复面部纹理图时获得的信息不完整、纹理细节不够真实等问题,提出一种基于生成对抗网络的人脸全景纹理图生成方法。方法将2维人脸图像与3维人脸模型之间的特征关系转换为编码器中的条件参数,从图像数据与人脸条件参数的多元高斯分布中得到隐层数据的概率分布,用于在生成器中学习人物的头面部纹理特征。在新创建的人脸纹理图数据集上训练一个全景纹理图生成模型,利用不同属性的鉴别器对输出结果进行评估反馈,提升生成纹理图的完整性和真实性。结果实验与当前最新方法进行了比较,在Celeb A-HQ和LFW(labled faces in the wild)数据集中随机选取单幅正面人脸测试图像,经生成结果的可视化对比及3维映射显示效果对比,纹理图的完整度和显示效果均优于其他方法。通过全局和面部区域的像素量化指标进行数据比较,相比于UVGAN,全局峰值信噪比(peak signal to noise ratio,PSNR)和全局结构相似性(structural similarity index,SSIM)分别提高了7.9 d B和0.088,局部PSNR和局部SSIM分别提高了2.8 d B和0...     

Extending 3D Lucas–Kanade tracking with adaptive templates for head pose estimation

The Lucas–Kanade tracker (LKT) is a commonly used method to track target objects over 2D images. The key principle behind the object tracking of an LKT is to warp the object appearance so as to minimize the difference between the warped object’s appearance and a pre-stored template. Accordingly, the 2D pose of the tracked object in terms of translation, rotation, and scaling can be recovered from the warping. To extend the LKT for 3D pose estimation, a model-based 3D LKT assumes a 3D geometric model for the target object in the 3D space and tries to infer the 3D object motion by minimizing the difference between the projected 2D image of the 3D object and the pre-stored 2D image template. In this paper, we propose an extended model-based 3D LKT for estimating 3D head poses by tracking human heads on video sequences. In contrast to the original model-based 3D LKT, which uses a template with each pixel represented by a single intensity value, the proposed model-based 3D LKT exploits an adaptive template with each template pixel modeled by a continuously updated Gaussian distribution during head tracking. This probabilistic template modeling improves the tracker’s ability to handle temporal fluctuation of pixels caused by continuous environmental changes such as varying illumination and dynamic backgrounds. Due to the new probabilistic template modeling, we reformulate the head pose estimation as a maximum likelihood estimation problem, rather than the original difference minimization procedure. Based on the new formulation, an algorithm to estimate the best head pose is derived. The experimental results show that the proposed extended model-based 3D LKT achieves higher accuracy and reliability than the conventional one does. Particularly, the proposed LKT is very effective in handling varying illumination, which cannot be well handled in the original LKT.     

Real-Time 3D Head Pose Tracking Through 2.5D Constrained Local Models with Local Neural Fields

Tracking the head in a video stream is a common thread seen within computer vision literature, supplying the research community with a large number of challenging and interesting problems. Head pose estimation from monocular cameras is often considered an extended application after the face tracking task has already been performed. This often involves passing the resultant 2D data through a simpler algorithm that best fits the data to a static 3D model to determine the 3D pose estimate. This work describes the 2.5D constrained local model, combining a deformable 3D shape point model with 2D texture information to provide direct estimation of the pose parameters, avoiding the need for additional optimization strategies. It achieves this through an analytical derivation of a Jacobian matrix describing how changes in the parameters of the model create changes in the shape within the image through a full-perspective camera model. In addition, the model has very low computational complexity and can run in real-time on modern mobile devices such as tablets and laptops. The point distribution model of the face is built in a unique way, so as to minimize the effect of changes in facial expressions on the estimated head pose and hence make the solution more robust. Finally, the texture information is trained via local neural fields—a deep learning approach that utilizes small discriminative patches to exploit spatial relationships between the pixels and provide strong peaks at the optimal locations.

     

Active blobs: region-based, deformable appearance models

A region-based approach to nonrigid motion tracking is described. Shape is defined in terms of a deformable triangular mesh that captures object shape plus a color texture map that captures object appearance. Photometric variations are also modeled. Nonrigid shape registration and motion tracking are achieved by posing the problem as an energy-based, robust minimization procedure. The approach provides robustness to occlusions, wrinkles, shadows, and specular highlights. The formulation is tailored to take advantage of texture mapping hardware available in many workstations, PCs, and game consoles. This enables nonrigid tracking at speeds approaching video rate.     

利用增广拉格朗日乘子的鲁棒跟踪算子

针对视频目标鲁棒跟踪问题,提出了一种基于稀疏表示的生成式算法。首先提取特征构建目标和背景模板,并利用随机抽样获得足够多的候选目标状态;然后利用多任务反向稀疏表示算法得到稀疏系数矢量构造相似度测量图,这里引入了增广拉格朗日乘子(ALM)算法解决L₁-min难题;最后从相似度图中使用加性池运算提取判别信息选择与目标模板相似度最高并与背景模板相似度最小的候选目标状态作为跟踪结果,该算法是在贝叶斯滤波框架下实现的。为了适应跟踪过程中目标外观由于光照变化、遮挡、复杂背景以及运动模糊等场景引起的变化,制定了简单却有效的更新机制,对目标和背景模板进行更新。对仿真结果的定性和定量评估均表明与其他跟踪算法相比,所提算法的跟踪准确性和稳定性有了一定的提高,能有效地解决光照和尺度变化、遮挡、复杂背景等场景的跟踪难题。     

Face tracking with automatic model construction

This paper describes an active model with a robust texture model built on-line. The model uses one camera and it is able to operate without active illumination. The texture model is defined by a series of clusters, which are built in a video sequence using previously encountered samples. This model is used to search for the corresponding element in the following frames. An on-line clustering method, named leaderP is described and evaluated on an application of face tracking. A 20-point shape model is used. This model is built offline, and a robust fitting function is used to restrict the position of the points. Our proposal is to serve as one of the stages in a driver monitoring system. To test it, a new set of sequences of drivers recorded outdoors and in a realistic simulator has been compiled. Experimental results for typical outdoor driving scenarios, with frequent head movement, turns and occlusions are presented. Our approach is tested and compared with the Simultaneous Modeling and Tracking (SMAT) [1], and the recently presented Stacked Trimmed Active Shape Model (STASM) [2], and shows better results than SMAT and similar fitting error levels to STASM, with much faster execution times and improved robustness.     

匹配图像与3维模型特征点的真实感3维头重建

提出并实现一种基于两张正交图像和一个标准3维头模型,并利用2D图像特征点和3D模型特征点的匹配进行3维头模型重建的算法。首先,进行面部区域和头发区域的分割,利用色彩传递对输入图像进行颜色处理。对正面图像利用改进后的ASM（主动形状模型）模型进行特征点定位。改进局部最大曲率跟踪(LMCT)方法,更为鲁棒的定位了侧面特征点。在匹配图像特征点与标准3维头上预先定义的特征点的基础上,利用径向基函数进行标准头形变,获得特定人的3维头部形状模型。采用重建好的3维头作为桥梁,自动匹配输入图像,进行无缝纹理融合。最后,将所得纹理映射到形状模型上,获得对应输入图像的特定真实感3维头模型。     

基于人脸标准化的纹理和光照保持3D人脸重构

现有人脸纹理重建方法对于人脸的皱纹、胡须、瞳孔颜色等重建效果往往不够细致.为了解决此问题,文中提出基于人脸标准化的纹理和光照保持3D人脸重构.首先对2D人脸图像标准化,使用光照信息和对称纹理重构人脸自遮挡区域的纹理.然后依据2D-3D点对应关系从标准化的2D人脸图像获取相应的3D人脸纹理,结合人脸形状重构和纹理信息,得到最终的3D人脸重构结果.实验表明文中方法有效保留原始2D图像的纹理和光照信息,重构的人脸更自然,具有更丰富的人脸细节.     

3D model based tracking for omnidirectional vision: A new spherical approach

The current work addresses the problem of 3D model tracking in the context of monocular and stereo omnidirectional vision in order to estimate the camera pose. To this end, we track 3D objects modeled by line segments because the straight line feature is often used to model the environment. Indeed, we are interested in mobile robot navigation using omnidirectional vision in structured environments. In the case of omnidirectional vision, 3D straight lines are projected as conics in omnidirectional images. Under certain conditions, these conics may have singularities.In this paper, we present two contributions. We, first, propose a new spherical formulation of the pose estimation withdrawing singularities, using an object model composed of lines. The theoretical formulation and the validation on synthetic images thus show that the new formulation clearly outperforms the former image plane one. The second contribution is the extension of the spherical representation to the stereovision case. We consider in the paper a sensor which combines a camera and four mirrors. Results in various situations show the robustness to illumination changes and local mistracking. As a final result, the proposed new stereo spherical formulation allows us to localize online a robot indoor and outdoor whereas the classical formulation fails.     

基于标准肤色的人脸图像纹理合成与三维重建应用

本文提出一种基于单幅人脸图像并结合标准肤色的人脸图像纹理合成和三维重建算法.首先,利用ASM算法提取人脸特征点,并通过基于局部线性嵌入算法的编辑传播实现颜色转换,使图像人脸色调与三维人脸模型标准肤色一致.接着,将人脸图像五官区域与标准肤色图进行泊松融合,并考虑眉毛遮挡情况,利用人脸对称性或眉毛模板还原眉毛.尤其对于半遮挡眉毛,采用Li模型和角点检测相结合的方法重建眉毛轮廓,得到最终人脸纹理图.最后通过纹理映射将人脸纹理图映射到三维人脸模型上,得到较好的个性化三维人脸重建效果.实验表明,本文算法能够适用于不同复杂背景和光照条件下拍摄的人脸图像,具有较快的处理速度,能够应用于人脸实时重建产品中.     

立体摄像系统及其在真实感头部化身生成中的应用

提出了一种新的真实感头部化身生成技术,该技术利用立体摄像机产生的二维图片和深度图生成化身脸部,而用三维模型和贴图生成化身其他部位,从而实现用有限的硬件和运算量得到较真实的头部化身。     

3D SSD tracking with estimated 3D planes

We present a tracking method where full camera position and orientation is tracked from intensity differences in a video sequence. The camera pose is calculated based on 3D planes, and hence does not depend on point correspondences. The plane based formulation also allows additional constraints to be naturally added, e.g., perpendicularity between walls, floor and ceiling surfaces, co-planarity of wall surfaces etc. A particular feature of our method is that the full 3D pose change is directly computed from temporal image differences without making a commitment to a particular intermediate (e.g., 2D feature) representation. We experimentally compared our method with regular 2D SSD tracking and found it more robust and stable. This is due to 3D consistency being enforced even in the low level registration of image regions. This yields better results than first computing (and hence committing to) 2D image features and then from these compute 3D pose.     

局部特征描述改进的LK跟踪注册方法

针对KLT跟踪方法抗光照变化和抗遮挡较差的问题,提出一种使用局部特征描述改进的LK跟踪注册方法(DF-LK)。使用ORB特征点求解初始位姿,通过自适应非极大值抑制重新划分特征点,选择均匀分布的特征点作为LK方法跟踪的控制点集。相邻帧图像之间的单应性矩阵通过在DF描述后的图像上使用LK方法进行求解,跟踪的结果由向前向后错误检测进行评估,由单应性矩阵和初始位姿求解出当前帧的摄像机位姿,并叠加虚拟信息。实验结果表明,该方法在光照变化、部分遮挡和透视变化时均有较好的稳定性和鲁棒性。     

An efficient camera calibration method for vision-based head tracking

The aim of this study is to develop and evaluate an efficient camera calibration method for vision-based head tracking. Tracking head movements is important in the design of an eye-controlled human/computer interface. A vision-based head tracking system is proposed to allow the user's head movements in the design of the eye-controlled human/computer interface. We propose an efficient camera calibration method to track the three-dimensional position and orientation of the user's head accurately. We also evaluate the performance of the proposed method and the influence of the configuration of calibration points on the performance. The experimental error analysis results showed that the proposed method can provide more accurate and stable pose (i.e. position and orientation) of the camera than the direct linear transformation method which has been used in camera calibration. The results for this study can be applied to the tracking of head movements related to the eye-controlled human/computer interface and the virtual reality technology.     

3D head tracking under partial occlusion

A new algorithm for 3D head tracking under partial occlusion from 2D monocular image sequences is proposed. The extended superquadric (ESQ) is used to generate a geometric 3D face model in order to reduce the shape ambiguity during tracking. Optical flow is then regularized by this model to estimate the 3D rigid motion. To deal with occlusion, a new motion segmentation algorithm using motion residual error analysis is developed. The occluded areas are successfully detected and discarded as noise. Furthermore, accumulation error is heavily reduced by a new post-regularization process based on edge flow. This makes the algorithm more stable over long image sequences. The algorithm is applied to both synthetic occlusion sequence and real image sequences. Comparisons with the ground truth indicate that our method is effective and is not sensitive to occlusion during head tracking.     

A Tracking Registration Method for Augmented Reality Based on Multi-modal Template Matching and Point Clouds

In order to overcome the defects where the surface of the object lacks sufficient texture features and the algorithm cannot meet the real-time requirements of augmented reality, a markerless augmented reality tracking registration method based on multimodal template matching and point clouds is proposed. The method first adapts the linear parallel multi-modal LineMod template matching method with scale invariance to identify the texture-less target and obtain the reference image as the key frame that is most similar to the current perspective. Then, we can obtain the initial pose of the camera and solve the problem of re-initialization because of tracking registration interruption. A point cloud-based method is used to calculate the precise pose of the camera in real time. In order to solve the problem that the traditional iterative closest point (ICP) algorithm cannot meet the real-time requirements of the system, Kd-tree (k-dimensional tree) is used under the graphics processing unit (GPU) to replace the part of finding the nearest points in the original ICP algorithm to improve the speed of tracking registration. At the same time, the random sample consensus (RANSAC) algorithm is used to remove the error point pairs to improve the accuracy of the algorithm. The results show that the proposed tracking registration method has good real-time performance and robustness.

     

稀疏表示的Lucas-Kanade目标跟踪

提出一种新的目标跟踪算法,将稀疏表示应用于LK(Lucas-Kanade)图像配准框架.通过最小化校准误差的L1范数来求解目标的状态参数,从而实现对目标的准确跟踪.对目标同时建立两个外观模型:动态字典和静态模板,其中动态模型由动态字典的稀疏表示来描述目标外观.为了解决由于动态字典不断更新造成的跟踪漂移问题,一个两阶段迭代机制被采用.两个阶段所采用的目标模型分别为动态字典和静态模板.大量的实验结果表明,本文算法能有效应对外观变化、局部遮挡、光照变化等挑战,同时具有较好的实时性.     

一种基于多信息综合的人脸跟踪算法

随着智能监视系统的发展需求,人脸跟踪作为智能监视系统中的一类特殊对象的跟踪方法,在跟踪时对环境的适应性要求很高。文章提出了一种基于多信息综合特征的,在变动场景下,对人脸进行跟踪的算法。为了使人脸跟踪在环境发生变化的情况下,可以进行有效跟踪,算法中用人脸区域内部的颜色信息、亮度梯度信息等获得综合的判别特征。在跟踪时,对于人脸的大小、朝向的变化以及光线变化,背景物等的干扰均有较强的鲁棒性。实验结果表明,算法在摄像机转动、仰俯、光线变化、人头晃动以及背景颜色分布与人脸近似等情况下均可完成有效跟踪。     

Efficient object tracking by condentional and cascaded image sensing

We introduce a robust multi-object tracking for abstract multi-dimensional feature vectors. The Condensation and the Wavelet Approximated Reduced Vector Machine (W-RVM) approach are joined to spend only as much as necessary effort for easy to discriminate regions (Condensation) and measurement locations (W-RVM) of the feature space, but most for regions and locations with high statistical likelihood to contain the object of interest. The new 3D Cascaded Condensation Tracking (CCT) yields more than 10 times faster tracking than state-of-art detection methods. We demonstrate HCI applications by high resolution face tracking within a large camera scene with an active dual camera system.