3D shape recovery from image focus using kernel regression in eigenspace

Shape from focus (SFF) is one of the optical passive methods for three dimensional (3D) shape recovery of an object from its two dimensional (2D) images. The focus measure plays important role in SFF algorithms. Mostly, conventional focus measures are based on gradient, so their performance is restricted under noisy conditions. Moreover, SFF methods also suffer from loss of focus information due to discreteness. This paper introduces a new SFF method based on principal component analysis (PCA) and kernel regression. The focus values are computed through PCA by considering a sequence of small 3D neighborhood for each object point. We apply unsupervised regression through Nadaraya and Watson Estimate (NWE) on depth values to get a refined 3D shape of the object. It reduces the effect of noise within a small surface area as well as approximates the accurate 3D shape by exploiting the depth dependencies in the neighborhood. Performance of the proposed scheme is investigated in the presence of different types of noises and textured areas. Experimental results demonstrate effectiveness of the proposed approach.     

Real-time face tracking and recognition by sparse eigentracker with associative mapping to 3D shape

This paper proposes a novel framework of real-time face tracking and recognition by combining two eigen-based methods. The first method is a novel extension of eigenface called augmented eigenface and the second method is a sparse 3D eigentemplate tracker controlled by a particle filter. The augmented eigenface is an eigenface augmented by an associative mapping to 3D shape that is specified by a set of volumetric face models. This paper discusses how to make up the augmented eigenface and how it can be used for inference of 3D shape from partial images. The associative mapping is also generalized to subspace-to-one mappings to cover photometric image changes for a fixed shape. A novel technique, called photometric adjustment, is introduced for simple implementation of associative mapping when an image subspace should be combined to a shape. The sparse 3D eigentemplate tracker is an extension of the 3D template tracker proposed by Oka et al. In combination with the augmented eigenface, the sparse 3D eigentemplate tracker facilitates real-time 3D tracking and recognition when a monocular image sequence is provided. In the tracking, sparse 3D eigentemplate is updated by the augmented eigenface while face pose is estimated by the sparse eigentracker. Since the augmented eigenface is constructed on the conventional eigenfaces, face identification and expression recognition are also accomplished efficiently during the tracking. In the experiment, an augmented eigenface was constructed from 25 faces where 24 images were taken in different lighting conditions for each face. Experimental results show that the augmented eigenface works with the 3D eigentemplate tracker for real-time tracking and recognition.     

Automatic reconstruction of 3D human motion pose from uncalibrated monocular video sequences based on markerless human motion tracking

We present a method to reconstruct human motion pose from uncalibrated monocular video sequences based on the morphing appearance model matching. The human pose estimation is made by integrated human joint tracking with pose reconstruction in depth-first order. Firstly, the Euler angles of joint are estimated by inverse kinematics based on human skeleton constrain. Then, the coordinates of pixels in the body segments in the scene are determined by forward kinematics, by projecting these pixels in the scene onto the image plane under the assumption of perspective projection to obtain the region of morphing appearance model in the image. Finally, the human motion pose can be reconstructed by histogram matching. The experimental results show that this method can obtain favorable reconstruction results on a number of complex human motion sequences.     

基于边的自适应实时三维跟踪

针对缺乏纹理特征的物体,提出了一种基于边的自适应实时三维跟踪方法。在已知物体三维模型的情况下,通过基于历史运动信息的物体边缘检测与跟踪,可以有效准确地求解出摄像机的外参。基于并扩展了现有的基于边的实时跟踪算法,其主要工作体现在以下三个方面： 1）提出自适应阈值和基于历史信息估计当前帧的运动趋势的方法,从而提高边匹配算法在快速运动时的稳定性;2）提出一种基于随机抽样一致性（RANSAC）的边匹配策略,可以有效剔除误匹配的边,从而提高复杂模型的跟踪稳定性;3）利用抽取轮廓边的算法将边跟踪算法从CAD模型扩展到一般的面片模型。实验结果证明了该方法的鲁棒高效,能够满足增强现实、虚拟装配等应用需求。     

EM-GPA: Generalized Procrustes analysis with hidden variables for 3D shape modeling

Aligning shapes is essential in many computer vision problems and generalized Procrustes analysis (GPA) is one of the most popular algorithms to align shapes. However, if some of the shape data are missing, GPA cannot be applied. In this paper, we propose EM-GPA, which extends GPA to handle shapes with hidden (missing) variables by using the expectation-maximization (EM) algorithm. For example, 2D shapes can be considered as 3D shapes with missing depth information due to the projection of 3D shapes into the image plane. For a set of 2D shapes, EM-GPA finds scales, rotations and 3D shapes along with their mean and covariance matrix for 3D shape modeling. A distinctive characteristic of EM-GPA is that it does not enforce any rank constraint often appeared in other work and instead uses GPA constraints to resolve the ambiguity in finding scales, rotations, and 3D shapes. The experimental results show that EM-GPA can recover depth information accurately even when the noise level is high and there are a large number of missing variables. By using the images from the FRGC database, we show that EM-GPA can successfully align 2D shapes by taking the missing information into consideration. We also demonstrate that the 3D mean shape and its covariance matrix are accurately estimated. As an application of EM-GPA, we construct a 2D + 3D AAM (active appearance model) using the 3D shapes obtained by EM-GPA, and it gives a similar success rate in model fitting compared to the method using real 3D shapes. EM-GPA is not limited to the case of missing depth information, but it can be easily extended to more general cases.     

Progressive 3D shape segmentation using online learning

In this article, we propose a progressive 3D shape segmentation method, which allows users to guide the segmentation with their interactions, and does segmentation gradually driven by their intents. More precisely, we establish an online framework for interactive 3D shape segmentation, without any boring collection preparation or training stages. That is, users can collect the 3D shapes while segment them, and the segmentation will become more and more precise as the accumulation of the shapes.Our framework uses Online Multi-Class LPBoost (OMCLP) to train/update a segmentation model progressively, which includes several Online Random forests (ORFs) as the weak learners. Then, it performs graph cuts optimization to segment the 3D shape by using the trained/updated segmentation model as the optimal data term. There exist three features of our framework. Firstly, the segmentation model can be trained gradually during the collection of the shapes. Secondly, the segmentation results can be refined progressively until users’ requirements are met. Thirdly, the segmentation model can be updated incrementally without retraining all shapes when users add new shapes. Experimental results demonstrate the effectiveness of our approach.     

A fast algorithm for ICP-based 3D shape biometrics

In a biometrics scenario, gallery images are enrolled into the database ahead of the matching step, which gives us the opportunity to build related data structures before the probe shape is examined. In this paper, we present a novel approach, called “Pre-computed Voxel Nearest Neighbor”, to reduce the computational time for shape matching in a biometrics context. The approach shifts the heavy computation burden to the enrollment stage, which is done offline. Experiments in 3D ear biometrics with 369 subjects and 3D face biometrics with 219 subjects demonstrate the effectiveness of our approach.     

Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video

3D video [IEEE Multimedia (1997) 18] is the ultimate image media recording dynamic visual events in the real world as is; it records time varying 3D object shape with high fidelity surface properties (i.e., color and texture). Its applications cover wide varieties of personal and social human activities: entertainment (e.g., 3D game and 3D TV), education (e.g., 3D animal picture books), sports (e.g., sport performance analysis), medicine (e.g., 3D surgery monitoring), culture (e.g., 3D archive of traditional dances), and so on. In this paper, we propose: (1) a PC cluster system for real-time reconstruction of dynamic 3D object action from multi-view video images, (2) a deformable 3D mesh model for reconstructing the accurate dynamic 3D object shape, and (3) an algorithm of rendering natural-looking texture on the 3D object surface from the multi-view video images. Experimental results with quantitative performance evaluations demonstrate the effectiveness of these methods in generating high fidelity 3D video from multi-view video images.     

多模式3维视频形状编码

目的 具有立体感和高端真实感的3D视频正越来越受到学术界和产业界的关注和重视,未来在3D影视、机器视觉、远程医疗、军事航天等领域将有着广泛的应用前景。对象基3D视频是未来3D视频技术的重要发展趋势,其中高效形状编码是对象基3D视频应用中的关键问题。但现有形状编码方法主要针对图像和视频对象,面向3D视频的形状编码算法还很少。为此,基于对象基3D视频的应用需求,提出一种基于轮廓和链码表示的高效多模式3D视频形状编码方法。方法 对于给定的3D视频形状序列逐帧进行对象轮廓提取并预处理后,进行对象轮廓活动性分析,将形状图像分成帧内模式编码图像和帧间预测模式编码图像。对于帧内编码图像,基于轮廓内链码方向约束和线性特征进行高效编码。对于帧间编码图像,采用基于链码表示的轮廓基运动补偿预测、视差补偿预测、联合运动与视差补偿预测等多种模式进行编码,以充分利用视点内对象轮廓的帧间时域相关性和视点间对象轮廓的空域相关性,从而达到高效编码的目的。结果 实验仿真结果显示所提算法性能优于经典和现有的最新同类方法,压缩效率平均能提高9.3%到64.8%不等。结论 提出的多模式3D视频形状编码方法可以有效去除对象轮廓的帧间和视点间冗余,能够进行高效编码压缩,性能优于现有同类方法,可广泛应用于对象基编码、对象基检索、对象基内容分析与理解等。     

PCA-based 3D shape reconstruction of human foot using multiple viewpoint cameras

This paper describes a multiple camera-based method to reconstruct the 3D shape of a human foot. From a foot database, an initial 3D model of the foot represented by a cloud of points is built. The shape parameters, which can characterize more than 92% of a foot, are defined by using the principal component analysis method. Then, using ＂active shape models＂, the initial 3D model is adapted to the real foot captured in multiple images by applying some constraints （edge points＇ distance and color variance）. We insist here on the experiment part where we demonstrate the efficiency of the proposed method on a plastic foot model, and also on real human feet with various shapes. We propose and compare different ways of texturing the foot which is needed for reconstruction. We present an experiment performed on the plastic foot model and on human feet and propose two different ways to improve the final 3D shapers accuracy according to the previous experiments＇ results. The first improvement proposed is the densification of the cloud of points used to represent the initial model and the foot database. The second improvement concerns the projected patterns used to texture the foot. We conclude by showing the obtained results for a human foot with the average computed shape error being only 1.06 mm.     

从视频中恢复三维人脸的实时方法

三维人脸恢复是视觉交互的一个难点问题,提出了一种从视频中实时恢复三维人脸的新方法.该方法利用主动形状模型进行人脸特征点提取和跟踪,确保了三维形状恢复和特征跟踪的有效性和一致性;采用非刚体形状和运动估计方法构建三维形变基,有效地适应人脸形状变化的多样性;采用非线性优化算法估算人脸姿态和三维形变基参数,实现了三维人脸形状和姿态的实时恢复.实验结果表明,该方法不仅能从视频中实时恢复三维人脸模型,而且可有效跟踪人脸各种姿态的变化.     

Automatic generation of LEGO building instructions from multiple photographic images of real objects

We introduce a system to reconstruct large scale LEGO models from multiple two dimensional images of objects taken from different views. We employ a unit voxel with an edge length ratio of 5:5:6 for the shape from silhouette method that reconstructs an octree voxel-based three dimensional model with color information from images. We then convert the resulting voxel model with color information into a LEGO sculpture. In order to minimize the number of LEGO bricks, we use a stochastic global optimization method, simulated annealing, to hollow the model as much as possible but keep its strength for portability. Several real complex LEGO models are provided to demonstrate the effectiveness of the proposed method.     

基于三维变换形状搜索的ASM人脸特征点定位

通过三维变换的特征点搜索方法对二维人脸图像进行形状搜索,形状搜索以人脸的标准三维模型为基础,将二维坐标三维化;通过三维变换、投影等过程迭代搜索逼近目标形状,最后得到三维变换的10个姿态参数。测试结果表明：在对非活动形状模型(ASM)训练集中的多姿态人脸进行形状搜索时,三维变换方法具有更好的逼近效果。较好地模拟了人脸姿态的实际变化,有效地解决了ASM人脸偏转的非线性问题。     

System for reconstruction of three-dimensional micro objects from multiple photographic images

We introduce a system to reconstruct a three-dimensiojnal (3D) polygonal model of 3D micro objects with outer dimensions ranging from several hundred microns to several millimeters from multiple two-dimensional (2D) images of an object taken from different views. The data acquisition system consists of a digital microscope that captures still images at a resolution of 1600 × 1200 pixels and a computer-controlled turntable. We employ the shape-from-silhouette (SFS) method to construct a voxel-based 3D model from silhouette images. The concave shapes are further carved by using the space carving technique. In order to make the resulting model compatible with a commercial CAD/CAM system, the voxel model is converted into a triangular mesh using the marching cubes algorithm. Because the mesh generated from the voxel model by using the marching cubes algorithm inherits the staircase effect, the mesh is adjusted to recover the object precisely by using silhouette images. Finally, we evaluate the accuracy of the proposed method. The reconstructed models of complex micro objects indicate the effectiveness of the 3D shape reconstruction system for micro objects.     

三维视频编码技术的发展与挑战

介绍了三维视频编码的核心技术。首先比较了纯视频格式和深度增强格式三维视频编码技术的发展方向及面临的挑战,其中重点分析了深度估计技术和视点合成技术的研究思路;然后概括了国际3DV/FTV标准的制定现状;最后对三维视频编码技术进行了总结和展望。     

使用形状变化描述三维模型

针对三维模型检索中的形状特征提取问题,提出利用三维模型自身形状变化信息构造形状特征描述符的方法.首先选择一组等间距互相平行的平面切割三维模型,得到三维模型的切片集合;然后定义相邻切片的 差来描述切片间的形状变化,并通过所有相邻切片间的差值来反映三维模型自身的形状变化,以此作为三维模型的形状特征描述符.该方法与三维模型的旋转、平移无关,同时不依赖于模型的点云分布,并且精简模型三角面片对算法的影响较小.实验结果验证了该方法的有效性.     

改进的三维模型形状分布检索算法

针对传统D1距离形状分布函数获取采样点计算复杂、模型内容描述不充分和检索速率低下等问题提出了一种改进方法。该方法的关键点是：首先采用平移和缩放对模型进行标准化处理,用于减少面片之间的差异,使得采样点均匀地落在模型的表面;其次采用三角面片的索引号进行随机数的生成,并且利用三角面片的重心和质心进行有效的计算,以便用于缩短模型的处理时间和提高检索速率。利用普林斯顿大学三维模型数据库中的部分模型作为实验数据,实现结果表明：改进的方法不会降低模型的检索性能,并有效地减少了模型查询和处理时间。