Observation-oriented silhouette-aware fast full body tracking with Kinect

We present a novel approach to track full human body mesh with a single depth camera, e.g. Microsoft Kinect, using a template body model. The proposed observation-oriented tracking mainly targets at fitting the body mesh silhouette to the 2D user boundary in video stream by deforming the body. It is fast to be integrated into real-time or interactive applications, which is impossible with traditional iterative optimization based approaches. Our method is a composite of two main stages: user-specific body shape estimation and on-line body tracking. We first develop a novel method to fit a 3D morphable human model to the actual body shape of the user in front of the depth camera. A strategy, making use of two constrains, i.e. point clouds from depth images and correspondence between foreground user mask contour and the boundary of projected body model, is designed. On-line tracking is made possible in successive steps. At each frame, the joint angles of template skeleton are optimized towards the captured Kinect skeleton. Then, the aforementioned contour correspondence is adopted to adjust the projected body model vertices towards the contour points of foreground user mask, using a Laplacian deformation technique. Experimental results show that our method achieves fast and high quality tracking. We also show that the proposed method is benefit to three applications: virtual try-on, full human body scanning and applications in manufacturing systems.     

基于Kinect的人体三维重建方法

通过Kinect体感仪,实现人体三维重建.使用Kinect体感仪,扫描获取人体三维数据,利用深度数据转换算法实现二维顶点的三维化,再通过红外相机姿态跟踪算法进行顶点集配准,求解出相机每次的相对位移与转动角度,实现相机姿态跟踪,并将每次拍摄到的点集转换到同一全局坐标系下,使用晶格化显示集成算法将点云集成到提前划分好精度及尺寸的体素晶格中,最后利用投影映射算法获得可视化的人体三维立体模型.使用Kinect体感仪及三脚架等辅助设备方便快捷地获取人体三维重建结果,并通过3D打印技术对模型进行输出.该研究实现了人体三维重建中人体扫描、处理、重建、输出全流程.     

3D Reconstruction by Kinect Sensor:A Brief Review

While Kinect was originally designed as a motion sensing input device of the gaming console Microsoft Xbox 360 for gaming purposes, it＇s easy-to-use, low-cost, reliability, speed of the depth measurement and relatively high quality of depth measurement make it can be used for 3D reconstruction. It could make 3D scanning technology more accessible to everyday users and turn 3D reconstruction models into much widely used asset for many applications. In this paper, we focus on Kinect 3D reconstruction.     

基于Kinect v2的实时精确三维重建系统

快速、低成本、精确的三维扫描技术一直是计算机视觉领域研究的热点.首先,本文分析了新一代Kinect v2（Kinect for windows v2 sensor）的技术参数、测量原理.设计实验测得其深度精度与测量距离成线性变换关系.其次,Kinect v2深度数据含有大量的噪声尤其是在物体边缘,常用的双边滤波器等去噪算法不能很好的去除这些噪声,对此本文设计了一种有效的去噪算法,提高重建质量.最后,实现了一套基于新一代Kinect v2重建系统.实验结果表明,本文中的重建系统能够实时精确的重建物体,可以广泛应用于低成本的快速三维成型.     

基于流形学习的人体动作识别

目的 提出了一个基于流形学习的动作识别框架,用来识别深度图像序列中的人体行为。方法 从Kinect设备获得的深度信息中评估出人体的关节点信息,并用相对关节点位置差作为人体特征表达。在训练阶段,利用LE（Lalpacian eigenmaps）流形学习对高维空间下的训练集进行降维,得到低维隐空间下的运动模型。在识别阶段,用最近邻差值方法将测试序列映射到低维流形空间中去,然后进行匹配计算。在匹配过程中,通过使用改进的Hausdorff距离对低维空间下测试序列和训练运动集的吻合度和相似度进行度量。结果 用Kinect设备捕获的数据进行了实验,取得了良好的效果;同时也在MSR Action3D数据库上进行了测试,结果表明在训练样本较多情况下,本文方法识别效果优于以往方法。结论 实验结果表明本文方法适用于基于深度图像序列的人体动作识别。     

基于深度数据的人体动作识别方法

本文提出了一个基于流形学习的动作识别框架,用来识别深度图像序列中的人体行为。本文从Kinect设备获得的深度信息中评估出人体的关节点信息,并用相对关节点位置差作为人体特征表达。在训练阶段,本文利用Lapacian eigenmaps(LE)流形学习对高维空间下的训练集进行降维,得到低维隐空间下的运动模型。在识别阶段,本文用最近邻差值方法将测试序列映射到低维流形空间中去,然后进行匹配计算。在匹配过程中,通过使用改进的Hausdorff距离对低维空间下测试序列和训练运动集的吻合度和相似度进行度量。本文用Kinect设备捕获的数据进行了实验,取得了良好的效果;同时本文也在MSR Action3D数据库上进行了测试,结果表明在训练样本较多情况下,本文识别效果优于以往方法。实验结果表明本文所提的方法适用于基于深度图像序列的人体动作识别。     

基于深度图像的三维场景重建系统

针对计算机图形学和视觉领域研究热点--三维场景重建,首先分析了 Kinect v2 (Kinect for Windows v2 sensor)获取深度图像的原理,说明深度图像噪声的来源。然后根据获取 深度图像的原理设计一种算法对点云采样范围进行裁剪。其次对点云离群点进行去除,填补点 云孔洞,以提高重建质量。常见的三维场景重建大都采用了 KinectFusion 的一个全局立方体方 案,但只能对小范围内的场景进行重建。对此设计了一种对大场景进行点云匹配的 ICP 算法。 最后对点云进行曲面重建,实现一套低成本、精确的针对大场景的三维重建系统。     

Controllable data sampling in the space of human poses

Markerless human pose recognition using a single‐depth camera plays an important role in interactive graphics applications and user interface design. Recent pose recognition algorithms have adopted machine learning techniques, utilizing a large collection of motion capture data. The effectiveness of the algorithms is greatly influenced by the diversity and variability of training data. We present a new sampling method that resamples a collection of human motion data to improve the pose variability and achieve an arbitrary size and level of density in the space of human poses. The space of human poses is high dimensional, and thus, brute‐force uniform sampling is intractable. We exploit dimensionality reduction and locally stratified sampling to generate either uniform or application specifically biased distributions in the space of human poses. Our algorithm learns to recognize such challenging poses as sitting, kneeling, stretching, and doing yoga using a remarkably small amount of training data. The recognition algorithm can also be steered to maximize its performance for a specific domain of human poses. We demonstrate that our algorithm performs much better than the Kinect software development kit for recognizing challenging acrobatic poses while performing comparably for easy upright standing poses. Copyright © 2015 John Wiley & Sons, Ltd.     

Kinect传感器的彩色和深度相机标定

目的 针对现有的Kinect传感器中彩色相机和深度相机标定尤其是深度相机标定精度差、效率低的现状,本文在现有的基于彩色图像和视差图像标定算法的基础上,提出一种快速、精确的改进算法。方法 用张正友标定法标定彩色相机,用泰勒公式化简深度相机中用于修正视差值的空间偏移量以简化由视差与深度的几何关系构建的视差畸变模型,并以该模型完成Kinect传感器的标定。结果 通过拍摄固定于标定平板上的标定棋盘在不同姿态下的彩色图像和视差图像,完成Kinect传感器的标定,获得彩色相机和深度相机的畸变参数及两相机之间的旋转和平移矩阵,标定时间为116 s,得到彩色相机的重投影误差为0.33,深度相机的重投影误差为0.798。结论 实验结果表明,该改进方法在保证标定精度的前提下,优化了求解过程,有效提高了标定效率。     

基于Kinect的人体姿态估计优化和动画生成

为了生成更准确流畅的虚拟人动画,采用Kinect设备捕获三维人体姿态数据的同时,使用单目人体三维姿态估计算法对Kinect的彩色信息进行骨骼点数据推理,从而实时优化人体姿态估计效果,并驱动虚拟人物模型生成动画。首先,提出了一种时空优化的骨骼点数据处理方法,以提高单目估计人体三维姿态的稳定性;其次,提出了一种Kinect和遮挡鲁棒姿势图（ORPM）算法融合的人体姿态估计方法来解决Kinect的遮挡问题;最后,研制了基于四元数向量插值和逆向运动学约束的虚拟人动画系统,其能够进行运动仿真和实时动画生成。与仅利用Kinect捕获人体运动来生成动画的方法相比,所提方法的人体姿态估计数据鲁棒性更强,具备一定的防遮挡能力,而与基于ORPM算法的动画生成方法相比,所提方法生成的动画在帧率上提高了两倍,效果更真实流畅。     

基于Kinect的旋转刚体三维重建方法

针对在非匀速非定轴旋转条件下利用Kinect进行刚体三维重建问题,提出一种改进的基于Kinect传感器的旋转刚体三维重建方法。首先利用Kinect采集深度图像,然后用改进的加权ICP(Iterative Closest Point)算法在非匀速非定轴旋转条件下进行配准,再将各点云变换到同一坐标系下,最后根据所得点云生成三维模型表面,通过GPU(Graphic Processing Unit)编程技术来提高计算速度以满足实际需求。实验结果表明:该方法具有重建效果良好的特点。     

基于RGB-D序列的人体动态建模方法

人体建模是计算机视觉研究领域的重要研究课题。人体建模被广泛应用于科研、动画、游戏、服装设计、工业等领域,具有非常广阔的应用前景。传统的建模方法可以在大体上还原人体的姿态,但细节上会有偏差。本文提出一种基于RGB—D序列的人体动态建模方法。人体在场景中自然活动,利用廉价的深度摄像设备Kinect可以获取人体的骨架信息和三维点云。利用获得的骨架信息将模板人体分段刚性地变形到目标位置,使用ICP算法将变形后的模型与Kinect获取的点云进行更精确的配准,使用TPS变形获得一个平滑的柔性形变人体。     

3D Reconstruction of Indoor Scenes via Image Registration

With the development of computer vision technologies, 3D reconstruction has become a hotspot. At present, 3D reconstruction relies heavily on expensive equipment and has poor real-time performance. In this paper, we aim at solving the problem of 3D reconstruction of an indoor scene with large vertical span. In this paper, we propose a novel approach for 3D reconstruction of indoor scenes with only a Kinect. Firstly, this method uses a Kinect sensor to get color images and depth images of an indoor scene. Secondly, the combination of scale-invariant feature transform and random sample consensus algorithm is used to determine the transformation matrix of adjacent frames, which can be seen as the initial value of iterative closest point (ICP). Thirdly, we establish the relative coordinate relation between pair-wise frames which are the initial point cloud data by using ICP. Finally, we achieve the 3D visual reconstruction model of indoor scene by the top-down image registration of point cloud data. This approach not only mitigates the sensor perspective restriction and achieves the indoor scene reconstruction of large vertical span, but also develops the fast algorithm of indoor scene reconstruction with large amount of cloud data. The experimental results show that the proposed algorithm has better accuracy, better reconstruction effect, and less running time for point cloud registration. In addition, the proposed method has great potential applied to 3D simultaneous location and mapping.     

Scanning 3D full human bodies using Kinects

Depth camera such as Microsoft Kinect, is much cheaper than conventional 3D scanning devices, and thus it can be acquired for everyday users easily. However, the depth data captured by Kinect over a certain distance is of extreme low quality. In this paper, we present a novel scanning system for capturing 3D full human body models by using multiple Kinects. To avoid the interference phenomena, we use two Kinects to capture the upper part and lower part of a human body respectively without overlapping region. A third Kinect is used to capture the middle part of the human body from the opposite direction. We propose a practical approach for registering the various body parts of different views under non-rigid deformation. First, a rough mesh template is constructed and used to deform successive frames pairwisely. Second, global alignment is performed to distribute errors in the deformation space, which can solve the loop closure problem efficiently. Misalignment caused by complex occlusion can also be handled reasonably by our global alignment algorithm. The experimental results have shown the efficiency and applicability of our system. Our system obtains impressive results in a few minutes with low price devices, thus is practically useful for generating personalized avatars for everyday users. Our system has been used for 3D human animation and virtual try on, and can further facilitate a range of home–oriented virtual reality (VR) applications.     

Guided depth enhancement via a fast marching method

Range imaging sensors such as Kinect and time-of-flight cameras can produce aligned depth and color images in real time. However, the depth maps captured by such sensors contain numerous invalid regions and suffer from heavy noise. These defects more or less influence the use of depth information in practical applications. In order to enhance the depth maps, this paper proposes a new inpainting approach based on the fast marching method (FMM). We extend the inpainting model and the propagation strategy of FMM to incorporate color information for depth inpainting. An edge-preserving guided filter is further applied for noise reduction. To validate our algorithm, we perform experiments on both Kinect data and Middlebury dataset which, respectively, provide qualitative and quantitative results. Meanwhile, we also compare it to the original FMM and other two state-of-the-art depth enhancement methods. Experimental results show that our method performs better than the local methods in terms of both visual and metric qualities, and it achieves visually comparable results to the time-consuming global method.     

Reconstruction of 3D human body pose from stereo image sequences based on top-down learning

This paper presents a novel method for reconstructing a 3D human body pose from stereo image sequences based on a top-down learning method. However, it is inefficient to build a statistical model using all training data. Therefore, the training data is hierarchically divided into several clusters to reduce the complexity of the learning problem. In the learning stage, the human body model database is hierarchically constructed by classifying the training data into several sub-clusters with silhouette images. The data of each cluster in the bottom level is represented by a linear combination of examples. In the reconstruction stage, the proposed method hierarchically searches a cluster for the best matching silhouette image using a silhouette history image (SHI). Then, the 3D human body pose is reconstructed from a depth image using a linear combination of examples method. By using depth information to reconstruct 3D human body pose, the similar poses in silhouette images are estimated as different 3D human body poses. The experimental results demonstrate that the proposed method is efficient and effective for reconstructing 3D human body poses.     

Kinect 下深度信息获取技术及其在三维目标识别中的应用综述

微软公司 2010 年推出的 Kinect 深度传感器能够同步提供场景深度和彩色信息,其应用的一个关键领域就是目标 识别。传统的目标识别大多限制在特殊的情形,如：手势识别、人脸识别,而大规模的目标识别是近年来的研究趋势。 通过 Kinect 得到的 RGB-D 数据集多为室内和办公环境下获取的多场景、多视角、分目标类型的数据集,为大规模的目标 识别算法设计提供了学习基础。同时,Kinect 获取的深度信息为目标识别提供了强有力的线索,利用深度信息的识别方法 较以前的方法具有无法比拟的优势,大大地提高了识别的精度。文章首先对 Kinect 的深度获取技术做了详细介绍;其次 对现有的 3D 目标识别方法进行综述,接着对已有的 3D 测试数据集进行分析和比较;最后对文章进行小结以及对未来 3D 目标识别算法和 3D 测试数据集的发展趋势作了简单的阐述。     

基于Kinect的机器人辅助超声扫描系统研究

提出一种采用Kinect传感器作为视觉伺服的机器人辅助超声扫描系统,来规划引导机器人的扫描路线,以实现机器人辅助的超声扫描操作。系统由Kinect传感器、机器人和超声探头组成。采用Kinect实时获取超声探头的RGB图像和深度图像,并计算探头当前位姿,结合坐标系配准结果,得到机器人的位姿信息,再根据术前的机器人轨迹规划,引导机器人的超声扫描路径。开展腿部模型实验验证本系统的可行性,通过对Kinect传感器的相机标定实验,计算得到了RGB相机和深度相机的内外参数,通过对探头上标识物的定位,进而计算出探头当前位姿,结合Kinect与机器人坐标系的配准结果,得到了两者的转换矩阵,并对机器人的位置给出指令,引导机械臂夹持探头到达指定扫描位置。在机器人夹持超声探头扫描过程中,实时计算探头与腿部之间的距离,以保证所采集超声图像的质量及扫描操作的安全性。实验结果表明,在Kinect视觉系统的导航引导下,机器人可以夹持超声探头实现自主超声扫描,以减少超声医师的扫描时间,降低医师的劳动强度。     

基于Kinect的深度数据融合方法

与传统三维激光扫描仪相比, Kinect作为一种新型深度相机, 具有价格低廉、深度数据获取能力强、RGB影像与深度影像同步获取等优势, 然而面对较大室内场景精细建模却存在数据量大、建模范围有限、对硬件环境依赖性强等问题。因此, 在现有单一模型建模基础上, 提出了基于Kinect深度影像的多模型数据融合方法, 实现模型间的自动拼接。最后通过两组实验对提出的数据融合方法进行了验证, 并取得了较好的模型拼接效果。     

Perceptual depth compression for stereo applications

Conventional depth video compression uses video codecs designed for color images. Given the performance of current encoding standards, this solution seems efficient. However, such an approach suffers from many issues stemming from discrepancies between depth and light perception. To exploit the inherent limitations of human depth perception, we propose a novel depth compression method that employs a disparity perception model. In contrast to previous methods, we account for disparity masking, and model a distinct relation between depth perception and contrast in luminance. Our solution is a natural extension to the H.264 codec and can easily be integrated into existing decoders. It significantly improves both the compression efficiency without sacrificing visual quality of depth of rendered content, and the output of depth‐reconstruction algorithms or depth cameras.