Kinect与Unity3D数据整合技术在体感游戏中的应用研究

通过分析Kinect与Unity3D数据整合关键技术,从WPF与Unity 3D内部调用方式展开系统设计。设计分为Unity3D场景展示模块、Unity3D的接口模块和Kinect的数据获取三模块。其中Unity3D接口模块实现了的场景设置,骨骼绑定、镜像运动、近景模式、平滑处理功能;Kinect数据获取模块通过代码实现设备控制、骨骼绑定算法、设备图像获取。测试证明,通过C#对非托管的dll的管理方式,导入Kinect硬件的驱动程序,调用自定义的数据结构和算法,实现在unity 3D场景中,使用Kinect体感镜头控制场景中的人物模型运动,提高了体感游戏的开发效率,在体感游戏的开发和应用中有一定的社会推广价值。     

基于深度图像的多学习者姿态识别

在数字化学习场景中,人体姿态的识别有助于分析学习者的学习状态。提出了一种基于深度图像的多学习者姿态识别方法。首先通过Kinect的红外传感器获取包含深度信息的图像,利用深度图像进行人像-背景分离;然后提取人体的轮廓特征Hu矩;最后采用SVM分类器对轮廓特征进行分类和识别。实验结果表明,本方法能有效地识别多个学习者的举手、正坐和低头等姿态。     

弱监督与少样本学习场景下视频行为识别综述

近年来各类人体行为识别算法利用大量标记数据进行训练,取得了良好的识别精度。但在实际应用中,数据的获取以及标注过程都是非常耗时耗力的,这限制了算法的实际落地。针对弱监督及少样本场景下的视频行为识别深度学习方法进行综述。首先,在弱监督情况下,分类总结了半监督行为识别方法和无监督领域自适应下的视频行为识别方法;然后,对少样本场景下的视频行为识别算法进行详细综述;接着,总结了当前相关的人体行为识别数据集,并在该数据集上对各相关视频行为识别算法性能进行分析比较;最后,进行概括总结,并展望人体行为识别的未来发展方向。     

基于语义分割的室外场景识别技术研究

针对于场景识别问题,提出一种基于开源的室外场景数据集以及自定义采集的数据集在deepLabV3+深度学习模型上进行实验,并运用一种改进的K-近邻算法对DeepLabV3+深度学习模型进行优化.与现有的测试数据集的方法不同,省去了对数据集进行标签的工作,减少了大量的前期准备工作,提高了模型的计算效率以及分类模型的准确率和...     

基于Kinect的民族弓弦乐器虚拟演奏系统

利用Kinect相机结合增强现实技术和手势识别方法设计并实现了一个弓弦乐器虚拟演奏系统——以二胡为例.将Kinect获取的现实场景和虚拟乐器融合在一起绘制成增强现实场景.通过Kinect得到的深度数据和贝叶斯肤色模型将用户的左手分割出来,并再次绘制在增强图像上形成新的图像,从而解决虚拟演奏场景中的虚实遮挡问题.利用基于反向动力学和马尔可夫模型的三维虚拟手势拟合方法,对演奏过程中的左手手势进行识别,并结合右手的运动状态完成乐器的虚拟演奏.     

基于Kinect骨骼信息与深度图像的指尖点检测

针对普通摄像头手势识别系统易受复杂环境和光照条件等因素影响,存在对指尖点的漏判、误判问题,提出一种基于Kinect 骨骼信息与深度图像的掌心点提取和指尖点检测的手势识别方法。在DRVI平台上创建Kinect的接口控件,对Kinect传感器获取人体骨骼信息和深度图像进行分析,采用了坐标映射、图像分割、距离变换的关键技术和方法从深度图中分割出手势部分区域,对手势区域形态学处理,结合凸包和K-曲率算法检测不同手势中指尖点的个数和位置,计算不同手势凸包轮廓上的点集生成的HOG(Histogram of Oriented Gradient)特征描述子,最后利用特征描述子对预定的6种数字手势进行识别。经实验测试可以在复杂环境和不同光照情况下正确识别指尖点。     

基于场景先验知识的室内人体行为识别方法

目前,室内人体行为识别技术被广泛应用于视频内容理解、居家养老、医疗护理等领域,现有研究方法更多的是对人体行为进行建模,忽略了视频中场景与人体行为间的联系。为了充分利用场景信息与室内人体运动的关联性,文中对基于场景先验知识的室内人体行为识别方法进行了研究,提出了一种基于场景先验知识的双流膨胀3D行为识别网络(Scene-Prior Know-ledge Inflated 3D ConvNet, SPI3D)。首先使用ResNet152网络提取场景特征进行场景分类,再基于场景分类的结果,引入量化后的场景先验知识,通过对权值进行约束来优化总体目标函数。另外,针对现有数据集多聚焦于人体行为特征、场景复杂且场景特征不明显的问题,自建了室内场景-行为识别数据集(Scene-Action DataBase, SADB)。实验结果表明,在SADB数据集上,SPI3D网络的识别准确率为87.9%,比直接利用I3D网络的识别准确率高6%。由此可见,引入场景先验知识后的室内人体行为识别模型具有更好的表现。     

基于多列深度3D卷积神经网络的手势识别

传统2D卷积神经网络对于视频连续帧图像的特征提取容易丢失目标时间轴上的运动信息,导致识别准确度较低。为此,提出一种基于多列深度3D卷积神经网络(3D CNN)的手势识别方法。采用3D卷积核对连续帧图像进行卷积操作,提取目标的时间和空间特征捕捉运动信息。为避免因单组3D CNN特征提取不充分而导致的误分类,训练多组具有较强分类能力的3D CNN结构组成多列深度3D CNN,该结构通过对多组3D CNN的输出结果进行权衡,将权重最大的类别判定为最终的输出结果。实验结果表明,将多列深度3D CNN应用于CHGDs数据集上进行手势识别,识别率达到95.09%,与单组3D CNN及传统2D CNN相比分别提高近7%,20%,对连续图像目标识别具有较好的识别能力。     

基于3D卷积神经网络的动态手势识别

传统的2D卷积神经网络在进行视频识别时容易丢失目标在时间维度上的相关特征信息,导致识别准确率降低。针对该问题,本文采用3D卷积网络作为基本的网络框架,使用3D卷积核进行卷积操作提取视频中的时空特征,同时集成多个3D卷积神经网络模型对动态手势进行识别。为了提高模型的收敛速度和训练的稳定性,运用批量归一化（BN）技术优化网络,使优化后的网络训练时间缩短。实验结果表明,本文方法对于动态手势的识别具有较好的识别结果,在Sheffield Kinect Gesture (SKIG)数据集上识别准确率达到98.06%。与单独使用RGB信息、深度信息以及传统2D CNN相比,手势识别率均有所提高,验证了本文方法的可行性和有效性。     

基于Kinect的无监督图像识别

现有的物体识别方法大都是有监督学习,而且主要处理的是2D图像数据,由于这类数据给出的背景信息和物体信息少于2.5D场景中的信息,传统有监督的学习方法对于非特定物体的识别效果还是不尽如人意,更别说基于无监督学习的识别系统了。随着Kinect设别的出现,获取场景的2.5D信息变得很容易。设计了一个全新的基于无监督物体识别系统。首先用平面拟合和空间聚合把物体从场景中分离以后。然后用几何基元拟合分割技术把物体分割为不同部件,在此基础上构建物体的图模型,最后把图模型嵌入为向量空间上的点集合,使用扩展陆地移动距离算法计算物体间的相似度。从实验结果来看效果不错。     

Face recognition based on Kinect

In this paper, we present a new algorithm that utilizes low-quality red, green, blue and depth (RGB-D) data from the Kinect sensor for face recognition under challenging conditions. This algorithm extracts multiple features and fuses them at the feature level. A Finer Feature Fusion technique is developed that removes redundant information and retains only the meaningful features for possible maximum class separability. We also introduce a new 3D face database acquired with the Kinect sensor which has released to the research community. This database contains over 5,000 facial images (RGB-D) of 52 individuals under varying pose, expression, illumination and occlusions. Under the first three variations and using only the noisy depth data, the proposed algorithm can achieve 72.5 % recognition rate which is significantly higher than the 41.9 % achieved by the baseline LDA method. Combined with the texture information, 91.3 % recognition rate has achieved under illumination, pose and expression variations. These results suggest the feasibility of low-cost 3D sensors for real-time face recognition.     

Recent advances in 3D object detection based on RGB-D: A survey

3D object detection is a critical part of environmental perception systems and one of the most fundamental tasks in understanding the 3D visual world, which benefit a series of downstream real-world applications. RGB-D images include object texture and semantic information, as well as depth information describing spatial geometry. Recently, numerous 3D object detection models for RGB-D images have been proposed with excellent performance, but summaries in this area are still absent. To stimulate future research, this paper provides a detailed analysis of current developments in 3D object detection methods for RGB-D images to motivate future research. It covers three major parts, including background on 3D object detection, RGB-D data details, and comparative results of state-of-the-art methods on several publicly available datasets, with an emphasis on contributions, design ideas, and limitations, as well as insightful observations and inspiring future research directions.     

基于注意力感知和语义感知的RGB-D室内图像语义分割算法

近年来,全卷积神经网络有效提升了语义分割任务的准确率.然而,由于室内环境的复杂性,室内场景语义分割仍然是一个具有挑战性的问题.随着深度传感器的出现,人们开始考虑利用深度信息提升语义分割效果.以往的研究大多简单地使用等权值的拼接或求和操作来融合RGB特征和深度特征,未能充分利用RGB特征与深度特征之间的互补信息.本文提出...     

Learning for classification of traffic-related object on RGB-D data

In order to detect and recognize the traffic-related object, a learning-based classification approach is proposed on RGB-D data. Since RGB-D data can provide the depth information and thus make it capable of tackling the baffling issues such as overlapping, clustered background, the depth data obtained by Microsoft Kinect sensor is introduced in the proposed method for efficiently detecting and extracting the objects in the traffic scene. Moreover, we construct a feature vector, which combine the histograms of oriented gradients, 2D features and 3D Spin Image features, to represent the traffic-related objects. The feature vector is used as the input of the random forest for training a classifier and classifying the traffic-related objects. In experiments, by conducting efficiency and accuracy tests on RGB-D data captured in different traffic scenarios, the proposed method performs better than the typical support vector machine method. The results show that traffic-related objects can be efficiently detected, and the accuracy of classification can achieve higher than 98 %.     

Robust RGB-D simultaneous localization and mapping using planar point features

RGB-D cameras like PrimeSense and Microsoft Kinect are popular sensors in the simultaneous localization and mapping researches on mobile robots because they can provide both vision and depth information. Most of the state-of-the-art RGB-D SLAM systems employ the Iterative Closest Point (ICP) algorithm to align point features, whose spatial positions are computed by the corresponding depth data of the sensors. However, the depth measurements of features are often disturbed by noise because visual features tend to lie at the margins of real objects. In order to reduce the estimation error, we propose a method that extracts and selects the features with reliable depth values, i.e. planar point features. The planar features can benefit the accuracy and robustness of traditional ICP, while holding a reasonable computation cost for real-time applications. An efficient RGB-D SLAM system based on planar features is also demonstrated, with trajectory and map results from open datasets and a physical robot in real-world experiments.     

利用3D打印标定球的机械臂与RGB-D相机手眼标定方法

因为彩色镜头和深度镜头不在同一位置,并且深度图像测量精度差、分辨率低、没有颜色纹理信息,传统的手眼标定方法并不适用于RGB-D相机．本文提出一种利用简单低成本的3D打印球作为标定件对机械臂与RGB-D相机进行手眼标定的方法．本方法只需要测量标定件的3D位置信息,避免使用测量不便、精度稍差的姿态信息．文中给出了该方法的封闭解和迭代优化解．100组仿真结果表明,标定精度与RGB-D相机自身测量精度一致;封闭解不需要机械臂与相机时间同步;迭代优化解的标定精度略有提升,误差最大值和误差方差都很稳定．最后,在7自由度的KUKA ⅡWA机械臂和Kinect相机上做了手眼标定实验,结果与仿真实验一致．总之,本文方法简单可靠,可实现机械臂与RGB-D相机之间的快速部署手眼标定．     

纹理边缘引导的深度图像超分辨率重建

目的 深度图像作为一种普遍的3维场景信息表达方式在立体视觉领域有着广泛的应用。Kinect深度相机能够实时获取场景的深度图像,但由于内部硬件的限制和外界因素的干扰,获取的深度图像存在分辨率低、边缘不准确的问题,无法满足实际应用的需要。为此提出了一种基于彩色图像边缘引导的Kinect深度图像超分辨率重建算法。方法 首先对深度图像进行初始化上采样,并提取初始化深度图像的边缘;进一步利用高分辨率彩色图像和深度图像的相似性,采用基于结构化学习的边缘检测方法提取深度图的正确边缘;最后找出初始化深度图的错误边缘和深度图正确边缘之间的不可靠区域,采用边缘对齐的策略对不可靠区域进行插值填充。结果 在NYU2数据集上进行实验,与8种最新的深度图像超分辨率重建算法作比较,用重建之后的深度图像和3维重建的点云效果进行验证。实验结果表明本文算法在提高深度图像的分辨率的同时,能有效修正上采样后深度图像的边缘,使深度边缘与纹理边缘对齐,也能抑制上采样算法带来的边缘模糊现象;3维点云效果显示,本文算法能准确区分场景中的前景和背景,应用于3维重建等应用能取得较其他算法更好的效果。结论 本文算法普遍适用于Kinect深度图像的超分辨率重建问题,该算法结合同场景彩色图像与深度图像的相似性,利用纹理边缘引导深度图像的超分辨率重建,可以得到较好的重建结果。     

An automatic 2D to 3D video conversion approach based on RGB-D images

3D movies/videos have become increasingly popular in the market; however, they are usually produced by professionals. This paper presents a new technique for the automatic conversion of 2D to 3D video based on RGB-D sensors, which can be easily conducted by ordinary users. To generate a 3D image, one approach is to combine the original 2D color image and its corresponding depth map together to perform depth image-based rendering (DIBR). An RGB-D sensor is one of the inexpensive ways to capture an image and its corresponding depth map. The quality of the depth map and the DIBR algorithm are crucial to this process. Our approach is twofold. First, the depth maps captured directly by RGB-D sensors are generally of poor quality because there are many regions missing depth information, especially near the edges of objects. This paper proposes a new RGB-D sensor based depth map inpainting method that divides the regions with missing depths into interior holes and border holes. Different schemes are used to inpaint the different types of holes. Second, an improved hole filling approach for DIBR is proposed to synthesize the 3D images by using the corresponding color images and the inpainted depth maps. Extensive experiments were conducted on different evaluation datasets. The results show the effectiveness of our method.

     

基于Kinect的三维视觉里程计的设计

针对移动服务机器人在未知环境下三维路径估计的问题,设计了一种基于Kinect的实时估计机器人运动轨迹的方法。该方法采用Kinect获取机器人运动过程中连续帧的彩色和深度信息,首先,提取并匹配目标帧和参考帧的SURF的特征点;然后,结合深度信息利用经典P3P问题的方法及改进的随机采样一致性(RANSAC)算法计算机器人的初始6自由度(DOF)位姿;最后,通过非线性最小二乘算法最小化初始位姿内点的双向投影误差来提高位姿精度,进而得到机器人的运动轨迹。同时对比了不同特征点及描述符结合下的里程计精度。实验结果表明,所提方法能够将里程计误差降低到3.1%,且能够满足实时要求,可为机器人同时定位与地图创建提供重要的先验信息。     

基于RGB-D图像核描述子的物体识别方法

针对传统的颜色-深度（RGB-D）图像物体识别的方法所存在的图像特征学习不全面、特征编码鲁棒性不够等问题,提出了基于核描述子局部约束线性编码（KD-LLC）的RGB-D图像物体识别方法。首先,在图像块间匹配核函数基础上,应用核主成分分析法提取RGB-D图像的3D形状、尺寸、边缘、颜色等多个互补性核描述子;然后,分别对它们进行LLC编码及空间池化处理以形成相应的图像编码向量;最后,把这些图像编码向量融合成具有鲁棒性、区分性的图像表示。基于RGB-D数据集的仿真实验结果表明,作为一种基于人工设计特征的RGB-D图像物体识别方法,由于所提算法综合利用深度图像和RGB图像的多方面特征,而且对传统深度核描述子的采样点选取和紧凑基向量的计算这两方面进行了改进,使得物体类别识别率达到86.8%,实体识别率达到92.7%,比其他同类方法具有更高的识别准确率。