High Dynamic Range Point Clouds for Real‐Time Relighting

Acquired 3D point clouds make possible quick modeling of virtual scenes from the real world. With modern 3D capture pipelines, each point sample often comes with additional attributes such as normal vector and color response. Although rendering and processing such data has been extensively studied, little attention has been devoted using the light transport hidden in the recorded per‐sample color response to relight virtual objects in visual effects (VFX) look‐dev or augmented reality (AR) scenarios. Typically, standard relighting environment exploits global environment maps together with a collection of local light probes to reflect the light mood of the real scene on the virtual object. We propose instead a unified spatial approximation of the radiance and visibility relationships present in the scene, in the form of a colored point cloud. To do so, our method relies on two core components: High Dynamic Range (HDR) expansion and real‐time Point‐Based Global Illumination (PBGI). First, since an acquired color point cloud typically comes in Low Dynamic Range (LDR) format, we boost it using a single HDR photo exemplar of the captured scene that can cover part of it. We perform this expansion efficiently by first expanding the dynamic range of a set of renderings of the point cloud and then projecting these renderings on the original cloud. At this stage, we propagate the expansion to the regions not covered by the renderings or with low‐quality dynamic range by solving a Poisson system. Then, at rendering time, we use the resulting HDR point cloud to relight virtual objects, providing a diffuse model of the indirect illumination propagated by the environment. To do so, we design a PBGI algorithm that exploits the GPU's geometry shader stage as well as a new mipmapping operator, tailored for G‐buffers, to achieve real‐time performances. As a result, our method can effectively relight virtual objects exhibiting diffuse and glossy physically‐based materials in real time. Furthermore, it accounts for the spatial embedding of the object within the 3D environment. We evaluate our approach on manufactured scenes to assess the error introduced at every step from the perfect ground truth. We also report experiments with real captured data, covering a range of capture technologies, from active scanning to multiview stereo reconstruction.     

3D-CDRNet: Retrieval-based dense point cloud reconstruction from a single image under complex background

3D reconstruction technique based on deep learning is gaining increasing attention from researchers. The majority of current 3D reconstruction techniques require a simple background, which limit their applications on complex background image. Extracting point cloud features comprehensively is also extremely difficult. This paper design a novel 3D reconstruction network to overcome these limitations. Firstly, we get the image and the retrieved point cloud that is the most similar to the input image. Secondly, to learn the features of the retrieved point cloud, the network encodes and decodes the single image and the retrieved point cloud to generate sparse point cloud. Finally, the proposed dense module densifies the sparse point cloud into the dense point cloud. We use single image of complex background and public dataset to evaluate our network. The reconstruction results indicate that the network surpasses previous reconstruction networks.     

基于先验知识的单视图三维点云重建算法研究

单幅图像的三维重建是一个不适定问题,由于图像与三维模型间存在的表示模式差异,通常存在物体自遮挡、低光照、多类对象等情况,针对目前单幅图像三维模型重建中重建模型具有歧义性的问题,提出了一种基于先验信息指导的多几何角度约束的三维点云模型重建方法。首先,通过预训练三维点云自编码器获得先验知识,并最小化输入图像特征向量与点云特征向量的差异,使得输入图像特征分布逼近点云特征分布;然后,利用可微投影模块将图像的三维点云表示形式从不同视角投影到二维平面;最后,通过最小化投影图与数据集中真实投影图的差异,优化初始重建点云。在ShapeNet和Pix3D数据集上与其他方法的定量定性比较结果表明了该方法的有效性。     

PointProNets: Consolidation of Point Clouds with Convolutional Neural Networks

With the widespread use of 3D acquisition devices, there is an increasing need of consolidating captured noisy and sparse point cloud data for accurate representation of the underlying structures. There are numerous algorithms that rely on a variety of assumptions such as local smoothness to tackle this ill‐posed problem. However, such priors lead to loss of important features and geometric detail. Instead, we propose a novel data‐driven approach for point cloud consolidation via a convolutional neural network based technique. Our method takes a sparse and noisy point cloud as input, and produces a dense point cloud accurately representing the underlying surface by resolving ambiguities in geometry. The resulting point set can then be used to reconstruct accurate manifold surfaces and estimate surface properties. To achieve this, we propose a generative neural network architecture that can input and output point clouds, unlocking a powerful set of tools from the deep learning literature. We use this architecture to apply convolutional neural networks to local patches of geometry for high quality and efficient point cloud consolidation. This results in significantly more accurate surfaces, as we illustrate with a diversity of examples and comparisons to the state‐of‐the‐art.     

Single‐image Tomography: 3D Volumes from 2D Cranial X‐Rays

As many different 3D volumes could produce the same 2D x‐ray image, inverting this process is challenging. We show that recent deep learning‐based convolutional neural networks can solve this task. As the main challenge in learning is the sheer amount of data created when extending the 2D image into a 3D volume, we suggest firstly to learn a coarse, fixed‐resolution volume which is then fused in a second step with the input x‐ray into a high‐resolution volume. To train and validate our approach we introduce a new dataset that comprises of close to half a million computer‐simulated 2D x‐ray images of 3D volumes scanned from 175 mammalian species. Future applications of our approach include stereoscopic rendering of legacy x‐ray images, re‐rendering of x‐rays including changes of illumination, view pose or geometry. Our evaluation includes comparison to previous tomography work, previous learning methods using our data, a user study and application to a set of real x‐rays.     

Crop recognition under weedy conditions based on 3D imaging for robotic weed control

A 3D time‐of‐flight camera was applied to develop a crop plant recognition system for broccoli and green bean plants under weedy conditions. The developed system overcame the previously unsolved problems caused by occluded canopy and illumination variation. An efficient noise filter was developed to remove the sparse noise points in 3D point cloud space. Both 2D and 3D features including the gradient of amplitude and depth image, surface curvature, amplitude percentile index, normal direction, and neighbor point count in 3D space were extracted and found effective for recognizing these two types of plants. Separate segmentation algorithms were developed for each of the broccoli and green bean plant in accordance with their 3D geometry and 2D amplitude characteristics. Under the experimental condition where the crops were heavily infested by various types of weed plants, detection rates over 88.3% and 91.2% were achieved for broccoli and green bean plant leaves, respectively. Additionally, the crop plants were segmented out with nearly complete shape. Moreover, the algorithms were computationally optimized, resulting in an image processing speed of over 30 frames per second.     

R-VPCG: RGB image feature fusion-based virtual point cloud generation for 3D car detection

Although 3D object detection methods based on feature fusion have made great progress, the methods still have the problem of low precision due to sparse point clouds. In this paper, we propose a new feature fusion-based method, which can generate virtual point cloud and improve the precision of car detection. Considering that RGB images have rich semantic information, this method firstly segments the cars from the image, and then projected the raw point clouds onto the segmented car image to segment point clouds of the cars. Furthermore, the segmented point clouds are input to the virtual point cloud generation module. The module regresses the direction of car, then combines the foreground points to generate virtual point clouds and superimposed with the raw point cloud. Eventually, the processed point cloud is converted to voxel representation, which is then fed into 3D sparse convolutional network to extract features, and finally a region proposal network is used to detect cars in a bird’s-eye view. Experimental results on KITTI dataset show that our method is effective, and the precision have significant advantages compared to other similar feature fusion-based methods.     

融合逆密度函数与关系形状卷积神经网络的点云分析

目的 3D点云与以规则的密集网格表示的图像不同,不仅不规则且无序,而且由于输入输出大小和顺序差异,具有密度不均匀以及形状和缩放比例存在差异的特性。为此,提出一种对3D点云进行卷积的方法,将关系形状卷积神经网络（relation-shape convolution neural network,RSCNN）与逆密度函数相结合,并在卷积网络中增添反卷积层,实现了点云更精确的分类分割效果。方法 在关系形状卷积神经网络中,将卷积核视为由权重函数和逆密度函数组成的3D点局部坐标的非线性函数。对给定的点,权重函数通过多层感知器网络学习,逆密度函数通过核密度估计（kernel density estimation,KDE）学习,逆密度函数的引入对点云采样率不均匀的情况进行弥补。在点云分割任务中,引入由插值和关系形状卷积层两部分组成的反卷积层,将特征从子采样点云传播回原始分辨率。结果 在ModelNet40、ShapeNet、ScanNet数据集上进行分类、部分分割和语义场景分割实验,验证模型的分类分割性能。在分类实验中,与PointNet++相比,整体精度提升3.1%,在PointNet++将法线也作为输入的情况下,精度依然提升了1.9%;在部分分割实验中,类平均交并比（mean intersection over union,mIoU）比PointNet++在法线作为输入情况下高6.0%,实例mIoU比PointNet++高1.4%;在语义场景分割实验中,mIoU比PointNet++高13.7%。在ScanNet数据集上进行不同步长有无逆密度函数的对比实验,实验证明逆密度函数将分割精度提升0.8%左右,有效提升了模型性能。结论 融合逆密度函数的关系形状卷积神经网络可以有效获取点云数据中的局部和全局特征,并对点云采样不均匀的情况实现一定程度的补偿,实现更优的分类和分割效果。     

Reconstructing 3D Shape,Albedo and Illumination from a Single Face Image

The morphable model has been employed to efficiently describe 3D face shape and the associated albedo with a reduced set of basis vectors. The spherical harmonics (SH) model provides a compact basis to well approximate the image appearance of a Lambertian object under different illumination conditions. Recently, the SH and morphable models have been integrated for 3D face shape reconstruction. However, the reconstructed 3D shape is either inconsistent with the SH bases or obtained just from landmarks only. In this work, we propose a geometrically consistent algorithm to reconstruct the 3D face shape and the associated albedo from a single face image iteratively by combining the morphable model and the SH model. The reconstructed 3D face geometry can uniquely determine the SH bases, therefore the optimal 3D face model can be obtained by minimizing the error between the input face image and a linear combination of the associated SH bases. In this way, we are able to preserve the consistency between the 3D geometry and the SH model, thus refining the 3D shape reconstruction recursively. Furthermore, we present a novel approach to recover the illumination condition from the estimated weighting vector for the SH bases in a constrained optimization formulation independent of the 3D geometry. Experimental results show the effectiveness and accuracy of the proposed face reconstruction and illumination estimation algorithm under different face poses and multiple‐light‐source illumination conditions.     

点云场景下基于结构感知的车辆检测

在自动驾驶领域,计算机对周围环境的感知和理解是必不可少的.其中,相比于二维目标检测,三维点云目标检测可以提供二维目标检测所不具有的物体的三维方位信息,这对于安全自动驾驶是至关重要的.针对三维目标检测中原始输入点云到检测结果之间跨度大的问题,首先,提出了基于结构感知的候选区域生成模块,其中定义了每个点的结构特征,充分利用...     

一种基于改进投影方法的点云拼接算法*

针对传统ICP（iterative closest point）算法在收敛速度或拼接精度上无法同时满足实时测量要求的问题,提出了一种适用于单、双目结构光三维测量系统的快速且高精度的点云拼接算法。该算法首先将参考视点云上的采样点反向投影到目标视点云的2D成像平面上,然后再将2D反向投影点正向投影到目标视点云上,展示了一种由目标视投影点到采样点法线上的投影直至收敛于线—面交点的迭代过程。实际测量结果表明,该算法在保证拼接精度的同时显著提高了收敛速度,是一种具有很高实用价值的拼接算法。     

结构光视觉三维点云逐层三角网格化算法

针对结构光视觉恢复的大规模三维点云的可投影特点,提出一种基于投影网格的底边驱动逐层网格化曲面重建算法。该算法首先将点云投影到一个二维平面上;然后基于点云投影区域建立规则投影网格,并将投影点映射到规则二维投影网格上,建立二维网格点与三维点云间的映射关系;接着对投影网格进行底边驱动的逐层网格化,建立二维三角网格;最后根据二维投影点与三维点的对应关系及二维三角网格拓扑关系获得最终的三维网格曲面。实验结果表明,算法曲面重建速度快,可较好地保持曲面细节特征。     

基于特征融合的三维人脸点云质量判断

针对使用双目结构光扫描仪获取的三维人脸点云,提出了一种特征融合网络(FFN)来完成人脸点云质量判断任务.首先,对三维点云预处理切割出人脸面部区域,使用点云和对应的二维平面投影得到的图像作为输入;其次,分别训练用于点云学习的动态图卷积神经网络(DGCNN)和ShuffleNet两个模块;然后,提取出两个网络模块的中间层特...     

结合坐标转换和时空信息注入的点云人体行为识别

目的 行为识别中广泛使用的深度图序列存在着行为数据时空结构信息体现不足、易受深色物体等因素影响的缺点,点云数据可以提供丰富的空间信息与几何特征,弥补了深度图像的不足,但多数点云数据集规模较小且没有时序信息。为了提高时空结构信息的利用率,本文提出了结合坐标转换和时空信息注入的点云人体行为识别网络。方法 通过将深度图序列转换为三维点云序列,弥补了点云数据集规模较小的缺点,并加入帧的时序概念。本文网络由两个模块组成,即特征提取模块和时空信息注入模块。特征提取模块提取点云深层次的外观轮廓特征。时空信息注入模块为轮廓特征注入时序信息,并通过一组随机张量投影继续注入空间结构信息。最后,将不同层次的多个特征进行聚合,输入到分类器中进行分类。结果 在3个公共数据集上对本文方法进行了验证,提出的网络结构展现出了良好的性能。其中,在NTU RGB+d60数据集上的精度分别比PSTNet（point spatio-temporal network）和SequentialPointNet提升了1.3%和0.2%,在NTU RGB+d120数据集上的精度比PSTNet提升了1.9%。为了确保网络模型的鲁棒性,在MSR Action3D小数据集上进行实验对比,识别精度比SequentialPointNet提升了1.07%。结论 提出的网络在获取静态的点云外观轮廓特征的同时,融入了动态的时空信息,弥补了特征提取时下采样导致的时空损失。     

基于二维点云图的三维人体建模方法

近年来基于二维图像的三维建模方法取得了快速发展,但就人体建模而言,由于摄像头采集到的二维人体图像包含衣物、发丝等大量的纹理信息,而像虚拟试衣等相关应用需要将人体表面的衣物褶皱等纹理信息去除,同时考虑到裸体数据采集侵犯了用户的隐私,因此提出一种基于二维点云图像到三维人体模型的新型建模方法。与摄像机等辅助设备进行二维图片数据集的采集不同,该算法的输入是由三维人体点云模型以顶点模式绘制的二维点云渲染图。主要工作是建立一个由二维点云图和相应的人体黑白二值图构成的数据集,并训练一个由前者生成后者的生成对抗网络模型。该模型将二维点云图转化为相应的黑白二值图。将该二值图输入一个训练好的卷积神经网络,用于评估二维图像到三维人体模型构建的效果。考虑到由不完整三维点云数据重建完整的三维人体网格模型是一个具有挑战性的问题,因此通过模拟二维点云的破损和残缺状态,使得算法能够处理不完整的二维点云图。大量的实验结果表明,该方法重建出的三维人体模型能够有效实现视觉上的真实感,为了对重建后的精度进行定量的分析,选取了人体特征中具有代表性的腰围特征作为误差评估;为了增加三维人体模型库中人体形态的多样性,还引入一种便捷的三维人体模型数据增强技术。实验结果表明,该算法只需要输入一张二维点云图像,就能快速创建出相应的数字化人体模型。     

多特征融合与残差优化的点云语义分割方法

目的 当前的大场景3维点云语义分割方法一般是将大规模点云切成点云块再进行处理。然而在实际计算过程中,切割边界的几何特征容易被破坏,使得分割结果呈现明显的边界现象。因此,迫切需要以原始点云作为输入的高效深度学习网络模型,用于点云的语义分割。方法 为了解决该问题,提出基于多特征融合与残差优化的点云语义分割方法。网络通过一个多特征提取模块来提取每个点的几何结构特征以及语义特征,通过对特征的加权获取特征集合。在此基础上,引入注意力机制优化特征集合,构建特征聚合模块,聚合点云中最具辨别力的特征。最后在特征聚合模块中添加残差块,优化网络训练。最终网络的输出是每个点在数据集中各个类别的置信度。结果 本文提出的残差网络模型在S3DIS （Stanford Large-scale 3D Indoor Spaces Dataset）与户外场景点云分割数据集Semantic3D等2个数据集上与当前的主流算法进行了分割精度的对比。在S3DIS数据集中,本文算法在全局准确率以及平均准确率上均取得了较高精度,分别为87.2%,81.7%。在Semantic3D数据集上,本文算法在全局准确率和平均交并比上均取得了较高精度,分别为93.5%,74.0%,比GACNet （graph attention convolution network）分别高1.6%,3.2%。结论 实验结果验证了本文提出的残差优化网络在大规模点云语义分割的应用中,可以缓解深层次特征提取过程中梯度消失和网络过拟合现象并保持良好的分割性能。     

Weakly Supervised Part-wise 3D Shape Reconstruction from Single-View RGB Images

In order for the deep learning models to truly understand the 2D images for 3D geometry recovery, we argue that single-view reconstruction should be learned in a part-aware and weakly supervised manner. Such models lead to more profound interpretation of 2D images in which part-based parsing and assembling are involved. To this end, we learn a deep neural network which takes a single-view RGB image as input, and outputs a 3D shape in parts represented by 3D point clouds with an array of 3D part generators. In particular, we devise two levels of generative adversarial network (GAN) to generate shapes with both correct part shape and reasonable overall structure. To enable a self-taught network training, we devise a differentiable projection module along with a self-projection loss measuring the error between the shape projection and the input image. The training data in our method is unpaired between the 2D images and the 3D shapes with part decomposition. Through qualitative and quantitative evaluations on public datasets, we show that our method achieves good performance in part-wise single-view reconstruction.     

Multiperson interaction recognition in images: A body keypoint based feature image analysis

Most interaction recognition approaches have been limited to single‐person action classification in videos. However, for still images where motion information is not available, the task becomes more complex. Aiming to this point, we propose an approach for multiperson human interaction recognition in images with keypoint‐based feature image analysis. Proposed method is a three‐stage framework. In the first stage, we propose feature‐based neural network (FCNN) for action recognition trained with feature images. Feature images are body features, that is, effective distances between a set of body part pairs and angular relation between body part triplets, rearranged in 2D gray‐scale image to learn effective representation of complex actions. In the later stage, we propose a voting‐based method for direction encoding to anticipate probable motion in steady images. Finally, our multiperson interaction recognition algorithm identifies which human pairs are interacting with each other using an interaction parameter. We evaluate our approach on two real‐world data sets, that is, UT‐interaction and SBU kinect interaction. The empirical experiments show that results are better than the state‐of‐the‐art methods with recognition accuracy of 95.83% on UT‐I set 1, 92.5% on UT‐I set 2, and 94.28% on SBU clean data set.     

Multi-view aggregation transformer for no-reference point cloud quality assessment

With the increasing maturity of 3D point cloud acquisition, storage, and transmission technologies, a large number of distorted point clouds without original reference exist in practical applications. Hence, it is necessary to design a no-reference point cloud quality assessment (PCQA) for point cloud systems. However, the existing no-reference PCQA metrics ignore the content differences and positional context among the projected images. For this, we propose a Multi-View Aggregation Transformer (MVAT) with two different fusion modules to extract the comprehensive feature representation of PCQA. Specifically, considering the content differences of different projected images, we first design a Content Fusion Module (CFM) to fuse multiple projected image features by adaptive weighting. Then, we design a Bidirectional Context Fusion Module (BCFM) to extract context features for reflecting the contextual relationship among projected images. Finally, we joint the above two fusion modules via Content-Position Fusion Module (CPFM) to fully mine the feature representation of point clouds. Experimental results show that our MVAT can achieve comparable or better performance than state-of-the-art metrics on three open point cloud datasets.