High Dynamic Range Point Clouds for Real‐Time Relighting

Acquired 3D point clouds make possible quick modeling of virtual scenes from the real world. With modern 3D capture pipelines, each point sample often comes with additional attributes such as normal vector and color response. Although rendering and processing such data has been extensively studied, little attention has been devoted using the light transport hidden in the recorded per‐sample color response to relight virtual objects in visual effects (VFX) look‐dev or augmented reality (AR) scenarios. Typically, standard relighting environment exploits global environment maps together with a collection of local light probes to reflect the light mood of the real scene on the virtual object. We propose instead a unified spatial approximation of the radiance and visibility relationships present in the scene, in the form of a colored point cloud. To do so, our method relies on two core components: High Dynamic Range (HDR) expansion and real‐time Point‐Based Global Illumination (PBGI). First, since an acquired color point cloud typically comes in Low Dynamic Range (LDR) format, we boost it using a single HDR photo exemplar of the captured scene that can cover part of it. We perform this expansion efficiently by first expanding the dynamic range of a set of renderings of the point cloud and then projecting these renderings on the original cloud. At this stage, we propagate the expansion to the regions not covered by the renderings or with low‐quality dynamic range by solving a Poisson system. Then, at rendering time, we use the resulting HDR point cloud to relight virtual objects, providing a diffuse model of the indirect illumination propagated by the environment. To do so, we design a PBGI algorithm that exploits the GPU's geometry shader stage as well as a new mipmapping operator, tailored for G‐buffers, to achieve real‐time performances. As a result, our method can effectively relight virtual objects exhibiting diffuse and glossy physically‐based materials in real time. Furthermore, it accounts for the spatial embedding of the object within the 3D environment. We evaluate our approach on manufactured scenes to assess the error introduced at every step from the perfect ground truth. We also report experiments with real captured data, covering a range of capture technologies, from active scanning to multiview stereo reconstruction.     

伪点云修正增强激光雷达数据

目的 激光雷达在自动驾驶中具有重要意义,但其价格昂贵,且产生的激光线束数量仍然较少,造成采集的点云密度较稀疏。为了更好地感知周围环境,本文提出一种激光雷达数据增强算法,由双目图像生成伪点云并对伪点云进行坐标修正,进而实现激光雷达点云的稠密化处理,提高3D目标检测精度。此算法不针对特定的3D目标检测网络结构,是一种通用的点云稠密化方法。方法 首先利用双目RGB图像生成深度图像,根据先验的相机参数和深度信息计算出每个像素点在雷达坐标系下的粗略3维坐标,即伪点云。为了更好地分割地面,本文提出了循环RANSAC （random sample consensus）算法,引入了一个分离平面型非地面点云的暂存器,改进复杂场景下的地面分割效果。然后将原始点云进行地面分割后插入KDTree （k-dimensional tree）,以伪点云中的每个点为中心在KDTree中搜索若干近邻点,基于这些近邻点进行曲面重建。根据曲面重建结果,设计一种计算几何方法导出伪点云修正后的精确坐标。最后,将修正后的伪点云与原始激光雷达点云融合得到稠密化点云。结果 实验结果表明,稠密化的点云在视觉上具有较好的质量,物体具有更加完整的形状和轮廓,并且在KITTI （Karlsruhe Institute of Technology and Toyota Technological Institute）数据集上提升了3D目标检测精度。在使用该数据增强方法后,KITTI数据集下AVOD （aggregate view object detection）检测方法的AP_3D-Easy （average precision of 3D object detection on easy setting）提升了8.25%,AVOD-FPN （aggregate view object detection with feature pyramid network）检测方法的AP_BEV-Hard （average precision of bird’s eye view on hard setting）提升了7.14%。结论 本文提出的激光雷达数据增强算法,实现了点云的稠密化处理,并使3D目标检测结果更加精确。     

混合视觉系统的运动物体检测和静态地图重建

复杂动态背景环境下的运动物体检测和静态地图重建中容易出现运动物体检测不完整的问题。针对上述问题,提出了一种混合视觉系统下点云分割辅助的运动物体检测方法。首先,提出了直通滤波+随机采样一致性（PassThrough+RANSAC）方法来克服大面积墙壁干扰以实现点云地面点的识别;其次,将非地面点数据作为特征点投射到图像上,并估计其光流运动向量和人工运动向量,从而对动态点进行检测;然后,采用动态阈值策略对点云进行欧氏聚类;最后,整合动态点检测结果与点云分割结果来完整地提取出运动物体。此外,通过八叉树地图（Octomap）工具将点云地图转换为三维栅格地图以完成地图的构建。通过实验结果和数据分析可知,所提方法可以有效提高运动物体检测的完整性,同时重建出低损耗、高实用性的静态栅格地图。     

Geographic,Geometrical and Semantic Reconstruction of Urban Scene from High Resolution Oblique Aerial Images

An effective approach is proposed for 3D urban scene reconstruction in the form of point cloud with semantic labeling. Starting from high resolution oblique aerial images, our approach proceeds through three main stages:geographic reconstruction, geometrical reconstruction and semantic reconstruction. The absolute position and orientation of all the cameras relative to the real world are recovered in the geographic reconstruction stage. Then, in the geometrical reconstruction stage, an improved multi-view stereo matching method is employed to produce 3D dense points with color and normal information by taking into account the prior knowledge of aerial imagery. Finally the point cloud is classified into three classes (building, vegetation, and ground) by a rule-based hierarchical approach in the semantic reconstruction step. Experiments on complex urban scene show that our proposed 3-stage approach could generate reasonable reconstruction result robustly and efficiently. By comparing our final semantic reconstruction result with the manually labeled ground truth, classification accuracies from 86.75% to 93.02% are obtained.      

3D Object Detection with Attention: Shell-Based Modeling

LIDAR point cloud-based 3D object detection aims to sense the surrounding environment by anchoring objects with the Bounding Box (BBox). However, under the three-dimensional space of autonomous driving scenes, the previous object detection methods, due to the pre-processing of the original LIDAR point cloud into voxels or pillars, lose the coordinate information of the original point cloud, slow detection speed, and gain inaccurate bounding box positioning. To address the issues above, this study proposes a new two-stage network structure to extract point cloud features directly by PointNet++, which effectively preserves the original point cloud coordinate information. To improve the detection accuracy, a shell-based modeling method is proposed. It roughly determines which spherical shell the coordinates belong to. Then, the results are refined to ground truth, thereby narrowing the localization range and improving the detection accuracy. To improve the recall of 3D object detection with bounding boxes, this paper designs a self-attention module for 3D object detection with a skip connection structure. Some of these features are highlighted by weighting them on the feature dimensions. After training, it makes the feature weights that are favorable for object detection get larger. Thus, the extracted features are more adapted to the object detection task. Extensive comparison experiments and ablation experiments conducted on the KITTI dataset verify the effectiveness of our proposed method in improving recall and precision.     

一种适用于室内服务机器人的实时物体识别系统

针对室内服务机器人在实际应用中的需求,提出一种结合三维点云分割和局部特征匹配的实时物体识别系统．该系统首先基于三维点云实现快速有效的物体检测,然后利用物体检测的结果定位物体在彩色图像中的区域,并采用基于SURF特征匹配的方法识别出物体的标识．实验结果表明,该系统可较好地满足室内服务机器人物体检测与识别的实时性和可靠性要求．     

Fully Automatic Registration of Image Sets on Approximate Geometry

The photorealistic acquisition of 3D objects often requires color information from digital photography to be mapped on the acquired geometry, in order to obtain a textured 3D model. This paper presents a novel fully automatic 2D/3D global registration pipeline consisting of several stages that simultaneously register the input image set on the corresponding 3D object. The first stage exploits Structure From Motion (SFM) on the image set in order to generate a sparse point cloud. During the second stage, this point cloud is aligned to the 3D object using an extension of the 4 Point Congruent Set (4PCS) algorithm for the alignment of range maps. The extension accounts for models with different scales and unknown regions of overlap. In the last processing stage a global refinement algorithm based on mutual information optimizes the color projection of the aligned photos on the 3D object, in order to obtain high quality textures. The proposed registration pipeline is general, capable of dealing with small and big objects of any shape, and robust. We present results from six real cases, evaluating the quality of the final colors mapped onto the 3D object. A comparison with a ground truth dataset is also presented.     

基于三维激光扫描的物体重构建模

物体拍摄环境具有测量数据量大、物体外轮廓信息复杂等特点,采用当前方法能够获得物体精确的三维点云数据,但缺乏颜色和纹理信息,导致物体重构精度不高,真实感较差;为此,提出一种基于三维激光扫描的物体重构建模方法;该方法通过三维激光扫描技术获取物体点云数据,采用显式的欧拉积分方法对物体整个三维曲面进行平滑,依据三角生长法进行物体三维空间三角划分,将物体网格顶点向球面进行映射,由此构造物体三角网格模型,通过迭代最近点算法对物体非同步点云数据初步匹配结果进行精确配准,利用最近点搜索算法将经多视图立体视觉算法优化后的物体颜色信息和三维点云数据坐标相融合;实验结果表明,所提方法可以快速精确地建立物体三维重构模型,验证了所提方法的可行性。     

误差云映射的深度融合3维重建

描述了一种应用于视频序列的3维稠密重建方法,主要针对基于多深度图恢复3维物体时存在的多对一映射重复投影问题,提出基于误差云概念的3维定点优化方法.该方法首先通过深度矫正,增加视频序列深度图的一致性;其次利用投影过滤分类算法,把所有投影点按照多对一映射的关系,以3维空间中所有不同点为类型,进行投影点依次映射,将每一点划分在各自的误差云类中以求得投影点与3维点的对应关系;随后采用空间高斯分布求出每个误差云所恢复出来的点坐标.最后通过多边形技术对恢复的点云数据进行网格化,使其重建出精确的目标物体或场景的3维轮廓.从实验结果可以观察到,本文3维重建方法可以有效减少深度图融合多对一映射重复投影问题所带来的负面影响,使重建结果更加接近于真实物体,达到较好的效果.     

特征点维度的静态场景三维重建运动目标剔除

基于语义分割的图像掩膜方法常用来解决静态场景三维重建任务中运动物体的干扰问题,然而利用掩膜成功剔除运动物体的同时会产生少量无效特征点.针对此问题,提出一种在特征点维度的运动目标剔除方法,利用卷积神经网络获取运动目标信息,并构建特征点过滤模块,使用运动目标信息过滤更新特征点列表,实现运动目标的完全剔除.通过采用地面图像和航拍图像两种数据集以及DeepLabV3、YOLOv4两种图像处理算法对所提方法进行验证,结果表明特征点维度的三维重建运动目标剔除方法可以完全剔除运动目标,不产生额外的无效特征点,且相较于图像掩膜方法平均缩短13.36%的点云生成时间,减小9.93%的重投影误差.     

一种有效的基于点云特征人体识别方法

基于视觉信息的运动目标识别是室内移动机器人自主导航的一个重要研究内容,现有的方法经常由于背景噪声干 扰、自然光照、机器人视角变化等因素影响识别稳定性。本文提出一种有效的目标识别方法,滤除地平面点云特征,并采用 分割聚类和支持向量积融合方法识别运动目标。实验表明,本文方法在满足实时性前提下,相对于传统人体骨骼、二维梯度 直方图算法具有更高的准确率,更适合复杂环境下的人体识别。     

多尺度Transformer激光雷达点云3D物体检测

激光雷达点云3D物体检测,对于小物体如行人、自行车的检测精度较低,容易漏检误检,提出一种多尺度Transformer激光雷达点云3D物体检测方法MSPT-RCNN(multi-scale point transformer-RCNN),提高点云3D物体检测精度.该方法包含两个阶段,即第一阶段(RPN)和第二阶段(RCN...     

基于双目的三维点云数据的获取与预处理

在计算机辅助几何设计、医学诊断、物体识别与定位等领域的应用需求下,三维点云数据的获取与处理技术受到越来越多的关注。现在有多种不同的方式可以获取现实世界中物体的三维点云数据,并对数据进行相应处理。为了能够很好地对三维数据点云进行前期的预处理,首先通过双目摄像机获取物体的三维点云,并采用八叉树法对点云数据进行相应的预处理,然后在逆向工程软件中描述出来,从逆向工程软件中可以看出得到的物体与实际物体比较接近,从而可以证明所获取的点云数据可以用来描述物体,并且点云数据的处理技术是可行的。     

面向三维点云单目标跟踪的提案聚合网络

与二维可见光图像相比,三维点云在空间中保留了物体真实丰富的几何信息,能够应对单目标跟踪问题中存在尺度变换的视觉挑战。针对三维目标跟踪精度受到点云数据稀疏性导致的信息缺失影响,以及物体位置变化带来的形变影响这两个问题,在端到端的学习模式下提出了由三个模块构成的提案聚合网络,通过在最佳提案内定位物体的中心来确定三维边界框从而实现三维点云中的单目标跟踪。首先,将模板和搜索区域的点云数据转换为鸟瞰伪图,模块一通过空间和跨通道注意力机制丰富特征信息;然后,模块二用基于锚框的深度互相关孪生区域提案子网给出最佳提案;最后,模块三先利用最佳提案对搜索区域的感兴趣区域池化操作来提取目标特征,随后聚合了目标与模板特征,利用稀疏调制可变形卷积层来解决点云稀疏以及形变的问题并确定了最终三维边界框。在KITTI跟踪数据集上把所提方法与最新的三维点云单目标跟踪方法进行比较的实验结果表明：在汽车类综合性实验中,真实场景中所提方法在成功率上提高了1.7个百分点,精确率上提高了0.2个百分点;在多类别扩展性实验上,即在汽车、货车、骑车人以及行人这4类上所提方法的平均成功率提高了0.8个百分点,平均精确率提高了2.8个百分点。可见,所提方法能够解决三维点云中的单目标跟踪问题,使得三维目标跟踪结果更加精确。     

CAD-based Pose Estimation Design for Random Bin Picking using a RGB-D Camera

This paper presents a CAD-based six-degrees-of-freedom (6-DoF) pose estimation design for random bin picking for multiple objects. A virtual camera generates a point cloud database for the objects using their 3D CAD models. To reduce the computational time of 3D pose estimation, a voxel grid filter reduces the number of points for the 3D cloud of the objects. A voting scheme is used for object recognition and to estimate the 6-DoF pose for different objects. An outlier filter filters out badly matching poses so that the robot arm always picks up the upper object in the bin, which increases the success rate. In a computer simulation using a synthetic scene, the average recognition rate is 97.81 % for three different objects with various poses. A series of experiments have been conducted to validate the proposed method using a Kuka robot arm. The average recognition rate for three objects is 92.39 % and the picking success rate is 89.67 %.     

Model-based reconstruction of featureless objects for automated storage and transportation systems

Three-dimensional (3D) reconstruction from images usually relies on matched point correspondences across images, and the number of detected feature points should be as great as possible to fully describe the object geometry. Featureless objects, such as heaps of raw ore that are of similar color and present very few feature points, however, can usually be seen in many industrial applications, and their 3D reconstruction is essential to developing automated storage and transportation systems. Therefore, we propose a model-based method that utilizes a generic model with an additional reference point to estimate the 3D structure of a featureless object which is the ore pile in our cases. The size, position, and orientation of the estimated object are then optimized, followed by redundant area removal to fit the finer details. We also extend the method to the reconstruction of multiple objects. Experimental results show that the method successfully reconstructs featureless objects having similar colors and textures.     

基于数据特征的多传感器融合实时目标检测

为了进一步降低目标检测出现的误检率,提出了一种基于传感器数据特征的融合目标检测算法。首先,为了减少部分离群噪声点对点云表达准确性的影响,采用统计滤波器对激光雷达原始点云进行滤波处理;其次,为了解决点云地面分割在坡度变化时,固定阈值会导致分割不理想的问题,提出了自适应坡度阈值的地面分割算法;然后,建立KD（k-dimensional）树索引,加速DBSCAN（density-based spatial clustering of applications with noise）点云聚类,基于Andrew最小凸包算法,拟合最小边界矩形,生成目标三维边界框,完成聚类后的目标点云位姿估计;最后,将激光雷达检测到的三维目标点云投影到图像上,投影边界框与图像检测的目标边界框通过IoU关联匹配,提出基于决策级的三维激光雷达与视觉图像信息融合算法。使用KITTI数据集进行的测试实验表明,提出的点云聚类平均耗时降低至173 ms,相比传统的欧氏距离聚类,准确性提升6%。搭建硬件实验平台,基于实测数据的实验结果表明,提出的融合算法在目标误检率上比YOLO v4网络降低了约10%。     

Multi-class US traffic signs 3D recognition and localization via image-based point cloud model using color candidate extraction and texture-based recognition

Continuous condition monitoring and inspection of traffic signs are essential to ensure that safety and performance criteria are met. The use of 3D point cloud modeling by the construction industry has been significantly increased in recent years especially for recording the as-is conditions of facilities. The high-precision and dense 3D point clouds generated by photogrammetry can facilitate the process of asset condition assessment. This paper presents an automated computer-vision based method that detects, classifies, and localizes traffic signs via street-level image-based 3D point cloud models. The proposed pipeline integrates 3D object detection algorithm. An improved Structure-from-Motion (SfM) procedure is developed to create a 3D point cloud of roadway assets from the street level imagery. In order to assist with accurate 3D recognition and localization by color and texture features extraction, an automated process of point cloud cleaning and noise removal is proposed. Using camera pose information from SfM, the points within the bounding box of detected traffic signs are then projected into the cleaned point cloud by using the triangulation method (linear and non-linear) and the 3D points corresponding to the traffic sign in question are labeled and visualized in 3D. The proposed framework is validated using real-life data, which represent the most common types of traffic signs. The robustness of the proposed pipeline is evaluated by analyzing the accuracy in detection of traffic signs as well as the accuracy in localization in 3D point cloud model. The results promise to better and more accurate visualize the location of the traffic signs with respect to other roadway assets in 3D environment.     

The Alignment of Objects with Smooth Surfaces

This paper examines the recognition of rigid objects bounded by smooth surfaces, using an alignment approach. The projected image of such an object changes during rotation in a manner that is generally difficult to predict. An approach to this problem is suggested, using the 3D surface curvature at the points along the silhouette. The curvature information requires a single number for each point along the object′s silhouette, the radial curvature at the point. We have implemented this method and tested it on images of complex 3D objects. Models of the viewed objects were acquired using three images of each object. The implemented scheme was found to give accurate predictions of the objects′ appearances for large transformations. Using this method, a small number of (viewer-centered) models can be used to predict the new appearance of an object from any given viewpoint.     

Research on Classification of Land Cover based on LiDAR Cloud and Aerial Images

Aerial images contain abundant spectral information,texture information and spatial information,and airborne LiDAR can provide three-dimensional information of ground objects.An object-oriented classification method was researched by taking advantages of the two types of data.Converting LiDAR point cloud into 2-D raster image by preprocessing,and matched it with aerial image.Then,multi-scale segmentation algorithm was applied to image segmentation based on spectral information and height information.Next,XGBoost algorithm were applied to select features extracted from segmented object respectively.The SVM classifier was used to classify and prove the superiority of XGBoost algorithm by comparing with two traditional feature selection algorithms：Relief and RFE.Finally,objects at shadow regions were distinguished and merged into real objects based on certain rules.Testing the method in three regions,the results showed that the method was feasible and effective,and could be well applied to the classification of urban ground object.