Efficient 3D object recognition using foveated point clouds

Recent hardware technologies have enabled acquisition of 3D point clouds from real world scenes in real time. A variety of interactive applications with the 3D world can be developed on top of this new technological scenario. However, a main problem that still remains is that most processing techniques for such 3D point clouds are computationally intensive, requiring optimized approaches to handle such images, especially when real time performance is required. As a possible solution, we propose the use of a 3D moving fovea based on a multiresolution technique that processes parts of the acquired scene using multiple levels of resolution. Such approach can be used to identify objects in point clouds with efficient timing. Experiments show that the use of the moving fovea shows a seven fold performance gain in processing time while keeping 91.6% of true recognition rate in comparison with state-of-the-art 3D object recognition methods.     

Instance-based object recognition in 3D point clouds using discriminative shape primitives

3D local shapes are a critical cue for object recognition in 3D point clouds. This paper presents an instance-based 3D object recognition method via informative and discriminative shape primitives. We propose a shape primitive model that measures geometrical informativity and discriminativity of 3D local shapes of an object. Discriminative shape primitives of the object are extracted automatically by model parameter optimization. We achieve object recognition from 2.5/3D scenes via shape primitive classification and recover the 3D poses of the identified objects simultaneously. The effectiveness and the robustness of the proposed method were verified on popular instance-based 3D object recognition datasets. The experimental results show that the proposed method outperforms some existing instance-based 3D object recognition pipelines in the presence of noise, varying resolutions, clutter and occlusion.     

An evolutionary approach to the extraction of object construction trees from 3D point clouds

In order to extract a construction tree from a finite set of points sampled on the surface of an object, we present an evolutionary algorithm that evolves set-theoretic expressions made of primitives fitted to the input point-set and modeling operations. To keep relatively simple trees, we use a penalty term in the objective function optimized by the evolutionary algorithm. We show with experiments successes but also limitations of this approach.     

Productive high-complexity 3D city modeling with point clouds collected from terrestrial LiDAR

To satisfy the needs of photo-realistic and ground-based representation of three-dimensional (3D) city models for a variety of applications, significant efforts have been made to automatically reconstruct detailed 3D building façades from terrestrial LiDAR data. Nonetheless, in real-world applications for high-quality 3D city modeling, three major problems are typically encountered: (1) very low productivity due to fully manual operation, (2) low geometric accuracy of 3D modeling resulting from the process of reducing original LiDAR data, and (3) system failure when importing huge LiDAR data to 3D drawing software. To overcome these limitations, the present study proposes a semi-automatic method entailing a plane component detection based on RANSAC segmentation, boundary tracing of the planar components, and manual drawing of details using the remaining, significantly reduced points. The proposed method was applied to point clouds of various buildings in a high-density area in Korea. In comparison with manual operation, the proposed method was proved to improve modeling productivity in the time-consumption aspect and to facilitate operators’ accurate object drawing. However, for additional automation and completeness of 3D modeling, further study is necessary. The proposed method requires a segmentation algorithm to heuristically determine parameters for the most desirable results as well as to detect curvilinear surfaces in modeling complex and curved façades.     

一种新的基于特殊离群样本优化的三维点云特征选择算法

随着元宇宙、数字孪生、虚拟现实与增强现实等前沿技术的快速发展,三维点云在电力、建筑、先进制造等行业中得到广泛应用,随之而来的,如何降低三维点云数据冗余度、有效进行点云特征选择,已在充分利用海量点云数据中扮演着关键角色。考虑到现有大多数三维点云特征选择算法忽略了特定样本在特征评估中的表现,提出一种新的有监督特征选择算法,即基于特殊离群样本优化的特征选择算法(FSSO)。具体地,为获得精准的特殊离群样本(SOs),FSSO优化均值中心并动态地界定类簇主体;计算SOs的类内相对偏离程度,通过减小类内相对偏离对特征进行打分,实现特征选择过程。在3个公共的三维点云模型分类数据集上(ModelNet40,IntrA,ShapeNetCore)的实验,以及4个高维人工特征数据集的验证实验结果表明,相较于其他特征选择算法,FSSO可选择出具有更强分类能力的特征子集,并提升分类准确率。     

Multi-scale tensor voting for feature extraction from unstructured point clouds

Identifying sharp features in a 3D model is essential for shape analysis, matching and a wide range of geometry processing applications. This paper presents a new method based on the tensor voting theory to extract sharp features from an unstructured point cloud which may contain random noise, outliers and artifacts. Our method first takes the voting tensors at every point using the corresponding neighborhoods and computes the feature weight to infer the local structure via eigenvalue analysis of the tensor. The optimal scale for a point is automatically determined by observing the feature weight variation in order to deal with both a noisy smooth region and a sharp edge. We finally extract the points at sharp features using adaptive thresholding of the feature weight and the feature completion process. The multi-scale tensor voting of a given point set improves noise sensitivity and scale dependency of an input model. We demonstrate the strength of the proposed method in terms of efficiency and robustness by comparing it with other feature detection algorithms.     

Urban 3D segmentation and modelling from street view images and LiDAR point clouds

3D urban maps with semantic labels and metric information are not only essential for the next generation robots such autonomous vehicles and city drones, but also help to visualize and augment local environment in mobile user applications. The machine vision challenge is to generate accurate urban maps from existing data with minimal manual annotation. In this work, we propose a novel methodology that takes GPS registered LiDAR (Light Detection And Ranging) point clouds and street view images as inputs and creates semantic labels for the 3D points clouds using a hybrid of rule-based parsing and learning-based labelling that combine point cloud and photometric features. The rule-based parsing boosts segmentation of simple and large structures such as street surfaces and building facades that span almost 75% of the point cloud data. For more complex structures, such as cars, trees and pedestrians, we adopt boosted decision trees that exploit both structure (LiDAR) and photometric (street view) features. We provide qualitative examples of our methodology in 3D visualization where we construct parametric graphical models from labelled data and in 2D image segmentation where 3D labels are back projected to the street view images. In quantitative evaluation we report classification accuracy and computing times and compare results to competing methods with three popular databases: NAVTEQ True, Paris-Rue-Madame and TLS (terrestrial laser scanned) Velodyne.     

3D Hough transform for sphere recognition on point clouds

Three-dimensional object recognition on range data and 3D point clouds is becoming more important nowadays. Since many real objects have a shape that could be approximated by simple primitives, robust pattern recognition can be used to search for primitive models. For example, the Hough transform is a well-known technique which is largely adopted in 2D image space. In this paper, we systematically analyze different probabilistic/randomized Hough transform algorithms for spherical object detection in dense point clouds. In particular, we study and compare four variants which are characterized by the number of points drawn together for surface computation into the parametric space and we formally discuss their models. We also propose a new method that combines the advantages of both single-point and multi-point approaches for a faster and more accurate detection. The methods are tested on synthetic and real datasets.     

无需阈值支持的机载LiDAR点云数据滤波方法

点云数据滤波仍旧是现阶段机载LiDAR数据后处理的首要步骤,但其发展尚未完全成熟。在回顾和总结已有滤波算法的基础上,将统计学中偏度与峰度的概念引入到算法中,提出了一种新的基于偏度平衡的地面点与非地面点非监督分类方法,利用统计矩原理从LiDAR点云数据生成的DSM中有效地提取DTM。该方法区别传统算法的最大的优势在于无需参数或者阈值支持,并且相对于LiDAR点云数据的格式和分辨率是独立的。实验结果证明,该方法切实可行,具有较强的适应性,并且能够较好地满足精度要求。     

Automatic model-based 3D object recognition by combining feature matching with tracking

We propose a vision-based robust automatic 3D object recognition, which provides object identification and 3D pose information by combining feature matching with tracking. For object identification, we propose a robust visual feature and a probabilistic voting scheme. An initial object pose is estimated using correlations between the model image and the 3D CAD model, which are predefined, and the homography, byproduct of the identification. In tracking, a Lie group formalism is used for robust and fast motion computation. Experimental results show that object recognition by the proposed method improves the recognition range considerably. Sungho Kim received the B.S. degree in Electrical Engineering from Korea University, Korea in 2000 and the M.S. degree in Electrical Engineering and Computer Science from Korea Advanced Institute of Science and Technology, Korea in 2002. He is currently pursuing his Ph.D. at the latter institution, concentrating on 3D object recognition and tracking. In So Kweon received the Ph.D. degree in robotics from Carnegie Mellon University, Pittsburgh, PA, in 1990. Since 1992, he has been a Professor of Electrical Engineering at KAIST. His current research interests include human visual perception, object recognition, real-time tracking, vision-based mobile robot localization, volumetric 3D reconstruction, and camera calibration. He is a member of the IEEE, and Korea Robotics Society (KRS).     

Multi-class object detection in tunnels from 3D point clouds: An auto-optimized lazy learning approach

This research develops an auto-optimized lazy learning approach named BOKNN (Bayesian optimized K nearest neighbor) method to detect seepage and multi-classify various objects (e.g., segment, pipe, track, support, and cable) in operating tunnels from 3D point clouds. Firstly, the 3D laser scanning is employed to acquire raw point cloud data, and the equidistant pooling for down-sampling is conducted to improve class imbalance issues and enhance the efficiency. Then, the K-nearest neighbor (KNN) model is built on the trimmed dataset, where the Bayesian optimization is performed to obtain the optimal combination of hyper-parameters in the KNN model. A realistic cross-river tunnel section in China is used as a case study to demonstrate the applicability and effectiveness of the developed approach. Results indicate that (1) The established BOKNN model displays a high performance in multi-class detection, together with a total accuracy of 0.935, a macro F1 score of 0.896, and a weighted F1 score of 0.939. (2) It performs well even in minor class detection, and the detection of seepage is conservative, where only 4.1% of seepage points are misclassified as non-seepage points. (3) It displays better detection performance than the other representative machine learning models (i.e, Adaboost, Support Vector Machine, and Naive Bayes). The developed approach is nonparametric and training-free, which can be used as a decision tool to substitute the present manual detection and improve the detection efficiency.     

轻量级的三维点云识别方法

针对当前三维点云识别方法存在时间和空间复杂度较高的问题,提出一种轻量级的三维点云识别方法.使用最远点采样法从原始点云中获取采样点,在采样点处构建K近邻图来有效获取点云的局部结构信息,大幅度降低网络的计算复杂度;使用注意力机制突出局部区域不同K近邻点的重要性,达到增强局部结构特征的目的;利用分组卷积提取高层次的局部结构特征的同时减少卷积层的参数量.在保证较高的识别准确率的前提下减少全连接层的参数量.在M odelNet40数据集上的实验结果表明,该方法的识别准确率达到同等或更优水平,网络模型的参数量得到大幅度减少,训练时间和测试时间更短,同时保持较高的鲁棒性.     

融合时序特征约束与联合优化的 点云3维人体姿态序列估计

目的 3维人体姿态估计传统方法通常采用单帧点云作为输入,可能会忽略人体运动平滑度的固有先验知识,导致产生抖动伪影。目前,获取2维人体姿态标注的真实图像数据集相对容易,而采集大规模的具有高质量3维人体姿态标注的真实图像数据集进行完全监督训练有一定难度。对此,本文提出了一种新的点云序列3维人体姿态估计方法。方法 首先从深度图像序列估计姿态相关点云,然后利用时序信息构建神经网络,对姿态相关点云序列的时空特征进行编码。选用弱监督深度学习,以利用大量的更容易获得的带2维人体姿态标注的数据集。最后采用多任务网络对人体姿态估计和人体运动预测进行联合训练,提高优化效果。结果 在两个数据集上对本文算法进行评估。在ITOP(invariant-top view dataset)数据集上,本文方法的平均精度均值(mean average precision,mAP)比对比方法分别高0.99%、13.18%和17.96%。在NTU-RGBD数据集上,本文方法的mAP值比最先进的WSM(weakly supervised adversarial learning methods)方法高7.03%。同时,在ITOP数据集上对模型进行消融实验,验证了算法各个不同组成部分的有效性。与单任务模型训练相比,多任务网络联合进行人体姿态估计和运动预测的mAP可以提高2%以上。结论 本文提出的点云序列3维人体姿态估计方法能充分利用人体运动连续性的先验知识,获得更平滑的人体姿态估计结果,在ITOP和NTU-RGBD数据集上都能获得很好的效果。采用多任务网络联合优化策略,人体姿态估计和运动预测两个任务联合优化求解,有互相促进的作用。     

Adaptive 3-D object recognition from multiple views

The authors address the problem of generating representations of 3-D objects automatically from exploratory view sequences of unoccluded objects. In building the models, processed frames of a video sequence are clustered into view categories called aspects, which represent characteristic views of an object invariant to its apparent position, size, 2-D orientation, and limited foreshortening deformation. The aspects as well as the aspect transitions of a view sequence are used to build (and refine) the 3-D object representations online in the form of aspect-transition matrices. Recognition emerges as the hypothesis that has accumulated the maximum evidence at each moment. The `winning' object continues to refine its representation until either the camera is redirected or another hypothesis accumulates greater evidence. This work concentrates on 3-D appearance modeling and succeeds under favorable viewing conditions by using simplified processes to segment objects from the scene and derive the spatial agreement of object features     

Slice-based building facade reconstruction from 3D point clouds

ABSTRACT

We propose a comprehensive strategy to reconstruct urban building geometry from three-dimensional (3D) point clouds. First, the point clouds are segmented using a rough-detail segmentation algorithm, and refinements guided by topological relationships are performed to rectify the segmentation mistakes. Then, the semantic features (such as facades and windows) that belong to the buildings are recognized and extracted. Next, each facade is cut into a sequence of slices. The initial models are recovered by sequentially detecting and connecting the anchor points. Finally, due to the regular arrangements of windows, a template-matching method relying on the similarity and repetitiveness of the windows is proposed to recover the details on building facades. The experimental results demonstrate that our method can automatically reconstruct the building geometry and detailed window structures are better depicted.     

改进的基于冗余点过滤的3D目标检测方法

VoxelNet网络模型是第一个基于点云的端对端目标检测网络，只利用点云数据来生成高精度的3D目标检测框，具有十分良好的效果。但是，VoxelNet使用完整场景的点云数据作为输入，导致耗费了更多的计算资源在背景点云数据上，而且只包含几何信息的点云对目标的识别粒度较低，在较复杂的场景中容易出现误检测和漏检测。针对这些问题对VoxelNet进行了改进，在VoxelNet模型中加入视锥体候选区。首先，通过RGB前视图对感兴趣目标进行定位；然后，将目标2D位置升维至空间视锥体，在点云中提取目标视锥体候选区，过滤冗余点云，仅对视锥体候选区中的点云数据进行计算来得到检测结果。改进后的算法与VoxelNet相比，降低了点云计算量，避免了对背景点云数据的计算，提升了有效运算率，同时，避免了过多背景点的干扰，降低了误检测和漏检测率。KITTI数据集上的实验结果表明，改进后的算法在简单、中等、困难三种模式下的3D平均精度分别为67.92%、59.98%、53.95%，优于VoxelNet模型。     

改进的基于冗余点过滤的3D目标检测方法

VoxelNet网络模型是第一个基于点云的端对端目标检测网络,只利用点云数据来生成高精度的3D目标检测框,具有十分良好的效果。但是,VoxelNet使用完整场景的点云数据作为输入,导致耗费了更多的计算资源在背景点云数据上,而且只包含几何信息的点云对目标的识别粒度较低,在较复杂的场景中容易出现误检测和漏检测。针对这些问题对VoxelNet进行了改进,在VoxelNet模型中加入视锥体候选区。首先,通过RGB前视图对感兴趣目标进行定位;然后,将目标2D位置升维至空间视锥体,在点云中提取目标视锥体候选区,过滤冗余点云,仅对视锥体候选区中的点云数据进行计算来得到检测结果。改进后的算法与VoxelNet相比,降低了点云计算量,避免了对背景点云数据的计算,提升了有效运算率,同时,避免了过多背景点的干扰,降低了误检测和漏检测率。KITTI数据集上的实验结果表明,改进后的算法在简单、中等、困难三种模式下的3D平均精度分别为67.92%、59.98%、53.95%,优于VoxelNet模型。