A self-supervised monocular depth estimation model with scale recovery and transfer learning for construction scene analysis

Estimating the depth of a construction scene from a single red-green-blue image is a crucial prerequisite for various applications, including work zone safety, localization, productivity analysis, activity recognition, and scene understanding. Recently, self-supervised representation learning methods have made significant progress and demonstrated state-of-the-art performance on monocular depth estimation. However, the two leading open challenges are the ambiguity of estimated depth up to an unknown scale and representation transferability for a downstream task, which severely hinders the practical deployment of self-supervised methods. We propose a prior information-based method, not depending on additional sensors, to recover the unknown scale in monocular vision and predict per-pixel absolute depth. Moreover, a new learning paradigm for a self-supervised monocular depth estimation model is constructed to transfer the pre-trained self-supervised model to other downstream construction scene analysis tasks. Meanwhile, we also propose a novel depth loss to enforce depth consistency when transferring to a new downstream task and two new metrics to measure transfer performance. Finally, we verify the effectiveness of scale recovery and representation transferability in isolation. The new learning paradigm with our new metrics and depth loss is expected to estimate the monocular depth of a construction scene without depth ground truth like light detection and ranging. Our models will serve as a good foundation for further construction scene analysis tasks.     

Rapid 3D object detection and modeling using range data from 3D range imaging camera for heavy equipment operation

Automated detection and modeling of 3D objects located in a construction work environment is critical for autonomous heavy equipment operation. Such automation allows for accurate, efficient, and autonomous operation of heavy equipment in a broad range of construction tasks by providing interactive background information. This paper proposes a 3D object detection and modeling system which utilizes range data obtained by 3D range imaging camera to generate 3D object models with an acceptable level of accuracy in a few seconds. The proposed system consists of four steps: data acquisition, pre-processing, object segmentation, and 3D model generation. The system was tested on the modeling of different classes of construction objects on actual construction sites. The results show that the proposed 3D object detection and modeling system achieves a good balance between speed and accuracy, and hence could be used to enhance efficiency and productivity in the autonomous operation of heavy equipment.     

Computer vision‐based recognition of 3D relationship between construction entities for monitoring struck‐by accidents

Struck‐by accidents often cause serious injuries in construction. Monitoring of the struck‐by hazards in terms of spatial relationship between a worker and a heavy vehicle is crucial to prevent such accidents. The computer vision‐based technique has been put forward for monitoring the struck‐by hazards but there exists shortages such as spatial relationship distortion due to two‐dimensional (2D) image pixels‐based estimation and self‐occlusion of heavy vehicles. This study is aimed to address these problems, including the detection of workers and heavy vehicles, three‐dimensional (3D) bounding box reconstruction for the detected objects, depth and range estimation in the monocular 2D vision, and 3D spatial relationship recognition. A series of experiments were conducted to evaluate the proposed method. The proposed method is expected to estimate 3D spatial relationship between construction worker and heavy vehicle in a real‐time and view‐invariant manner, thus enhancing struck‐by hazards monitoring at the construction site.     

Segmentation of building point cloud models including detailed architectural/structural features and MEP systems

Accurate and rapidly produced 3D models of the as-built environment can be significant assets for a variety of Engineering scenarios. Starting with a point cloud of a scene—generated using laser scanners or image-based reconstruction methods—the user must first identify collections of points that belong to individual surfaces, and then, fit surfaces and solid geometry objects appropriate for the analysis. When performed manually, this task is often prohibitively time consuming and, in response, several research groups have recently focused on developing methods for automating the modeling process. Due to the limitations of the data collection processes as well as the complexity of as-built scenes, automated 3D modeling still presents many challenges. To overcome existing limitations, in this paper, we propose a new region growing method for robust context-free segmentation of unordered point clouds based on geometrical continuities. In our method, the user sets a single parameter which accounts for the desired level of abstraction. We treat this parameter as a locally adaptive threshold to account for local context. Our method of segmentation starts with a multi-scale feature detection, describing surface roughness and curvature around each 3D point, and is followed by seed finding and region growing steps. Experimental results from seven challenging point clouds of the built environment demonstrate that our method can account for variability in point cloud density, surface roughness, curvature, and clutter within a single scene.     

A deep learning approach for construction vehicles fill factor estimation and bucket detection in extreme environments

The development of autonomous detection technology is imperative in the field of construction. The bucket fill factor is one of the main indicators for evaluating the productivity of construction vehicles. Bucket detection is a prerequisite for bucket trajectory planning. However, previous studies have been conducted under ideal environments, a specific single environment, and several normal environments without considering the actual harsh environments at construction sites. Therefore, seven extreme environments are set in this paper to fill this gap, and an effective method is proposed. First, a novel framework for image restoration under extreme environments is proposed. It applies to all tasks conducted by vision on construction sites. Second, a combination of segmentation and classification networks is used for the first time in this area. Multitask learning is used to discover a positive correlation between fill factor estimation and bucket detection. Furthermore, probabilistic methods and transfer learning were introduced, and excellent results were achieved (97.40% accuracy in fill factor estimation and 99.76% accuracy in bucket detection for seven extreme environments).     

Automated annotation for visual recognition of construction resources using synthetic images

The recognition of construction equipment is always necessary and important to monitor the progress and the safety of a construction project. Recently, the potentials of computer vision (CV) techniques have been investigated to facilitate the current equipment recognition method. However, the process of manually collecting and annotating a large image dataset of different equipment is one of the most time-consuming tasks that may delay the application of the CV techniques for construction equipment recognition. Moreover, collecting effective negative samples brings more difficulties for training the object detectors. This research aims to introduce an automated method for creating and annotating synthetic images of construction equipment while significantly reducing the required time. The synthetic images of the equipment are created from the three-dimensional (3D) models of construction machines combined with various background images taken from construction sites. The location of the equipment in the images is known since that equipment is the only object over the single-color background. This location can be extracted by applying segmentation techniques and then used for the annotation purpose. Furthermore, an automated negative image sampler is introduced in this paper to automatically generate many negative samples with different sizes out of one general image of a construction site in a way that the samples do not include the target object. The test results show that the proposed method is able to reduce the required time for annotating the images in comparison with traditional annotation methods while improving the detection accuracy.     

Automated Model‐Based Finding of 3D Objects in Cluttered Construction Point Cloud Models

Finding construction components in cluttered point clouds is a critical pre‐processing task that requires intensive and manual operations. Accurate isolation of an object from point clouds is a key for further processing steps such as positive identification, scan‐to‐building information modeling (BIM), and robotic manipulation. Manual isolaton is tedious, time consuming, and disconnected from the automated tasks involved in the process. This article adapts and examines a method for finding objects within 3D point clouds robustly, quickly, and automatically. A local feature on a pair of points is employed for representing 3D shapes. The method has three steps: (1) offline model library generation, (2) online searching and matching, and (3) match refinement and isolation. Experimental tests are carried out for finding industrial (curvilinear) and structural (rectilinear) elements. The method is verified under various circumstances in order to measure its performance toward addressing the major challenges involved in 3D object finding. Results show that the method is sufficiently quick and robust to be integrated with automated process control frameworks.     

Structure-aware dehazing of sewer inspection images based on monocular depth cues

In sewer pipes, haze caused by the humid environment seriously impairs the quality of closed-circuit television (CCTV) images, which leads to poor performance of subsequent pipe defects detection. Meanwhile, the complexity of sewer images, such as steep depth change and extensive textureless regions, brings great challenges to the performance or application of general dehazing algorithms. Therefore, this study estimates sewer depth maps first with the help of the water–pipewall borderlines to produce the paired dehazing dataset. Then a structure-aware nonlocal network (SANL-Net) is proposed with the detected borderlines and the dehazing result as two supervisory signals. SANL-Net shows its superiority over other state-of-the-art approaches with 147 in mean square error (MSE), 27.28 in peak signal to noise ratio (PSNR), 0.8963 in structural similarity index measure (SSIM), and 15.47M in parameters. Also, the outstanding performance in real image dehazing implies the accuracy of depth estimation. Experimental results indicate that SANL-Net significantly improves the performance of defects detection tasks, such as an increase of 23.16% in mean intersection over union (mIoU) for semantic segmentation.     

Automated Object Identification Using Optical Video Cameras on Construction Sites

Abstract: Visual recording devices such as video cameras, CCTVs, or webcams have been broadly used to facilitate work progress or safety monitoring on construction sites. Without human intervention, however, both real‐time reasoning about captured scenes and interpretation of recorded images are challenging tasks. This article presents an exploratory method for automated object identification using standard video cameras on construction sites. The proposed method supports real‐time detection and classification of mobile heavy equipment and workers. The background subtraction algorithm extracts motion pixels from an image sequence, the pixels are then grouped into regions to represent moving objects, and finally the regions are identified as a certain object using classifiers. For evaluating the method, the formulated computer‐aided process was implemented on actual construction sites, and promising results were obtained. This article is expected to contribute to future applications of automated monitoring systems of work zone safety or productivity.     

A Spatial‐Context‐Based Approach for Automated Spatial Change Analysis of Piece‐Wise Linear Building Elements

Changes of designs and construction plans often cause propagative design modifications, tedious construction coordination, cascading effects of errors, reworks, and delays in project management. Among various building elements, those having piece‐wise linear geometries (i.e., connected straight line segments), such as connected straight sections of ducts in mechanical, electrical, and plumbing systems, frequently undergo spatial changes in response to the changes of their surroundings. On the other hand, the piece‐wise linear geometries pose challenges to analyzing and controlling changes in construction and facility management. State‐of‐the‐art 3D change detection algorithms often face ambiguities about which points belong to which objects when piece‐wise linear object are spacked in small spaces. This article examines a spatial‐context‐based framework that uses spatial relationships between piece‐wise linear building elements (ducts in this article) to enable fast and reliable association of 3D data with ducts in as‐designed models for supporting reliable change analysis. Three case studies showed that this framework outperformed a conventional change detection method, and could handle large dislocations of piece‐wise linear elements and occlusions.     

Scalable Algorithm for Resolving Incorrect Occlusion in Dynamic Augmented Reality Engineering Environments

Abstract:  Augmented reality (AR) offers significant potential in construction, manufacturing, and other engineering disciplines that employ graphical visualization to plan and design their operations. As a result of introducing real-world objects into the visualization, less virtual models have to be deployed to create a realistic visual output that directly translates into less time and effort required to create, render, manipulate, manage, and update three-dimensional (3D) virtual contents (CAD model engineering) of the animated scene. At the same time, using the existing layout of land or plant as the background of visualization significantly alleviates the need to collect data about the surrounding environment prior to creating the final visualization while providing visually convincing representations of the processes being studied. In an AR animation, virtual and real objects must be simultaneously managed and accurately displayed to a user to create a visually convincing illusion of their coexistence and interaction. A critical challenge impeding this objective is the problem of incorrect occlusion that manifests itself when real objects in an AR scene partially or wholly block the view of virtual objects. In the presented research, a new AR occlusion handling system based on depth-sensing algorithms and frame buffer manipulation techniques was designed and implemented. This algorithm is capable of resolving incorrect occlusion occurring in dynamic AR environments in real time using depth-sensing equipment such as laser detection and ranging (LADAR) devices, and can be integrated into any mobile AR platform that allows a user to navigate freely and observe a dynamic AR scene from any vantage position.     

Intelligent recognition of joints and fissures in tunnel faces using an improved mask region-based convolutional neural network algorithm

To address the challenges of low recognition accuracy, low robustness, and low detection efficiency in existing tunnel face joint and fissure recognition methods, we present a deep learning recognition segmentation algorithm called the mask region convolutional neural network (Mask R-CNN) that is enhanced by an advanced Transformer attention mechanism and deformable convolution network (Mask R-CNN-TD). The Transformer attention mechanism improves the backbone network's ability to extract image features by focusing on important areas. A deformable convolutional network enables the network to more precisely conform to the morphological characteristics of joints and fissures on the tunnel face, thereby enhancing the accuracy of detection. Experimental results demonstrate that Mask R-CNN-TD achieves superior performance, compared to Mask R-CNN series algorithms and other instance segmentation methods in terms of detection accuracy, with mean average precision scores of 70.5%, 70.8%, 53.2%, and 63.3% for detection box and mask segmentation at thresholds of 0.5 and 0.75, respectively. Based on the stable and efficient Mask R-CNN-TD model, we developed a mobile application called tunnel face detector to automatically detect tunnel faces on the construction site.     

面向施工机械的深度学习图像数据集合成方法

工程现场环境复杂,获取包含丰富信息的图像难度大且标注成本高,造成基于计算机视觉的深度学习施工机械图像数据集构建困难。为满足快速、高质量构建建筑工程领域施工机械深度学习图像数据集,提出一种基于三维建模引擎的施工机械图像生成与自动标注方法,并以挖掘机为例构建了名为SCED(Synthesized Construction Equipment Dataset)的挖掘机数据集。首先,采用三维建模引擎UE4对目标挖掘机设备进行模型构建,然后借助UnrealCV工具对原始模型进行多角度、多区域的图像采集,使用自编写模块实现自动语义分割与掩码图像生成,并完成图像的自动标注,最终生成包含10 000张图像的数据集。与现有公开机械数据集进行了目标尺寸、数量与构建工作量的对比,并比较了构建效率与成本,最后进行了图像数据集质量与效果验证。结果表明:该构建方法综合效率更高且成本更低,构建的SCED图像数据集丰富性和泛化能力更好,针对小目标物具有更好的检测效果; 研究成果可为今后建筑施工领域深度学习图像数据集的构建提供参考依据。     

A Saliency-Based Method for Early Smoke Detection in Video Sequences

Video-based smoke detection requires suspected smoke regions to be segmented from the complex background in the initial stage of detection. This segmentation is also important to the subsequent processes of detection. This paper proposes a novel method of segmenting a smoke region in smoke pixel classification based on saliency detection. A salient smoke detection model based on color and motion features is used. First, smoke regions are identified by enhancing the smoke color nonlinearly. The enhanced map and motion map are then used to measure saliency. Finally, the motion energy and saliency map are used to estimate the suspected smoke regions. The estimation result is regarded as our final smoke pixel segmentation result. The performance of the proposed algorithm is verified on a set of videos containing smoke. In the experiments, the method achieves average smoke segmentation precision of 93.0%, and the precision is as high as 99.0% for forest fires. The results are compared with those of three other methods used in the literature, revealing the proposed method to have both a better segmentation result and better precision. We also present encouraging results of smoke segmentation in video sequences obtained using the proposed saliency detection method. Furthermore, the proposed smoke segmentation method can be used for real-time fire detection in color video sequences.     

数码校园GIS中的三维建模

为满足“数字地球”时代的发展要求 ,地理信息系统 (GeographicInformationSystem ,GIS)必须把三维世界真实地表达出来 ,因此如何更有效地获取地球上的各种信息是个关键问题。在此以清华大学三维景观的建立为例 ,利用数字摄影测量技术对地面模型及地物进行三维建模 ,对于不能直接从航片上获取的复杂地物 ,则采用 3DGIS软件进行单个模型的建立 ,通过这种建模方式的结合 ,可大大节省实现三维可视化的时间。实践证明 ,这种建模方法的效果令人满意 ,不仅是一种快速有效的建模方法 ,而且更新也很方便 ,特别是对大范围区域更显示其优越性     

Fire detection and 3D surface reconstruction based on stereoscopic pictures and probabilistic fuzzy logic

In this paper, a new fire detection method is proposed, which is based on using a stereo camera to calculate the distance between the camera and the fire region and to reconstruct the 3D surface of the fire front. For the purpose of fire detection, candidate fire regions are identified using generic color models and a simple background difference model. Gaussian membership functions (GMFs) for the shape, size, and motion variation of the fire are then generated, because fire regions in successive frames change constantly. These three GMFs are then applied to fuzzy logic for real-time fire verification. After segmentation of the fire regions from left and right images, feature points are extracted using a matching algorithm and their disparities are computed for distance estimation and 3D surface reconstruction. Our proposed algorithm was successfully applied to a fire video dataset and its detection performance was shown to be better than that of other methods. In addition, the distance estimation method yielded reasonable results when the fire was a short distance from the camera and the reconstruction of the 3D surface showed a shape that was almost the same as that of the real fire.     

3D Object Classification Using Geometric Features and Pairwise Relationships

Object classification is a key differentiator of building information modeling (BIM) from three‐dimensional (3D) computer‐aided design (CAD). Incorrect object classification impedes the full exploitation of BIM models. Models prepared using domain‐specific software cannot ensure correct object classification when transferred to other domains, and research on reconstruction of BIM models using spatial survey has not proved a full capability to classify objects. This research proposed an integrated approach to object classification that applied domain experts’ knowledge of shape features and pairwise relationships of 3D objects to effectively classify objects using a tailored matching algorithm. Among its contributions: the algorithms implemented for shape and spatial feature identification could process various complex 3D geometry; the method devised for compilation of the knowledge base considered both rigor and confidence of the inference; the algorithm for matching provides mathematical measurement of the object classification results. The integrated approach has been applied to classify 3D bridge objects in two models: a model prepared using incorrect object types and a model manually reconstructed using point cloud data. All these objects were successfully classified.