Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network

Deep learning‐based structural damage detection methods overcome the limitation of inferior adaptability caused by extensively varying real‐world situations (e.g., lighting and shadow changes). However, most deep learning‐based methods detect structural damage at the image level and grid‐cell level. To provide pixel‐level detection of multiple damages, a Fully Convolutional Network (FCN)‐based multiple damages detection method for concrete structure is proposed. To realize this method, a database of 2,750 images (with 504 × 376 pixels) including crack, spalling, efflorescence, and hole images in concrete structure is built, and the four damages included in those images are labeled manually. Then, the architecture of the FCN is modified, trained, validated, and tested using this database. A strategy of model‐based transfer learning is used to initialize the parameters of the FCN during the training process. The results show 98.61% pixel accuracy (PA), 91.59% mean pixel accuracy (MPA), 84.53% mean intersection over union (MIoU), and 97.34% frequency weighted intersection over union (FWIoU). Subsequently, the robustness and adaptability of the trained FCN model is tested and the damage is extracted, where damage areas are provided according to a calibrated relation between the ratio (the pixel area and true area of the detected object) and the distance from the smartphone to the concrete surface using a laser range finder. A comparative study is conducted to examine the performance of the proposed FCN‐based approach using a SegNet‐based method. The results show that the proposed method substantiates quite better performance and can indeed detect multiple concrete damages at the pixel level in realistic situations.     

Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques

Sanitary sewer systems are designed to collect and transport sanitary wastewater and stormwater. Pipe inspection is important in identifying both the type and location of pipe defects to maintain the normal sewer operations. Closed-circuit television (CCTV) has been commonly utilized for sewer pipe inspection. Currently, interpretation of the CCTV images is mostly conducted manually to identify the defect type and location, which is time-consuming, labor-intensive and inaccurate. Conventional computer vision techniques are explored for automated interpretation of CCTV images, but such process requires large amount of image pre-processing and the design of complex feature extractor for certain cases. In this study, an automated approach is developed for detecting sewer pipe defects based on a deep learning technique namely faster region-based convolutional neural network (faster R-CNN). The detection model is trained using 3000 images collected from CCTV inspection videos of sewer pipes. After training, the model is evaluated in terms of detection accuracy and computation cost using mean average precision (mAP), missing rate, detection speed and training time. The proposed approach is demonstrated to be applicable for detecting sewer pipe defects accurately with high accuracy and fast speed. In addition, a new model is constructed and several hyper-parameters are adjusted to study the influential factors of the proposed approach. The experiment results demonstrate that dataset size, initialization network type and training mode, and network hyper-parameters have influence on model performance. Specifically, the increase of dataset size and convolutional layers can improve the model accuracy. The adjustment of hyper-parameters such as filter dimensions or stride values contributes to higher detection accuracy, achieving an mAP of 83%. The study lays the foundation for applying deep learning techniques in sewer pipe defect detection as well as addressing similar issues for construction and facility management.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

基于残差网络的混凝土结构病害分类识别研究

针对混凝土结构病害识别类型单一、精度较低的现状,提出了基于残差网络和迁移学习的病害分类识别方法,通过构建多属性病害数据集,利用迁移学习优化残差网络模型,提出混凝土结构健康状态识别的多个任务。首先收集混凝土结构的病害状态图像,依次通过数据清洗、尺寸均一化、数据扩增和多人投票标注,最终得到包含6 680张图像的混凝土结构病害多属性数据集,并依据不同标注属性进行了相应训练集、验证集和测试集的划分; 然后利用迁移学习对预训练的ResNet-34网络前3个部分进行参数冻结,后续2个部分的参数进行重新训练,并在模型末端添加新的参数,基于已构建的数据集进行训练; 最后在提出的构件类别检测、剥落检测、病害检测和病害类别检测任务中,分别获得84.88%、98.56%、97.18%和85.34%的F1分数。结果表明:通过构建多属性标注的混凝土结构病害数据集训练深度学习模型,可较好地实现多场景特征下的病害识别效果; 采用迁移学习技术可从开源数据中获取较好的特征提取效果; 改进的ResNet-34网络可克服网络退化问题,并针对混凝土结构病害识别的多个任务获得较好的效果; 相对于单一的混凝土结构病害识别,进行病害部位、程度、多类别的系统性检测,可为结构状态评估提供详细信息,更贴合工程实际需要。     

Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning

Although crack inspection is a routine practice in civil infrastructure management (especially for highway bridge structures), it is time‐consuming and safety‐concerning to trained engineers and costly to the stakeholders. To automate this in the near future, the algorithmic challenge at the onset is to detect and localize cracks in imagery data with complex scenes. The rise of deep learning (DL) sheds light on overcoming this challenge through learning from imagery big data. However, how to exploit DL techniques is yet to be fully explored. One primary component of practical crack inspection is that it is not merely detection via visual recognition. To evaluate the potential risk of structural failure, it entails quantitative characterization, which usually includes crack width measurement. To further facilitate the automation of machine‐vision‐based concrete crack inspection, this article proposes a DL‐enabled quantitative crack width measurement method. In the detection and mapping phase, dual‐scale convolutional neural networks are designed to detect cracks in complex scene images with validated high accuracy. Subsequently, a novel crack width estimation method based on the use of Zernike moment operator is further developed for thin cracks. The experimental results based on a laboratory loading test agree well with the direct measurements, which substantiates the effectiveness of the proposed method for quantitative crack detection.     

Unified Vision‐Based Methodology for Simultaneous Concrete Defect Detection and Geolocalization

Vision‐based autonomous inspection of concrete surface defects is crucial for efficient maintenance and rehabilitation of infrastructures and has become a research hot spot. However, most existing vision‐based inspection methods mainly focus on detecting one kind of defect in nearly uniform testing background where defects are relatively large and easily recognizable. But in the real‐world scenarios, multiple types of defects often occur simultaneously. And most of them occupy only small fractions of inspection images and are swamped in cluttered background, which easily leads to missed and false detections. In addition, the majority of the previous researches only focus on detecting defects but few of them pay attention to the geolocalization problem, which is indispensable for timely performing repair, protection, or reinforcement works. And most of them rely heavily on GPS for tracking the locations of the defects. However, this method is sometimes unreliable within infrastructures where the GPS signals are easily blocked, which causes a dramatic increase in searching costs. To address these limitations, we present a unified and purely vision‐based method denoted as defects detection and localization network, which can detect and classify various typical types of defects under challenging conditions while simultaneously geolocating the defects without requiring external localization sensors. We design a supervised deep convolutional neural network and propose novel training methods to optimize its performance on specific tasks. Extensive experiments show that the proposed method is effective with a detection accuracy of 80.7% and a localization accuracy of 86% at 0.41 s per image (at a scale of 1,200 pixels in the field test experiment), which is ideal for integration within intelligent autonomous inspection systems to provide support for practical applications.     

Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization

Many bridge structures, one of the most critical components in transportation infrastructure systems, exhibit signs of deteriorations and are approaching or beyond the initial design service life. Therefore, structural health inspections of these bridges are becoming critically important, especially after extreme events. To enhance the efficiency of such an inspection, in recent years, autonomous damage detection based on computer vision has become a research hotspot. This article proposes a three‐level image‐based approach for post‐disaster inspection of the reinforced concrete bridge using deep learning with novel training strategies. The convolutional neural network for image classification, object detection, and semantic segmentation are, respectively, proposed to conduct system‐level failure classification, component‐level bridge column detection, and local damage‐level damage localization. To enable efficient training and prediction using a small data set, the model robustness is a crucial aspect to be taken into account, generally through its hyperparameters’ selection. This article, based on Bayesian optimization, proposes a principled manner of such selection, with which very promising results (well over 90% accuracies) and robustness are observed on all three‐level deep learning models.     

Postdisaster image‐based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks

Reinforced concrete (RC) buildings are commonly used around the world. With recent earthquakes worldwide, rapid structural damage inspection and repair cost evaluation are crucial for building owners and policy makers to make informed risk management decisions. To improve the efficiency of such inspection, advanced computer vision techniques based on convolutional neural networks have been adopted in recent research to rapidly quantify the damage state (DS) of structures. In this article, an advanced object detection neural network, named YOLOv2, is implemented, which achieves 98.2% and 84.5% average precision in training and testing, respectively. The proposed YOLOv2 is used in combination with the classification neural network, which improves the identification accuracy for critical DS of RC structures by 7.5%. The improved classification procedures allow engineers to rapidly and more accurately quantify the DSs of the structure, and also localize the critical damage features. The identified DS can then be integrated with the state‐of‐the‐art performance evaluation framework to quantify the financial losses of critical RC buildings. The results can be used by the building owners and decision makers to make informed risk management decisions immediately after the strong earthquake shaking. Hence, resources can be allocated rapidly to improve the resiliency of the community.     

A deep‐learning‐based computer vision solution for construction vehicle detection

This paper aims at providing researchers and engineering professionals from the first step of solution development to the last step of solution deployment with a practical and comprehensive deep‐learning‐based solution for detecting construction vehicles. This paper places particular focus on the often‐ignored last step of deployment. Our first phase of solution development involved data preparation, model selection, model training, and model validation. Given the necessarily small‐scale nature of construction vehicle image datasets, we propose as detection model an improved version of the single shot detector MobileNet, which is suitable for embedded devices. Our study's second phase comprised model optimization, application‐specific embedded system selection, economic analysis, and field implementation. Several embedded devices were proposed and compared. Results including a consistent above 90% mean average precision confirm the superior real‐time performance of our proposed solutions. Finally, the practical field implementation of our proposed solutions was investigated. This study validates the practicality of deep‐learning‐based object detection solutions for construction scenarios. Moreover, the detailed information provided by the current study can be employed for several purposes such as safety monitoring, productivity assessments, and managerial decision making.     

A large-scale dataset of buildings and construction sites

With the rapid development of deep learning and machine automation technology, as well as workforce aging, increasing labor costs, and other issues, an increasing number of scholars have paid attention to the use of these techniques to solve problems in civil engineering. Although progress has been made in applying deep learning to damage detection, many subfields in civil engineering are still in the initial stage, and a large amount of data has not been used. Moreover, the rapid development of a field cannot be separated from large open-source datasets and many researchers. Therefore, this study attempts to construct a dataset named the BCS dataset of nearly 212,000 photos of buildings and construction sites using multi-threaded parallel crawler technology and offline collection. The dataset will be expanded regularly. As a practical demonstration, the StyleGAN3 and StyleGAN2 generative adversarial networks were utilized on the dataset to create faked safety hat images and high-resolution architectural images. Subsequently, four classic classification models were employed to validate the dataset, achieving a Top-1 accuracy of up to 0.947. These results underscore the dataset's excellent potential for practical applications.     

Deep learning-based automatic detection of muck types for earth pressure balance shield tunneling in soft ground

For the earth pressure balance shield, the muck can reflect soil information at the tunnel face to track the change in geologic conditions. Thus, this paper presents a general framework for automatic detection of muck types based on the on-site surveillance camera using deep learning algorithms. A simplified muck classification method and the corresponding muck recognition criteria are proposed for the muck detection task. The muck detection model (MDM) based on You Only Look Once v4, is established on the muck dataset for Shanghai (MSH) after some optimization treatments. The mean average precision value of 97.73% of MDM is twice that of the original model of 48.47%. The MDM is then applied to Metro Line 14 in Shanghai. Results show that The MDM performs well and meets the real-time requirements with frames per second of 60, and it outperforms other state-of-the-art detection models both in accuracy and speed.     

面向施工机械的深度学习图像数据集合成方法

工程现场环境复杂,获取包含丰富信息的图像难度大且标注成本高,造成基于计算机视觉的深度学习施工机械图像数据集构建困难。为满足快速、高质量构建建筑工程领域施工机械深度学习图像数据集,提出一种基于三维建模引擎的施工机械图像生成与自动标注方法,并以挖掘机为例构建了名为SCED(Synthesized Construction Equipment Dataset)的挖掘机数据集。首先,采用三维建模引擎UE4对目标挖掘机设备进行模型构建,然后借助UnrealCV工具对原始模型进行多角度、多区域的图像采集,使用自编写模块实现自动语义分割与掩码图像生成,并完成图像的自动标注,最终生成包含10 000张图像的数据集。与现有公开机械数据集进行了目标尺寸、数量与构建工作量的对比,并比较了构建效率与成本,最后进行了图像数据集质量与效果验证。结果表明:该构建方法综合效率更高且成本更低,构建的SCED图像数据集丰富性和泛化能力更好,针对小目标物具有更好的检测效果; 研究成果可为今后建筑施工领域深度学习图像数据集的构建提供参考依据。     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system

Crack information provides important evidence of structural degradation and safety in civil structures. Existing inspection methods are inefficient and difficult to rapidly deploy. A real‐time crack inspection method is proposed in this study to address this difficulty. Within this method, a wall‐climbing unmanned aerial system (UAS) is developed to acquire detailed crack images without distortion, then a wireless data transmission method is applied to fulfill real‐time detection requirements, allowing smartphones to receive real‐time video taken from the UAS. Next, an image data set including 1,330 crack images taken by the wall‐climbing UAS is established and used for training a deep‐learning model. For increasing detection speed, state‐of‐the‐art convolutional neural networks (CNNs) are compared and employed to train the crack detector; the selected model is transplanted into an android application so that the detection of cracks can be undertaken on a smartphone in real time. Following this, images with cracks are separated and crack width is calculated using an image processing method. The proposed method is then applied to a building where crack information is acquired and calculated accurately with high efficiency, thus verifying the practicability of the proposed method and system.     

复杂环境公路隧道裂缝快速识别与分割算法研究

随着我国公路隧道由建设为主朝建养并重转化,在运营里程快速增长与既有隧道劣化加剧的双重作用下,移动检测及结构安全快速诊断已成为目前公路隧道运营维养领域的研究热点之一。我国已研发了多种类型的隧道检测车,为裂缝、渗漏水等表观病害的快速检测提供了手段,然而公路隧道衬砌图像背景复杂、干扰因素多、裂缝占比小的特点,给检测数据的快速分析带来巨大挑战,已成为制约技术推广的主要瓶颈。基于深度学习算法,本文提出了一种将目标识别与语义分割进行组合的裂缝快速提取方法,首先采用Faster R-CNN网络对原始衬砌图像进行目标识别,判定所采集图片是否存在裂缝并智能框选出裂缝区域;随后对框选出的裂缝区域自动裁切,由此过滤不含裂缝的图片并去除含裂缝图片中的干扰背景,再利用U-Net语义分割网络对裂缝进行像素级分割。通过实际工程验证发现,单幅图像裂缝整体分割时间小于0.15 s,在常见各类干扰因素下,目标识别F1值可达到92%,语义分割像素准确度可达到98%以上。与阈值分割和同类智能分割算法相比,本方法显著提高了识别速度与精度,为从隧道检测车海量数据中进行快速准确的裂缝提取提供了良好手段。     

Iterative application of generative adversarial networks for improved buried pipe detection from images obtained by ground-penetrating radar

Ground-penetrating radar (GPR) is widely used to determine the location of buried pipes without excavation, and machine learning has been researched to automatically identify the location of buried pipes from the reflected wave images obtained by GPR. In object detection using machine learning, the accuracy of detection is affected by the quantity and quality of training data, so it is important to expand the training data to improve accuracy. This is especially true in the case of buried pipes that are located underground and whose existence cannot be easily confirmed. Therefore, this study developed a method for increasing training data using you only look once v5 (YOLOv5) and StyleGAN2-ADA to automate the annotation process. Of particular importance is developing a framework for generating images by generative adversarial networks with an emphasis on images that are challenging to detect buried pipes in YOLOv5 and add them to a training dataset to repeat training recursively, which has greatly improved the detection accuracy. Specifically, F-values of 0.915, 0.916, and 0.924 were achieved by automatically generating training images step by step from only 500, 1000, and 2000 training images, respectively. These values exceed the F-value of 0.900, which is obtained from training by manually annotating 15,000 images, a much larger number. In addition, we applied the method to a road in Shizuoka Prefecture, Japan, and confirmed that the method can detect the location of buried pipes with high accuracy on a real road. This method can contribute to labor-saving training data expansion, which is time-consuming and costly in practice, and as a result, the method contributes to improving detection accuracy.     

Pavement defect detection with fully convolutional network and an uncertainty framework

Image segmentation has been implemented for pavement defect detection, from which types, locations, and geometric information can be obtained. In this study, an integration of a fully convolutional network with a Gaussian‐conditional random field (G‐CRF), an uncertainty framework, and probability‐based rejection is proposed for detecting pavement defects. First, a fully convolutional network is designed to generate preliminary segmentation results, and a G‐CRF is used to refine the segmentation. Second, epistemic and aleatory uncertainties in the model and database are considered to overcome the disadvantages of traditional deep‐learning methods. Last, probability‐based rejection is conducted to remove unreasonable segmentations. The proposed method is evaluated on a data set of images that were obtained from 16 highways. The proposed integration segments pavement distresses from digital images with desirable performance. It also provides a satisfactory means to improve the accuracy and generalization performance of pavement defect detection without introducing a delay into the segmentation process.     

Attention-guided multiscale neural network for defect detection in sewer pipelines

Sanitary sewer systems are major infrastructures in every modern city, which are essential in protecting water pollution and preventing urban waterlogging. Since the conditions of sewer systems continuously deteriorate over time due to various defects and extrinsic factors, early intervention in the defects is necessary to prolong the service life of the pipelines. However, prior works for defect inspection are limited by accuracy, efficiency, and economic cost. In addition, the current loss functions in object detection approaches are unable to handle the imbalanced data well. To address the above drawbacks, this paper proposes an automatic defect detection framework that accurately identifies and localizes eight types of defects in closed-circuit television videos based on a deep neural network. First, an effective attention module is introduced and used in the backbone of the detector for better feature extraction. Then, a novel feature fusion mechanism is presented in the neck to alleviate the problem of feature dilution. After that, an efficient loss function that can reasonably adjust the weight of training samples is proposed to tackle the imbalanced data problem. Also, a publicly available dataset is provided for defect detection tasks. The proposed detection framework is robust against the imbalanced data and achieves a state-of-the-art mean average precision of 73.4%, which is potentially applied in realistic sewer defect inspections.     

Intelligent recognition of joints and fissures in tunnel faces using an improved mask region-based convolutional neural network algorithm

To address the challenges of low recognition accuracy, low robustness, and low detection efficiency in existing tunnel face joint and fissure recognition methods, we present a deep learning recognition segmentation algorithm called the mask region convolutional neural network (Mask R-CNN) that is enhanced by an advanced Transformer attention mechanism and deformable convolution network (Mask R-CNN-TD). The Transformer attention mechanism improves the backbone network's ability to extract image features by focusing on important areas. A deformable convolutional network enables the network to more precisely conform to the morphological characteristics of joints and fissures on the tunnel face, thereby enhancing the accuracy of detection. Experimental results demonstrate that Mask R-CNN-TD achieves superior performance, compared to Mask R-CNN series algorithms and other instance segmentation methods in terms of detection accuracy, with mean average precision scores of 70.5%, 70.8%, 53.2%, and 63.3% for detection box and mask segmentation at thresholds of 0.5 and 0.75, respectively. Based on the stable and efficient Mask R-CNN-TD model, we developed a mobile application called tunnel face detector to automatically detect tunnel faces on the construction site.     

Recording of bridge damage areas by 3D integration of multiple images and reduction of the variability in detected results

Machine learning models have been developed to perform damage detection using images to improve bridge inspection efficiency. However, in damage detection using images alone, the 3D coordinates of the damage cannot be recorded. Furthermore, the accuracy of the detection depends on the quality of the images. This paper proposes a method to integrate and record the damage detected from multiple images into a 3D model using deep learning to detect the damage from bridge images and structure from motion to identify the shooting position. The proposed method reduces the variability of the detection results between images and can assess the scale of damage or, conversely, where there is no damage and the extent of inspection omissions. The proposed method has been applied to a real bridge, and it has been shown that the actual damage locations can be recorded as a 3D model.