Encoder–decoder network for pixel‐level road crack detection in black‐box images

Timely monitoring of pavement cracks is essential for successful maintenance of road infrastructure. Accurate information concerning crack location and severity enables proactive management of the infrastructure. Black‐box cameras, which are becoming increasingly widespread at an affordable price, can be used as efficient road‐image collectors over a wide area. However, the cracks in these images are difficult to detect, because the images containing them often include objects other than roads. Thus, we propose a pixel‐level detection method for identifying road cracks in black‐box images using a deep convolutional encoder–decoder network. The encoder consists of convolutional layers of the residual network for extracting crack features, and the decoder consists of deconvolutional layers for localizing the cracks in an input image. The proposed network was trained on 427 out of 527 images extracted from black‐box videos and tested on the remaining 100 images. Compared with VGG‐16, ResNet‐50, ResNet‐101, ResNet‐200 with transfer learning, and ResNet‐152 without transfer learning, ResNet‐152 with transfer learning exhibited the best performance, achieving recall, precision, and intersection of union of 71.98%, 77.68%, and 59.65%, respectively. The experimental results prove that the proposed method is optimal for detecting cracks in black‐box images at the pixel level.     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

基于卷积神经网络的结构损伤识别研究进展

为系统梳理基于卷积神经网络的工程结构损伤识别方法的发展脉络和研究现状,分别从结构损伤的识别目的和在不同类型结构中的应用两方面进行了归类、分析和评价。介绍了卷积神经网络的基本结构和评价指标,回顾了卷积神经网络的研究和应用历程。在损伤的识别目的方面,主要针对混凝土结构损伤的分类、定位和分割,详细介绍了基于不同类型卷积神经网络的结构损伤识别方法,即基于分类的方法、基于回归的方法和像素级的图像分割算法; 分析了各类方法所使用的卷积神经网络模型的结构特点、计算流程、训练方法和损伤识别性能。在不同类型结构的损伤识别方面,分析了卷积神经网络在砌体结构、钢结构桥梁和古建筑木结构裂缝识别中的应用。最后,基于对卷积神经网络优缺点的思考,提出了发展建议和展望。结果表明:训练样本中结构损伤的多样性对模型的损伤识别效果影响较大; 现有基于卷积神经网络的损伤分割方法模型参数较多,计算量大; 采用数据增广和迁移学习方法可有效防止模型过拟合,提高模型训练效率; 针对微小损伤和不同类型结构损伤的识别,此类方法的性能有待提高。     

A unified convolutional neural network integrated with conditional random field for pipe defect segmentation

Semantic segmentation of closed‐circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel on the image, from which defect types, locations, and geometric information can be obtained. In this study, a unified neural network, namely DilaSeg‐CRF, is proposed by fully integrating a deep convolutional neural network (CNN) with dense conditional random field (CRF) for improving the segmentation accuracy. First, DilaSeg is constructed with dilated convolution and multiscale techniques for producing feature maps with high resolution. The steps of the dense CRF inference algorithm are converted into CNN operations, which are then formulated as recurrent neural network (RNN) layers. The DilaSeg‐CRF is proposed by integrating DilaSeg with the RNN layers. Images containing three common types of sewer defects are collected from CCTV inspection videos and are annotated with ground truth labels, after which the proposed models are trained and evaluated. Experiments demonstrate that the end‐to‐end trainable DilaSeg‐CRF can improve the segmentation significantly, with an increase of 32% and 20% in mean intersection over union (mIoU) values compared with fully convolutional network (FCN‐8s) and DilaSeg, respectively. Our proposed DilaSeg‐CRF also achieves faster inference speed than FCN and eliminates the manual postprocessing for refining the segmentation results.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

Self-training approach for crack detection using synthesized crack images based on conditional generative adversarial network

Urban infrastructure plays a crucial role in determining the quality of life for citizens. However, given the increasing number of aging infrastructures, regular inspections are essential to prevent accidents. Deep learning studies have been conducted to detect structural damage and ensure high accuracy and reliability of these inspections. However, these detection algorithms often face challenges due to scarcity of damage data. To overcome this issue, this paper proposes a method for synthesizing crack images and utilizing them for crack detection. Initially, crack images are synthesized from labeled images by using a conditional generative adversarial network. Subsequently, a new self-training method is implemented wherein the synthesized crack images from the prediction images were incorporated into the learning process to enhance data diversity. The proposed approach yields promising results with a mean intersection over union of 80.34% and F1-score of 76.31% on average. The proposed method can aid further research on virtual image generation for crack detection, seeking to reduce the reliance on extensive image collection.     

Automatic detection method of cracks from concrete surface imagery using two‐step light gradient boosting machine

Automated crack detection based on image processing is widely used when inspecting concrete structures. The existing methods for crack detection are not yet accurate enough due to the difficulty and complexity of the problem; thus, more accurate and practical methods should be developed. This paper proposes an automated crack detection method based on image processing using the light gradient boosting machine (LightGBM), one of the supervised machine learning methods. In supervised machine learning, appropriate features should be identified to obtain accurate results. In crack detection, the pixel values of the target pixels and geometric features of the cracks that occur when they are connected linearly should be considered. This paper proposes a methodology for generating features based on pixel values and geometric shapes in two stages. The accuracy of the proposed methodology is investigated using photos of concrete structures with adverse conditions, such as shadows and dirt. The proposed methodology achieves an accuracy of 99.7%, sensitivity of 75.71%, specificity of 99.9%, precision of 68.2%, and an F‐measure of 0.6952. The experimental results demonstrate that the proposed method can detect cracks with higher performance than the pix2pix‐based approach. Furthermore, the training time is 7.7 times shorter than that of the XGBoost and 2.3 times shorter than that of the pix2pix. The experimental results demonstrate that the proposed method can detect cracks with high accuracy.     

Machine learning based novelty detection using modal analysis

Among many structural assessment methods, the change of modal characteristics is considered a well‐accepted damage detection method. However, the presence of environmental or operational variations may pollute the baseline and prevent a dependable assessment of the change. In recent years, the use of machine learning algorithms gained interest within structural health community, especially due to their ability and success in the elimination of ambient uncertainty. This paper proposes an end‐to‐end architecture to detect damage reliably by employing machine learning algorithms. The proposed approach streamlines (a) collection of structural response data, (b) modal analysis using system identification, (c) learning model, and (d) novelty detection. The proposed system aims to extract latent features of accessible modal parameters such as natural frequencies and mode shapes measured at undamaged target structure under temperature uncertainty and to reconstruct a new representation of these features that is similar to the original using well‐established machine learning methods for damage detection. The deviation between measured and reconstructed parameters, also known as novelty index, is the essential information for detecting critical changes in the system. The approach is evaluated by analyzing the structural response data obtained from finite element models and experimental structures. For the machine learning component of the approach, both principal component analysis (PCA) and autoencoder (AE) are examined. While mode shapes are known to be a well‐researched damage indicator in the literature, to our best knowledge, this research is the first time that unsupervised machine learning is applied using PCA and AE to utilize mode shapes in addition to natural frequencies for effective damage detection. The detection performance of this pipeline is compared to a similar approach where its learning model does not utilize mode shapes. The results demonstrate that the effectiveness of the damage detection under temperature variability improves significantly when mode shapes are used in the training of learning algorithm. Especially for small damages, the proposed algorithm performs better in discriminating system changes.     

Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization

Many bridge structures, one of the most critical components in transportation infrastructure systems, exhibit signs of deteriorations and are approaching or beyond the initial design service life. Therefore, structural health inspections of these bridges are becoming critically important, especially after extreme events. To enhance the efficiency of such an inspection, in recent years, autonomous damage detection based on computer vision has become a research hotspot. This article proposes a three‐level image‐based approach for post‐disaster inspection of the reinforced concrete bridge using deep learning with novel training strategies. The convolutional neural network for image classification, object detection, and semantic segmentation are, respectively, proposed to conduct system‐level failure classification, component‐level bridge column detection, and local damage‐level damage localization. To enable efficient training and prediction using a small data set, the model robustness is a crucial aspect to be taken into account, generally through its hyperparameters’ selection. This article, based on Bayesian optimization, proposes a principled manner of such selection, with which very promising results (well over 90% accuracies) and robustness are observed on all three‐level deep learning models.     

A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects

Tunnel lining defects are an important indicator reflecting the safety status of shield tunnels. Inspired by the state‐of‐the‐art deep learning, a method for automatic intelligent classification and detection methodology of tunnel lining defects is presented. A fully convolutional network (FCN) model for classification is proposed. Information about defects, collected using charge‐coupled device cameras, was used to train the model. The model's performance was compared to those of GoogLeNet and VGG. The best‐set accuracy of the proposed model was over 95% at a test‐time speed of 48 ms per image. For defects detection, image features were computed from large‐scale images by the FCN and then detected using a region proposal network and position‐sensitive region of interest pooling. Some indices (detection rate, detection accuracy, and detection efficiency, locating accuracy) were used to evaluate the model. The comparisons with faster R‐CNN and a traditional method were conducted. The results show that the model is very fast and efficient, allowing automatic intelligent classification and detection of tunnel lining defects.     

Multi-source information fusion applied to structural damage diagnosis

This study aims to import multi-source information fusion (MSIF) into structural damage diagnosis to improve its validity. Two structural damage identification methods based on MSIF are put forward, one of which is to fuse two or more structural damage detection methods by MSIF and another of which is the improved modal strain energy method by multi-mode information processing based on MSIF. Through a concrete plate experiment it is proved that, if two methods are integrated by character-level information fusion, structural initial damages can be more accurately identified than by a single method. In a simulation of a concrete box beam bridge, it is indicated that the improved modal strain energy method has a nice sensitivity to structural initial damages and a favorable robusticity to noise. These two structural damage diagnosis methods based on MSIF have good effects on structural damage identification and good practicability to actual structures.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.     

Concrete crack detection using context‐aware deep semantic segmentation network

Computer‐vision and deep‐learning techniques are being increasingly applied to inspect, monitor, and assess infrastructure conditions including detection of cracks. Traditional vision‐based methods to detect cracks lack accuracy and generalization to work on complicated infrastructural conditions. This paper presents a novel context‐aware deep convolutional semantic segmentation network to effectively detect cracks in structural infrastructure under various conditions. The proposed method applies a pixel‐wise deep semantic segmentation network to segment the cracks on images with arbitrary sizes without retraining the prediction network. Meanwhile, a context‐aware fusion algorithm that leverages local cross‐state and cross‐space constraints is proposed to fuse the predictions of image patches. This method is evaluated on three datasets: CrackForest Dataset (CFD) and Tomorrows Road Infrastructure Monitoring, Management Dataset (TRIMMD) and a Customized Field Test Dataset (CFTD) and achieves Boundary F1 (BF) score of 0.8234, 0.8252, and 0.7937 under 2‐pixel error tolerance margin in CFD, TRIMMD, and CFTD, respectively. The proposed method advances the state‐of‐the‐art performance of BF score by approximately 2.71% in CFD, 1.47% in TRIMMD, and 4.14% in CFTD. Moreover, the averaged processing time of the proposed system is 0.7 s on a typical desktop with Intel^® Quad‐Core? i7‐7700 CPU@3.6 GHz Processor, 16GB RAM and NVIDIA GeForce GTX 1060 6GB GPU for an image of size 256 × 256 pixels.     

Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning

Although crack inspection is a routine practice in civil infrastructure management (especially for highway bridge structures), it is time‐consuming and safety‐concerning to trained engineers and costly to the stakeholders. To automate this in the near future, the algorithmic challenge at the onset is to detect and localize cracks in imagery data with complex scenes. The rise of deep learning (DL) sheds light on overcoming this challenge through learning from imagery big data. However, how to exploit DL techniques is yet to be fully explored. One primary component of practical crack inspection is that it is not merely detection via visual recognition. To evaluate the potential risk of structural failure, it entails quantitative characterization, which usually includes crack width measurement. To further facilitate the automation of machine‐vision‐based concrete crack inspection, this article proposes a DL‐enabled quantitative crack width measurement method. In the detection and mapping phase, dual‐scale convolutional neural networks are designed to detect cracks in complex scene images with validated high accuracy. Subsequently, a novel crack width estimation method based on the use of Zernike moment operator is further developed for thin cracks. The experimental results based on a laboratory loading test agree well with the direct measurements, which substantiates the effectiveness of the proposed method for quantitative crack detection.     

Automated Analysis of Mobile LiDAR Data for Component‐Level Damage Assessment of Building Structures during Large Coastal Storm Events

Rapid assessment of building damages due to natural disasters is a critical element in disaster management. Although airborne‐based remote sensing techniques have been successfully applied in many postdisaster scenarios, automated building component‐level damage assessment with terrestrial/mobile LiDAR data is still challenging to achieve due to lack of reliable segmentation methods for damaged buildings. In this research, a novel building segmentation and damage detection approach is proposed to realize automated component‐level damage assessment for major building envelop elements including wall, roof, balcony, column, and handrail. The proposed approach first conducts semantic segmentation of building point cloud data using a rule‐based approach. The detected building components are then evaluated to determine if the components are damaged. The authors applied this method on a mobile LiDAR data set collected after Hurricane Sandy. The results demonstrate that the approach is capable of achieving 96% and 86% parsing accuracy for wall façades and roof facets, and obtain 82% and 78% of detection accuracy for damaged walls and roof facets.     

Fractal Dimension‐Based Damage Detection Method for Beams with a Uniform Cross‐Section

Abstract: For civil structures, damage usually occurs in localized areas. As fractal dimension (FD) analysis can provide insight to local complexity in geometry, a damage detection approach based on Katz's estimation of the FD measure of displacement mode shape for homogeneous, uniform cross‐sectional beam structures is proposed in this study. An FD‐based index for damage localization (FDIDL) is developed utilizing the difference of angles of sliding windows between two successive points, which is expressed in FD. To improve robustness against noise, FDIDL is calculated using multisliding windows. The influence of the spatial sampling interval length and the number of 2‐sampling sliding windows on sensitivity to damage and robustness against noise is investigated. The relationship between the angle expressed in FD and the modal strain energy is established and thereby an FD‐based index for the estimation of damage extent (FDIDE) is presented. The two damage indices are applied to a simply supported beam to detect the simulated damage in the beam. The results indicate that the proposed FDIDL can locate the single or multiple damages, and FDIDE can reliably quantify the damage extent. The optimal spatial sampling interval and the number of sliding windows are investigated. Furthermore, the simulation with measurement noise is carried out to demonstrate the effectiveness and robustness of the two defined FD‐based damage indices. Finally, experiments are conducted on simply supported steel beams damaged at different locations. It is demonstrated that the proposed approach can locate the damages to a satisfactory precision.     

Betonkorrosion durch Schwefelsäure an Abwasserbauwerken. Innovative Schadensdiagnose

Concrete Corrosion at Sewage Plants caused by Sulphuric Acid Innovative Damage Analysis Successful concrete repair is based on reliable data gained from a fundamental damage‐analysis carried out by an expert. In case of chloride‐induced corrosion of the reinforcement the procedure of testing and repair methods are well known and established. That is not the case for damages at sewage plants caused by sulphuric acid. This report shows how the combination of classical chemical analysis, measuring the tensile strength of the concrete surface and a modern method like Laser‐induced Breakdown Spectroscopy (LIBS) provides reliable data for a further concrete repair pointing out the possibilities but also limits of these methods.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces with a Recurrent Neural Network

A recurrent neural network (RNN) called CrackNet‐R is proposed in the article for fully automated pixel‐level crack detection on three‐dimensional (3D) asphalt pavement surfaces. In the article, a new recurrent unit, gated recurrent multilayer perceptron (GRMLP), is proposed to recursively update the internal memory of CrackNet‐R. Unlike the widely used long short‐term memory (LSTM) and gated recurrent unit (GRU), GRMLP is intended for deeper abstractions on the inputs and hidden states by conducting multilayer nonlinear transforms at gating units. CrackNet‐R implements a two‐phase sequence processing: sequence generation and sequence modeling. Sequence generation is specifically developed in the study to find the best local paths that are most likely to form crack patterns. Sequence modeling predicts timely probabilities of the input sequence being a crack pattern. In terms of sequence modeling, GRMLP slightly outperforms LSTM and GRU by using only one more nonlinear layer at each gate. In addition to sequence processing, an output layer is proposed to produce pixel probabilities based on timely probabilities predicted for sequences. The proposed output layer is critical for pixel‐perfect accuracy, as it accomplishes the transition from sequence‐level learning to pixel‐level learning. Using 3,000 diverse 3D images, the training of CrackNet‐R is completed through optimizing sequence modeling, sequence generation, and the output layer serially. The experiment using 500 testing pavement images shows that CrackNet‐R can achieve high Precision (88.89%), Recall (95.00%), and F‐measure (91.84%) simultaneously. Compared with the original CrackNet, CrackNet‐R is about four times faster and introduces tangible improvements in detection accuracy.     

Use of Ritz method for damage detection of reinforced and post-tensioned concrete beams

Damage-induced changes in modal characteristics can be detected using experimental modal analysis. In this article, based on changes in natural frequency, mode shapes, and damping ratios, a methodology for detecting damage location and severity is presented. The damage was induced by application of point load at half span location on the reinforced and post-tensioned concrete beams. The load was gradually increased to obtain different crack patterns to be used in simulation of damage scenarios. Experimental modal analysis was performed on the undamaged and damaged beams. The natural frequency and mode shapes were used to determine the location of damage. The approach is developed at an element level with a conventional finite element (FE) model by Ritz method, which is called Ritz damage detection method (RDDM). The mathematical model for both damped and undamped damaged structures have been established through the eigenvalue equations. The singular value decomposition (SVD) technique is used for determination of damage or sound index. These indexes are sensitive to the change of dynamic characteristics due to damages. This approach is applied to five simply supported post-tensioned concrete beams. The numerical results show that the exact location and severity of damage for different simulated damage scenarios could be efficiency found by the present methodology.