Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning

Although crack inspection is a routine practice in civil infrastructure management (especially for highway bridge structures), it is time‐consuming and safety‐concerning to trained engineers and costly to the stakeholders. To automate this in the near future, the algorithmic challenge at the onset is to detect and localize cracks in imagery data with complex scenes. The rise of deep learning (DL) sheds light on overcoming this challenge through learning from imagery big data. However, how to exploit DL techniques is yet to be fully explored. One primary component of practical crack inspection is that it is not merely detection via visual recognition. To evaluate the potential risk of structural failure, it entails quantitative characterization, which usually includes crack width measurement. To further facilitate the automation of machine‐vision‐based concrete crack inspection, this article proposes a DL‐enabled quantitative crack width measurement method. In the detection and mapping phase, dual‐scale convolutional neural networks are designed to detect cracks in complex scene images with validated high accuracy. Subsequently, a novel crack width estimation method based on the use of Zernike moment operator is further developed for thin cracks. The experimental results based on a laboratory loading test agree well with the direct measurements, which substantiates the effectiveness of the proposed method for quantitative crack detection.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

Vision‐based automated bridge component recognition with high‐level scene consistency

This research investigates vision‐based automated bridge component recognition, which is critical for automating visual inspection of bridges during initial response after earthquakes. Semantic segmentation algorithms with up to 45 convolutional layers are applied to recognize bridge components from images of complex scenes. One of the challenges in such scenarios is to get the recognition results consistent with high‐level scene structure using limited amount of training data. To impose the high‐level scene consistency, this research combines 10‐class scene classification and 5‐class bridge component classification. Three approaches are investigated to combine scene classification results into bridge component classification: (a) naïve configuration, (b) parallel configuration, and (c) sequential configuration of classifiers. The proposed approaches, sequential configuration in particular, are demonstrated to be effective in recognizing bridge components in complex scenes, showing less than 1% of accuracy loss from the naïve/parallel configuration for bridge images, and less than 1% false positives for the nonbridge images.     

Vibration‐based structural state identification by a 1‐dimensional convolutional neural network

Deep learning has ushered in many breakthroughs in vision‐based detection via convolutional neural networks (CNNs), but the vibration‐based structural damage detection by CNN remains being refined. Thus, this study proposes a simple one‐dimensional CNN that detects tiny local structural stiffness and mass changes, and validates the proposed CNN on actual structures. Three independent acceleration databases are established based on a T‐shaped steel beam, a short steel girder bridge (in test field), and a long steel girder bridge (in service). The raw acceleration data are not pre‐processed and are directly used as the training and validation data. The well‐trained CNN almost perfectly identifies the locations of small local changes in the structural mass and stiffness, demonstrating the high sensitivity of the proposed simple CNN to tiny structural state changes in actual structures. The convolutional kernels and outputs of the convolutional and max pooling layers are visualized and discussed as well.     

Automated crack evaluation of a high‐rise bridge pier using a ring‐type climbing robot

This article proposes a deep learning‐based automated crack evaluation technique for a high‐rise bridge pier using a ring‐type climbing robot. First, a ring‐type climbing robot system composed of multiple vision cameras, climbing robot, and control computer is developed. By spatially moving the climbing robot system along a target bridge pier with close‐up scanning condition, high‐quality raw vision images are continuously obtained. The raw vision images are then processed through feature control‐based image stitching, deep learning‐based semantic segmentation, and Euclidean distance transform–based crack quantification algorithms. Finally, a digital crack map on the region of interest (ROI) of the target bridge pier can be automatically established. The proposed technique is experimentally validated using in situ test data obtained from Jang–Duck bridge in South Korea. The test results reveal that the proposed technique successfully evaluates cracks on the entire ROI of the bridge pier with precision of 90.92% and recall of 97.47%.     

Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network

Deep learning‐based structural damage detection methods overcome the limitation of inferior adaptability caused by extensively varying real‐world situations (e.g., lighting and shadow changes). However, most deep learning‐based methods detect structural damage at the image level and grid‐cell level. To provide pixel‐level detection of multiple damages, a Fully Convolutional Network (FCN)‐based multiple damages detection method for concrete structure is proposed. To realize this method, a database of 2,750 images (with 504 × 376 pixels) including crack, spalling, efflorescence, and hole images in concrete structure is built, and the four damages included in those images are labeled manually. Then, the architecture of the FCN is modified, trained, validated, and tested using this database. A strategy of model‐based transfer learning is used to initialize the parameters of the FCN during the training process. The results show 98.61% pixel accuracy (PA), 91.59% mean pixel accuracy (MPA), 84.53% mean intersection over union (MIoU), and 97.34% frequency weighted intersection over union (FWIoU). Subsequently, the robustness and adaptability of the trained FCN model is tested and the damage is extracted, where damage areas are provided according to a calibrated relation between the ratio (the pixel area and true area of the detected object) and the distance from the smartphone to the concrete surface using a laser range finder. A comparative study is conducted to examine the performance of the proposed FCN‐based approach using a SegNet‐based method. The results show that the proposed method substantiates quite better performance and can indeed detect multiple concrete damages at the pixel level in realistic situations.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

Recording of bridge damage areas by 3D integration of multiple images and reduction of the variability in detected results

Machine learning models have been developed to perform damage detection using images to improve bridge inspection efficiency. However, in damage detection using images alone, the 3D coordinates of the damage cannot be recorded. Furthermore, the accuracy of the detection depends on the quality of the images. This paper proposes a method to integrate and record the damage detected from multiple images into a 3D model using deep learning to detect the damage from bridge images and structure from motion to identify the shooting position. The proposed method reduces the variability of the detection results between images and can assess the scale of damage or, conversely, where there is no damage and the extent of inspection omissions. The proposed method has been applied to a real bridge, and it has been shown that the actual damage locations can be recorded as a 3D model.     

A deep‐learning‐based computer vision solution for construction vehicle detection

This paper aims at providing researchers and engineering professionals from the first step of solution development to the last step of solution deployment with a practical and comprehensive deep‐learning‐based solution for detecting construction vehicles. This paper places particular focus on the often‐ignored last step of deployment. Our first phase of solution development involved data preparation, model selection, model training, and model validation. Given the necessarily small‐scale nature of construction vehicle image datasets, we propose as detection model an improved version of the single shot detector MobileNet, which is suitable for embedded devices. Our study's second phase comprised model optimization, application‐specific embedded system selection, economic analysis, and field implementation. Several embedded devices were proposed and compared. Results including a consistent above 90% mean average precision confirm the superior real‐time performance of our proposed solutions. Finally, the practical field implementation of our proposed solutions was investigated. This study validates the practicality of deep‐learning‐based object detection solutions for construction scenarios. Moreover, the detailed information provided by the current study can be employed for several purposes such as safety monitoring, productivity assessments, and managerial decision making.     

Autonomous UAVs for Structural Health Monitoring Using Deep Learning and an Ultrasonic Beacon System with Geo‐Tagging

Visual inspection has traditionally been used for structural health monitoring. However, assessments conducted by trained inspectors or using contact sensors on structures for monitoring are costly and inefficient because of the number of inspectors and sensors required. To date, data acquisition using unmanned aerial vehicles (UAVs) equipped with cameras has become popular, but UAVs require skilled pilots or a global positioning system (GPS) for autonomous flight. Unfortunately, GPS cannot be used by a UAV for autonomous flight near some parts of certain structures (e.g., beneath a bridge), but these are the critical locations that should be inspected to monitor and maintain structural health. To address this difficulty, this article proposes an autonomous UAV method using ultrasonic beacons to replace the role of GPS, a deep convolutional neural network (CNN) for damage detection, and a geo‐tagging method for the localization of damage. Concrete cracks, as an example of structural damage, were successfully detected with 97.7% specificity and 91.9% sensitivity, by processing video data collected from an autonomous UAV.     

Postdisaster image‐based damage detection and repair cost estimation of reinforced concrete buildings using dual convolutional neural networks

Reinforced concrete (RC) buildings are commonly used around the world. With recent earthquakes worldwide, rapid structural damage inspection and repair cost evaluation are crucial for building owners and policy makers to make informed risk management decisions. To improve the efficiency of such inspection, advanced computer vision techniques based on convolutional neural networks have been adopted in recent research to rapidly quantify the damage state (DS) of structures. In this article, an advanced object detection neural network, named YOLOv2, is implemented, which achieves 98.2% and 84.5% average precision in training and testing, respectively. The proposed YOLOv2 is used in combination with the classification neural network, which improves the identification accuracy for critical DS of RC structures by 7.5%. The improved classification procedures allow engineers to rapidly and more accurately quantify the DSs of the structure, and also localize the critical damage features. The identified DS can then be integrated with the state‐of‐the‐art performance evaluation framework to quantify the financial losses of critical RC buildings. The results can be used by the building owners and decision makers to make informed risk management decisions immediately after the strong earthquake shaking. Hence, resources can be allocated rapidly to improve the resiliency of the community.     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

Structure‐aware 3D reconstruction for cable‐stayed bridges: A learning‐based method

A powerful deep learning‐based three‐dimensional (3D) reconstruction method for reconstructing structure‐aware semantic 3D models of cable‐stayed bridges is proposed herein. Typically, conventional bridge semantic 3D model reconstruction methods are not robust when low‐quality point clouds are used. Furthermore, they are suited particularly for their respective fields and less generalized for cable‐stayed bridges. Hence, a structure‐aware learning‐based cable‐stayed bridge 3D reconstruction framework is proposed. The encoder part of the network uses both multiview images and a photogrammetric point cloud as input, whereas the decoder part uses a recursive binary tree network to model a high‐level structural relation graph and low‐level 3D geometric shapes. Two actual cable‐stayed bridges are employed as examples to evaluate the proposed method. Test results demonstrate that the proposed method successfully reconstructs the bridge model with structural components and their relations. Quantitative results indicate that the predicted models achieved an average F₁ score of 99.01%, a Chamfer distance of 0.0259, and a mesh‐to‐cloud distance of 1.78 m. The achieved result is similar to that obtained using the manual reconstruction approach in terms of component‐wise accuracy, and it is considerably better than that obtained using the manual approach in terms of spatial accuracy. In addition, the proposed recursive binary tree network is robust to noise and partial scans. The potential applications of the obtained 3D bridge models are discussed.     

Structural Damage Detection with Automatic Feature‐Extraction through Deep Learning

Structural damage detection is still a challenging problem owing to the difficulty of extracting damage‐sensitive and noise‐robust features from structure response. This article presents a novel damage detection approach to automatically extract features from low‐level sensor data through deep learning. A deep convolutional neural network is designed to learn features and identify damage locations, leading to an excellent localization accuracy on both noise‐free and noisy data set, in contrast to another detector using wavelet packet component energy as the input feature. Visualization of the features learned by hidden layers in the network is implemented to get a physical insight into how the network works. It is found the learned features evolve with the depth from rough filters to the concept of vibration mode, implying the good performance results from its ability to learn essential characteristics behind the data.     

Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system

Crack information provides important evidence of structural degradation and safety in civil structures. Existing inspection methods are inefficient and difficult to rapidly deploy. A real‐time crack inspection method is proposed in this study to address this difficulty. Within this method, a wall‐climbing unmanned aerial system (UAS) is developed to acquire detailed crack images without distortion, then a wireless data transmission method is applied to fulfill real‐time detection requirements, allowing smartphones to receive real‐time video taken from the UAS. Next, an image data set including 1,330 crack images taken by the wall‐climbing UAS is established and used for training a deep‐learning model. For increasing detection speed, state‐of‐the‐art convolutional neural networks (CNNs) are compared and employed to train the crack detector; the selected model is transplanted into an android application so that the detection of cracks can be undertaken on a smartphone in real time. Following this, images with cracks are separated and crack width is calculated using an image processing method. The proposed method is then applied to a building where crack information is acquired and calculated accurately with high efficiency, thus verifying the practicability of the proposed method and system.     

Image‐based crack assessment of bridge piers using unmanned aerial vehicles and three‐dimensional scene reconstruction

Crack assessment of bridge piers using unmanned aerial vehicles (UAVs) eliminates unsafe factors of manual inspection and provides a potential way for the maintenance of transportation infrastructures. However, the implementation of UAV‐based crack assessment for real bridge piers is hindered by several key issues, including the following: (a) both perspective distortion and the geometry distortion by nonflat structural surfaces usually appear on crack images taken by the UAV system from the pier surface; however, these two kinds of distortions are difficult to correct at the same time; and (b) the crack image taken by a close‐range inspection flight UAV system is partially imaged, containing only a small part of the entire surface of the pier, and thereby hinders crack localization. In this paper, a new image‐based crack assessment methodology for bridge piers using UAV and three‐dimensional (3D) scene reconstruction is proposed. First, the data acquisition of UAV‐based crack assessment is discussed, and the UAV flight path and photography strategy for bridge pier assessment are proposed. Second, image‐based crack detection and 3D reconstruction are conducted to obtain crack width feature pair sequences and 3D surface models, respectively. Third, a new method of projecting cracks onto a meshed 3D surface triangular model is proposed, which can correct both the perspective distortion and geometry distortion by nonflat structural surfaces, and realize the crack localization. Field test investigations of crack assessment of a real bridge pier using a UAV are carried out for illustration, validation, and error analysis of the proposed methodology.     

Deep learning-based automatic classification of three-level surface information in bridge inspection

Bridge inspection ensures that in-service bridges are managed and maintained in conformity. To enhance the accuracy and efficiency of bridge inspection, an automatic hierarchical model is proposed, which enables the classification and correlation of bridge surface images at three levels, namely, at the structure, component, and defect type level. Thus, the impact of both the defect types and the affected components on bridge safety can be simultaneously considered. The proposed model uses a group of sub-models instead of the common flat network to realize the multiple tasks, which is advantageous in accuracy, training simplicity, and scalability. The classification accuracy of the hierarchical model in three levels has reached 96%, 92%, and 81%. Results demonstrate the effectiveness of the proposed method in the classification of multi-scale targets. This study may provide a new strategy for developing a systematic and easily adaptable detection framework for practical bridge engineering.     

Vibration‐based semantic damage segmentation for large‐scale structural health monitoring

Toward reduced recovery time after extreme events, near real‐time damage diagnosis of structures is critical to provide reliable information. For this task, a fully convolutional encoder–decoder neural network is developed, which considers the spatial correlation of sensors in the automatic feature extraction process through a grid environment. A cost‐sensitive score function is designed to include the consequences of misclassification in the framework while considering the ground motion uncertainty in training. A 10‐story‐10‐bay reinforced concrete (RC) moment frame is modeled to present the design process of the deep learning architecture. The proposed models achieve global testing accuracies of 96.3% to locate damage and 93.2% to classify 16 damage mechanisms. Moreover, to handle class imbalance, three strategies are investigated enabling an increase of 16.2% regarding the mean damage class accuracy. To evaluate the generalization capacities of the framework, the classifiers are tested on 1,080 different RC frames by varying model properties. With less than a 2% reduction in global accuracy, the data‐driven model is shown to be reliable for the damage diagnosis of different frames. Given the robustness and capabilities of the grid environment, the proposed framework is applicable to different domains of structural health monitoring research and practice to obtain reliable information.     

Fractal Dimension‐Based Damage Detection Method for Beams with a Uniform Cross‐Section

Abstract: For civil structures, damage usually occurs in localized areas. As fractal dimension (FD) analysis can provide insight to local complexity in geometry, a damage detection approach based on Katz's estimation of the FD measure of displacement mode shape for homogeneous, uniform cross‐sectional beam structures is proposed in this study. An FD‐based index for damage localization (FDIDL) is developed utilizing the difference of angles of sliding windows between two successive points, which is expressed in FD. To improve robustness against noise, FDIDL is calculated using multisliding windows. The influence of the spatial sampling interval length and the number of 2‐sampling sliding windows on sensitivity to damage and robustness against noise is investigated. The relationship between the angle expressed in FD and the modal strain energy is established and thereby an FD‐based index for the estimation of damage extent (FDIDE) is presented. The two damage indices are applied to a simply supported beam to detect the simulated damage in the beam. The results indicate that the proposed FDIDL can locate the single or multiple damages, and FDIDE can reliably quantify the damage extent. The optimal spatial sampling interval and the number of sliding windows are investigated. Furthermore, the simulation with measurement noise is carried out to demonstrate the effectiveness and robustness of the two defined FD‐based damage indices. Finally, experiments are conducted on simply supported steel beams damaged at different locations. It is demonstrated that the proposed approach can locate the damages to a satisfactory precision.     

Automatic detection method of cracks from concrete surface imagery using two‐step light gradient boosting machine

Automated crack detection based on image processing is widely used when inspecting concrete structures. The existing methods for crack detection are not yet accurate enough due to the difficulty and complexity of the problem; thus, more accurate and practical methods should be developed. This paper proposes an automated crack detection method based on image processing using the light gradient boosting machine (LightGBM), one of the supervised machine learning methods. In supervised machine learning, appropriate features should be identified to obtain accurate results. In crack detection, the pixel values of the target pixels and geometric features of the cracks that occur when they are connected linearly should be considered. This paper proposes a methodology for generating features based on pixel values and geometric shapes in two stages. The accuracy of the proposed methodology is investigated using photos of concrete structures with adverse conditions, such as shadows and dirt. The proposed methodology achieves an accuracy of 99.7%, sensitivity of 75.71%, specificity of 99.9%, precision of 68.2%, and an F‐measure of 0.6952. The experimental results demonstrate that the proposed method can detect cracks with higher performance than the pix2pix‐based approach. Furthermore, the training time is 7.7 times shorter than that of the XGBoost and 2.3 times shorter than that of the pix2pix. The experimental results demonstrate that the proposed method can detect cracks with high accuracy.