Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images

Safety has been a concern for the construction industry for decades. Unsafe conditions and behaviors are considered as the major causes of construction accidents. The current safety inspection of conditions and behaviors heavily rely on human efforts which are limited onsite. To improve the safety performance of the industry, a more efficient approach to identify the unsafe conditions on site is required to supplement the current manual inspection practice. A promising way to supplement the current manual safety inspection is automated and intelligent monitoring/inspection through information and sensing technologies, including localization techniques, environment monitoring, image processing and etc. To assess the potential benefits of contemporary technologies for onsite safety inspection, the authors focused on real-time guardrail detection, as unprotected edges are the ones cause for workers falling from heights.In this paper, the authors developed a safety guardrail detection model based on convolutional neural network (CNN). An augmented data set is generated with the addition of background image to guardrail 3D models and used as training set. Transfer learning is utilized and the Visual Geometry Group architecture with 16 layers (VGG-16) model is adopted to construct the basic features extraction for the neural network. In the CNN implementation, 4000 augmented images were used to train the proposed model, while another 2000 images collected from real construction jobsites and 2000 images from Google were used to validate the proposed model. The proposed CNN-based guardrail detection model obtained a high accuracy of 96.5%. In addition, this study indicates that the synthetic images generated by augment technology can be used to create a large training dataset, and CNN-based image detection algorithm is a promising approach in construction jobsite safety monitoring.     

Image‐based crack assessment of bridge piers using unmanned aerial vehicles and three‐dimensional scene reconstruction

Crack assessment of bridge piers using unmanned aerial vehicles (UAVs) eliminates unsafe factors of manual inspection and provides a potential way for the maintenance of transportation infrastructures. However, the implementation of UAV‐based crack assessment for real bridge piers is hindered by several key issues, including the following: (a) both perspective distortion and the geometry distortion by nonflat structural surfaces usually appear on crack images taken by the UAV system from the pier surface; however, these two kinds of distortions are difficult to correct at the same time; and (b) the crack image taken by a close‐range inspection flight UAV system is partially imaged, containing only a small part of the entire surface of the pier, and thereby hinders crack localization. In this paper, a new image‐based crack assessment methodology for bridge piers using UAV and three‐dimensional (3D) scene reconstruction is proposed. First, the data acquisition of UAV‐based crack assessment is discussed, and the UAV flight path and photography strategy for bridge pier assessment are proposed. Second, image‐based crack detection and 3D reconstruction are conducted to obtain crack width feature pair sequences and 3D surface models, respectively. Third, a new method of projecting cracks onto a meshed 3D surface triangular model is proposed, which can correct both the perspective distortion and geometry distortion by nonflat structural surfaces, and realize the crack localization. Field test investigations of crack assessment of a real bridge pier using a UAV are carried out for illustration, validation, and error analysis of the proposed methodology.     

Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization

Many bridge structures, one of the most critical components in transportation infrastructure systems, exhibit signs of deteriorations and are approaching or beyond the initial design service life. Therefore, structural health inspections of these bridges are becoming critically important, especially after extreme events. To enhance the efficiency of such an inspection, in recent years, autonomous damage detection based on computer vision has become a research hotspot. This article proposes a three‐level image‐based approach for post‐disaster inspection of the reinforced concrete bridge using deep learning with novel training strategies. The convolutional neural network for image classification, object detection, and semantic segmentation are, respectively, proposed to conduct system‐level failure classification, component‐level bridge column detection, and local damage‐level damage localization. To enable efficient training and prediction using a small data set, the model robustness is a crucial aspect to be taken into account, generally through its hyperparameters’ selection. This article, based on Bayesian optimization, proposes a principled manner of such selection, with which very promising results (well over 90% accuracies) and robustness are observed on all three‐level deep learning models.     

Automated classification of civil structure defects based on convolutional neural network

Today, the most commonly used civil infrastructure inspection method is based on a visual assessment conducted by certified inspectors following prescribed protocols. However, the increase in aggressive environmental and load conditions, coupled with the achievement of many structures of the life-cycle end, has highlighted the need to automate damage identification and satisfy the number of structures that need to be inspected. To overcome this challenge, this paper presents a method for automating concrete damage classification using a deep convolutional neural network. The convolutional neural network was designed after an experimental investigation of a wide number of pretrained networks, applying the transfer-learning technique. Training and validation were conducted using a database built with 1352 images balanced between “undamaged”, “cracked”, and “delaminated” concrete surfaces. To increase the network robustness compared to images in real-world situations, different image configurations have been collected from the Internet and on-field bridge inspections. The GoogLeNet model, with the highest validation accuracy of approximately 94%, was selected as the most suitable network for concrete damage classification. The results confirm that the proposed model can correctly classify images from real concrete surfaces of bridges, tunnels, and pavement, resulting in an effective alternative to the current visual inspection techniques.     

Active learning structural model updating of a multisensory system based on Kriging method and Bayesian inference

Model updating techniques are often applied to calibrate the numerical models of bridges using structural health monitoring data. The updated models can facilitate damage assessment and prediction of responses under extreme loading conditions. Some researchers have adopted surrogate models, for example, Kriging approach, to reduce the computations, while others have quantified uncertainties with Bayesian inference. It is desirable to further improve the efficiency and robustness of the Kriging-based model updating approach and analytically evaluate its uncertainties. An active learning structural model updating method is proposed based on the Kriging method. The expected feasibility learning function is extended for model updating using a Bayesian objective function. The uncertainties can be quantified through a derived likelihood function. The case study for verification involves a multisensory vehicle-bridge system comprising only two sensors, with one installed on a vehicle parked temporarily on the bridge and another mounted directly on the bridge. The proposed algorithm is utilized for damage detection of two beams numerically and an aluminum model beam experimentally. The proposed method can achieve satisfactory accuracy in identifying damage with much less data, compared with the general Kriging model updating technique. Both the computation and instrumentation can be reduced for structural health monitoring and model updating.     

A Test Method for Damage Diagnosis of Suspension Bridge Suspender Cables

Suspender cables are one of the most vulnerable components of a suspension bridge; therefore, development of effective methods for damage detection is imperative. Many previous damage detection methods require an accurate finite element model, which is often difficult to obtain. This article proposes a model‐free test method for damage diagnosis of suspender cables that avoids this problem. The method includes two procedures: the mean normalized curvature difference procedure and the curvature difference probability procedure. Numerical results for single and multiple damage cases indicate that: (1) both procedures can be effective for damage diagnosis of suspender cables; (2) small damage can be more easily diagnosed in long suspender cables than short ones, for example, 5% stiffness reduction in long suspender cables can be diagnosed; and (3) noise is generally not a problem, because the signal‐to‐noise ratio can be improved by increasing the pulse excitation magnitude for a suspender cable. The proposed test method does not eliminate the need for manual inspection, but changes it from observation to a more quantified method. All of these points increase the potential of the proposed method for practical applications.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system

Crack information provides important evidence of structural degradation and safety in civil structures. Existing inspection methods are inefficient and difficult to rapidly deploy. A real‐time crack inspection method is proposed in this study to address this difficulty. Within this method, a wall‐climbing unmanned aerial system (UAS) is developed to acquire detailed crack images without distortion, then a wireless data transmission method is applied to fulfill real‐time detection requirements, allowing smartphones to receive real‐time video taken from the UAS. Next, an image data set including 1,330 crack images taken by the wall‐climbing UAS is established and used for training a deep‐learning model. For increasing detection speed, state‐of‐the‐art convolutional neural networks (CNNs) are compared and employed to train the crack detector; the selected model is transplanted into an android application so that the detection of cracks can be undertaken on a smartphone in real time. Following this, images with cracks are separated and crack width is calculated using an image processing method. The proposed method is then applied to a building where crack information is acquired and calculated accurately with high efficiency, thus verifying the practicability of the proposed method and system.     

Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques

Sanitary sewer systems are designed to collect and transport sanitary wastewater and stormwater. Pipe inspection is important in identifying both the type and location of pipe defects to maintain the normal sewer operations. Closed-circuit television (CCTV) has been commonly utilized for sewer pipe inspection. Currently, interpretation of the CCTV images is mostly conducted manually to identify the defect type and location, which is time-consuming, labor-intensive and inaccurate. Conventional computer vision techniques are explored for automated interpretation of CCTV images, but such process requires large amount of image pre-processing and the design of complex feature extractor for certain cases. In this study, an automated approach is developed for detecting sewer pipe defects based on a deep learning technique namely faster region-based convolutional neural network (faster R-CNN). The detection model is trained using 3000 images collected from CCTV inspection videos of sewer pipes. After training, the model is evaluated in terms of detection accuracy and computation cost using mean average precision (mAP), missing rate, detection speed and training time. The proposed approach is demonstrated to be applicable for detecting sewer pipe defects accurately with high accuracy and fast speed. In addition, a new model is constructed and several hyper-parameters are adjusted to study the influential factors of the proposed approach. The experiment results demonstrate that dataset size, initialization network type and training mode, and network hyper-parameters have influence on model performance. Specifically, the increase of dataset size and convolutional layers can improve the model accuracy. The adjustment of hyper-parameters such as filter dimensions or stride values contributes to higher detection accuracy, achieving an mAP of 83%. The study lays the foundation for applying deep learning techniques in sewer pipe defect detection as well as addressing similar issues for construction and facility management.     

Applicability of machine learning to a crack model in concrete bridges

The growing demand for a more efficient maintenance of concrete bridges requires a model that tracks the deterioration of each bridge based on inspection data. Although it has been expected that machine learning could be applied to this problem, inspection data sparsely distributed over time are not suitable for machine learning in contrast to the continuous big data usually targeted. This study applies machine learning to a regression model of crack formation and propagation using inspection data to confirm the applicability. It includes the selection of the optimal algorithm, development of the model based on a novel methodology, and factor analysis using the model. Accordingly, the model was constructed by Gaussian process regression and it could appropriately extract the differences in the progress of crack damage due to multiple influential factors. The results demonstrate the excellent applicability of machine learning even to sparse data.     

Structure‐aware 3D reconstruction for cable‐stayed bridges: A learning‐based method

A powerful deep learning‐based three‐dimensional (3D) reconstruction method for reconstructing structure‐aware semantic 3D models of cable‐stayed bridges is proposed herein. Typically, conventional bridge semantic 3D model reconstruction methods are not robust when low‐quality point clouds are used. Furthermore, they are suited particularly for their respective fields and less generalized for cable‐stayed bridges. Hence, a structure‐aware learning‐based cable‐stayed bridge 3D reconstruction framework is proposed. The encoder part of the network uses both multiview images and a photogrammetric point cloud as input, whereas the decoder part uses a recursive binary tree network to model a high‐level structural relation graph and low‐level 3D geometric shapes. Two actual cable‐stayed bridges are employed as examples to evaluate the proposed method. Test results demonstrate that the proposed method successfully reconstructs the bridge model with structural components and their relations. Quantitative results indicate that the predicted models achieved an average F₁ score of 99.01%, a Chamfer distance of 0.0259, and a mesh‐to‐cloud distance of 1.78 m. The achieved result is similar to that obtained using the manual reconstruction approach in terms of component‐wise accuracy, and it is considerably better than that obtained using the manual approach in terms of spatial accuracy. In addition, the proposed recursive binary tree network is robust to noise and partial scans. The potential applications of the obtained 3D bridge models are discussed.     

Unmanned aerial vehicle inspection of the Placer River Trail Bridge through image-based 3D modelling

Unmanned aerial vehicles (UAV) are now a viable option for augmenting bridge inspections. Utilising an integrated combination of a UAV and computer vision can decrease costs, expedite inspections and facilitate bridge access. Any such inspection must consider the design of the UAV, the choice of cameras, data acquisition, geometrical resolution, safety regulations and pilot protocols. The Placer River Trail Bridge in Alaska recently served as a test bed for a UAV inspection methodology that integrates these considerations. The end goal was to produce a three-dimensional (3D) model of the bridge using UAV-captured images and a hierarchical Dense Structure-from-Motion algorithm. To maximise the quality of the model and its benefits to inspectors, this goal guided UAV design and mission planning. The resulting inspection methodology integrates UAV design, data capture and data analysis together to provide an optimised 3D model. This model provides inspection documentation while enabling the monitoring of defects. The developed methodology is presented herein, as well as analyses of the 3D models. The results are compared against models generated through laser scanning. The findings demonstrate that the UAV inspection methodology provided superior 3D models with the accuracy to resolve defects and support the needs of infrastructure managers.     

Automatic multi-image stitching for concrete bridge inspection by combining point and line features

Most of the current techniques for concrete bridge inspection are based on human visual interpretation, which often is dangerous and time-consuming. To address this problem, we introduce in this paper a newly developed vehicle-based robot inspection system that can automatically capture thousands of bottom surface images with a group of high-resolution industrial cameras, which are then stitched into a single composite image. However, traditional image stitching methods generally fail with large drift due to the great number (more than 2000) and sparse texture of linearly distributed images in sequence. Therefore, a novel image stitching method was developed for our robot inspection system, which combines both the 2D image point features and the 3D line features to reduce the drift. First, the bottom surface images are arranged into different strips based on their acquisition order and rough poses, and images in a single strip are divided into several groups. Then, the proposed image stitching method is performed in a bottom-up way, as follows: 1) the images within a single group initially are aligned via their point and line features; 2) the groups within a single strip are then stitched together via a homographic refinement procedure; 3) the strips are aligned into a single composite image that completely covers the bottom surface of the bridge; and 4) after all the stitching procedure are complete, a multi-band blending algorithm is applied to generate the mosaicked panorama as seamlessly as possible. The experimental results on a set of representative images acquired from the bottom surfaces of a real bridge demonstrate the capabilities and the limitations of the proposed approach.     

Structural health monitoring of a cable-stayed bridge with Bayesian neural networks

In recent years, there has been an increasing interest in permanent observation of the dynamic behaviour of bridges for long-term monitoring purpose. This is due not only to the ageing of a lot of structures, but also for dealing with the increasing complexity of new bridges. The long-term monitoring of bridges produces a huge quantity of data that need to be effectively processed. For this purpose, there has been a growing interest on the application of soft computing methods. In particular, this work deals with the applicability of Bayesian neural networks for the identification of damage of a cable-stayed bridge. The selected structure is a real bridge proposed as benchmark problem by the Asian-Pacific Network of Centers for Research in Smart Structure Technology (ANCRiSST). They shared data coming from the long-term monitoring of the bridge with the structural health monitoring community in order to assess the current progress on damage detection and identification methods with a full-scale example. The data set includes vibration data before and after the bridge was damaged, so they are useful for testing new approaches for damage detection. In the first part of the paper, the Bayesian neural network model is discussed; then in the second part, a Bayesian neural network procedure for damage detection has been tested. The proposed method is able to detect anomalies on the behaviour of the structure, which can be related to the presence of damage. In order to obtain a confirmation of the obtained results, in the last part of the paper, they are compared with those obtained by using a traditional approach for vibration-based structural identification.     

Vision-aided three-dimensional damage quantification and finite element model geometric updating for reinforced concrete structures

This article presents a vision-aided framework to achieve three-dimensional (3D) concrete damage quantification and finite element (FE) model geometric updating for reinforced concrete structures. The framework can process images and point clouds to extract damage information and update it in an FE model. First, a mask region convolutional neural network was used to realize highly precise damage detection and segmentation based on images. Second, a 3D point cloud was adopted in conjunction with the processed images for 3D damage qualification. The model-updating method enables an FE model to delete concrete elements to update the variations in volume caused by structural damage. This framework supports interaction with mainstream FE software for further analysis. To demonstrate the efficiency of the proposed framework, it was used in an experiment on a reinforced-concrete shear wall.     

Experimental validation of a deep neural network—Sparse representation classification ensemble method

In the current study, a new pattern recognition‐based damage detection technique is developed using the frequency response function of the structure. Principal component analysis is employed as an authoritative feature extraction method for dimensional reduction of the measured frequency response function data and constructing distinct feature patterns. Subsequently, as a novel approach, an ensemble of 2 powerful classifiers containing deep neural networks and couple sparse coding classification is utilized for damage prediction of the structure because there is no individual optimal classifier for all the problems. Verification of the proposed method is evaluated by an aluminum beam experimental setup besides a numerical 3D finite element model of a truss bridge. Damage detection results elucidate that the ensemble method decisions are much more accurate compared with the individual classifier decision. The proposed ensemble method verifies to be a novel, robust, and powerful damage detection process.     

Autoencoders for unsupervised real-time bridge health assessment

Over the last decades, the rising number of aging infrastructures has progressively fueled much interest toward the field of structural health monitoring. Following the increasing popularity of artificial intelligence algorithms, an autoencoder-based damage detection technique within the context of unsupervised learning is proposed in this paper to provide support for practical engineering applications. The developed methodology uses the autoencoder to reconstruct raw acceleration sequences of user-defined length collected from a healthy structure. To quantify the errors between the original input and the reconstructed output, which may be representative of damage occurrence, two indexes of reconstruction loss are selected as damage-sensitive features. To support damage detection, a selected number of short-time sequences are finally grouped into a unique macrosequence. The novel procedure can effectively both work at the single sensor level, as well as combine the predictive models using an ensemble learning strategy. Avoiding system identification, results obtained in the Z24 bridge demonstrate that the proposed method is quite effective for local damage detection with limited computational effort and using a limited number of sensors, thereby suitable to be easily applicable in the context of real-time bridge assessment.     

Multi-scale fatigue damage prognosis for long-span steel bridges under vehicle loading

Fatigue damage prognosis for long-span steel bridges is of the utmost importance in bridge maintenance and management. In this study, a multi-scale fatigue damage prognosis algorithm is developed to calculate the trans-scale fatigue damage accumulation of newly-built long-span steel bridges under vehicle loading. The necessity and procedure of establishing a multi-scale finite element (FE) model of a newly-built long-span bridge for fatigue damage prognosis are first introduced. The future vehicle loading on the bridge is forecasted using the recorded weigh-in-motion (WIM) data and the agent-based traffic flow micro-simulation method. Then, the multi-scale fatigue damage prognosis algorithm is developed based on the multi-scale FE model and using the future vehicle loading. Finally, the proposed algorithm is applied to a newly-built long-span cable-stayed bridge for the time period from 2010 to 2020. The results show that the macro-scale fatigue damage accumulation and micro-scale short crack evolution of the critical components of the bridge can be simultaneously predicted and visualized. The proposed algorithm can be used as a numerical tool for fatigue damage prognosis of steel bridges where (or near where) WIM station is installed.     

Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning

Although crack inspection is a routine practice in civil infrastructure management (especially for highway bridge structures), it is time‐consuming and safety‐concerning to trained engineers and costly to the stakeholders. To automate this in the near future, the algorithmic challenge at the onset is to detect and localize cracks in imagery data with complex scenes. The rise of deep learning (DL) sheds light on overcoming this challenge through learning from imagery big data. However, how to exploit DL techniques is yet to be fully explored. One primary component of practical crack inspection is that it is not merely detection via visual recognition. To evaluate the potential risk of structural failure, it entails quantitative characterization, which usually includes crack width measurement. To further facilitate the automation of machine‐vision‐based concrete crack inspection, this article proposes a DL‐enabled quantitative crack width measurement method. In the detection and mapping phase, dual‐scale convolutional neural networks are designed to detect cracks in complex scene images with validated high accuracy. Subsequently, a novel crack width estimation method based on the use of Zernike moment operator is further developed for thin cracks. The experimental results based on a laboratory loading test agree well with the direct measurements, which substantiates the effectiveness of the proposed method for quantitative crack detection.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.