Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.     

Concrete crack detection using context‐aware deep semantic segmentation network

Computer‐vision and deep‐learning techniques are being increasingly applied to inspect, monitor, and assess infrastructure conditions including detection of cracks. Traditional vision‐based methods to detect cracks lack accuracy and generalization to work on complicated infrastructural conditions. This paper presents a novel context‐aware deep convolutional semantic segmentation network to effectively detect cracks in structural infrastructure under various conditions. The proposed method applies a pixel‐wise deep semantic segmentation network to segment the cracks on images with arbitrary sizes without retraining the prediction network. Meanwhile, a context‐aware fusion algorithm that leverages local cross‐state and cross‐space constraints is proposed to fuse the predictions of image patches. This method is evaluated on three datasets: CrackForest Dataset (CFD) and Tomorrows Road Infrastructure Monitoring, Management Dataset (TRIMMD) and a Customized Field Test Dataset (CFTD) and achieves Boundary F1 (BF) score of 0.8234, 0.8252, and 0.7937 under 2‐pixel error tolerance margin in CFD, TRIMMD, and CFTD, respectively. The proposed method advances the state‐of‐the‐art performance of BF score by approximately 2.71% in CFD, 1.47% in TRIMMD, and 4.14% in CFTD. Moreover, the averaged processing time of the proposed system is 0.7 s on a typical desktop with Intel^® Quad‐Core? i7‐7700 CPU@3.6 GHz Processor, 16GB RAM and NVIDIA GeForce GTX 1060 6GB GPU for an image of size 256 × 256 pixels.     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system

Crack information provides important evidence of structural degradation and safety in civil structures. Existing inspection methods are inefficient and difficult to rapidly deploy. A real‐time crack inspection method is proposed in this study to address this difficulty. Within this method, a wall‐climbing unmanned aerial system (UAS) is developed to acquire detailed crack images without distortion, then a wireless data transmission method is applied to fulfill real‐time detection requirements, allowing smartphones to receive real‐time video taken from the UAS. Next, an image data set including 1,330 crack images taken by the wall‐climbing UAS is established and used for training a deep‐learning model. For increasing detection speed, state‐of‐the‐art convolutional neural networks (CNNs) are compared and employed to train the crack detector; the selected model is transplanted into an android application so that the detection of cracks can be undertaken on a smartphone in real time. Following this, images with cracks are separated and crack width is calculated using an image processing method. The proposed method is then applied to a building where crack information is acquired and calculated accurately with high efficiency, thus verifying the practicability of the proposed method and system.     

Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning

Although crack inspection is a routine practice in civil infrastructure management (especially for highway bridge structures), it is time‐consuming and safety‐concerning to trained engineers and costly to the stakeholders. To automate this in the near future, the algorithmic challenge at the onset is to detect and localize cracks in imagery data with complex scenes. The rise of deep learning (DL) sheds light on overcoming this challenge through learning from imagery big data. However, how to exploit DL techniques is yet to be fully explored. One primary component of practical crack inspection is that it is not merely detection via visual recognition. To evaluate the potential risk of structural failure, it entails quantitative characterization, which usually includes crack width measurement. To further facilitate the automation of machine‐vision‐based concrete crack inspection, this article proposes a DL‐enabled quantitative crack width measurement method. In the detection and mapping phase, dual‐scale convolutional neural networks are designed to detect cracks in complex scene images with validated high accuracy. Subsequently, a novel crack width estimation method based on the use of Zernike moment operator is further developed for thin cracks. The experimental results based on a laboratory loading test agree well with the direct measurements, which substantiates the effectiveness of the proposed method for quantitative crack detection.     

A graph deep learning method for short‐term traffic forecasting on large road networks

Short‐term traffic flow prediction on a large‐scale road network is challenging due to the complex spatial–temporal dependencies, the directed network topology, and the high computational cost. To address the challenges, this article develops a graph deep learning framework to predict large‐scale network traffic flow with high accuracy and efficiency. Specifically, we model the dynamics of the traffic flow on a road network as an irreducible and aperiodic Markov chain on a directed graph. Based on the representation, a novel spatial–temporal graph inception residual network (STGI‐ResNet) is developed for network‐based traffic prediction. This model integrates multiple spatial–temporal graph convolution (STGC) operators, residual learning, and the inception structure. The proposed STGC operators can adaptively extract spatial–temporal features from multiple traffic periodicities while preserving the topology information of the road network. The proposed STGI‐ResNet inherits the advantages of residual learning and inception structure to improve prediction accuracy, accelerate the model training process, and reduce difficult parameter tuning efforts. The computational complexity is linearly related to the number of road links, which enables citywide short‐term traffic prediction. Experiments using a car‐hailing traffic data set at 10‐, 30‐, and 60‐min intervals for a large road network in a Chinese city shows that the proposed model outperformed various state‐of‐the‐art baselines for short‐term network traffic flow prediction.     

Pixel-level thin crack detection on road surface using convolutional neural network for severely imbalanced data

Because roads are the major backbone of the transportation network, research about crack detection on road surfaces has been popular in computer and infrastructure engineering. When training a convolutional neural network (CNN) for pixel-level road crack detection, three common challenges include (1) the data are severely imbalanced, (2) crack pixels can be easily confused with normal road texture and other visual noises, and (3) there are many unexplainable characteristics regarding the CNN itself. When it comes to very fine and thin cracks, these challenges are exaggerated and a new challenge is introduced, as there can be a discrepancy between the actual width and the annotated width of a crack. To tackle all these challenges of thin crack detection, this paper proposes a new variant of CNN named ThinCrack U-Net, designed to provide thin results upon pixel-level crack detection on road surfaces. The main contribution is to demystify how pixel-level thin crack detection results are affected by different loss functions as well as various combinations of the U-Net components. The experimental results show that ThinCrack U-Net yields a significant performance boost in CrackTree260, from 65.71% to 94.48% F-measure, compared to the existing variant of U-Net previously proposed in the context of pixel-level thin crack detection. Finally, this paper locates the source of undesirable result thickness and solves it with the balanced usage of downsampling/upsampling layers and atrous convolution. Unlike suggested by previous works, different loss functions show no significant impact on ThinCrack U-Net, whereas normalization layers are proved crucial in pixel-level thin crack detection.     

Modeling urban growth using video prediction technology: A time‐dependent convolutional encoder–decoder architecture

This paper presents a novel methodology for urban growth prediction using a machine learning approach. The methodology treats successive historical satellite images of an urban area as a video for which future frames are predicted. It adopts a time‐dependent convolutional encoder–decoder architecture. The methodology's input includes a satellite image for the base year and the prediction horizon. It constructs an image that predicts the growth of the urban area for any given target year within the specified horizon. A sensitivity analysis is performed to determine the best combination of parameters to achieve the highest prediction performance. As a case study, the methodology is applied to predict the urban growth pattern for the Dallas–Fort Worth area in Texas, with focus on two of its counties that observed significant growth over the past decade. The methodology is shown to produce results that are consistent with other growth prediction studies conducted for the areas.     

Vibration‐based semantic damage segmentation for large‐scale structural health monitoring

Toward reduced recovery time after extreme events, near real‐time damage diagnosis of structures is critical to provide reliable information. For this task, a fully convolutional encoder–decoder neural network is developed, which considers the spatial correlation of sensors in the automatic feature extraction process through a grid environment. A cost‐sensitive score function is designed to include the consequences of misclassification in the framework while considering the ground motion uncertainty in training. A 10‐story‐10‐bay reinforced concrete (RC) moment frame is modeled to present the design process of the deep learning architecture. The proposed models achieve global testing accuracies of 96.3% to locate damage and 93.2% to classify 16 damage mechanisms. Moreover, to handle class imbalance, three strategies are investigated enabling an increase of 16.2% regarding the mean damage class accuracy. To evaluate the generalization capacities of the framework, the classifiers are tested on 1,080 different RC frames by varying model properties. With less than a 2% reduction in global accuracy, the data‐driven model is shown to be reliable for the damage diagnosis of different frames. Given the robustness and capabilities of the grid environment, the proposed framework is applicable to different domains of structural health monitoring research and practice to obtain reliable information.     

Structure‐aware 3D reconstruction for cable‐stayed bridges: A learning‐based method

A powerful deep learning‐based three‐dimensional (3D) reconstruction method for reconstructing structure‐aware semantic 3D models of cable‐stayed bridges is proposed herein. Typically, conventional bridge semantic 3D model reconstruction methods are not robust when low‐quality point clouds are used. Furthermore, they are suited particularly for their respective fields and less generalized for cable‐stayed bridges. Hence, a structure‐aware learning‐based cable‐stayed bridge 3D reconstruction framework is proposed. The encoder part of the network uses both multiview images and a photogrammetric point cloud as input, whereas the decoder part uses a recursive binary tree network to model a high‐level structural relation graph and low‐level 3D geometric shapes. Two actual cable‐stayed bridges are employed as examples to evaluate the proposed method. Test results demonstrate that the proposed method successfully reconstructs the bridge model with structural components and their relations. Quantitative results indicate that the predicted models achieved an average F₁ score of 99.01%, a Chamfer distance of 0.0259, and a mesh‐to‐cloud distance of 1.78 m. The achieved result is similar to that obtained using the manual reconstruction approach in terms of component‐wise accuracy, and it is considerably better than that obtained using the manual approach in terms of spatial accuracy. In addition, the proposed recursive binary tree network is robust to noise and partial scans. The potential applications of the obtained 3D bridge models are discussed.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces with a Recurrent Neural Network

A recurrent neural network (RNN) called CrackNet‐R is proposed in the article for fully automated pixel‐level crack detection on three‐dimensional (3D) asphalt pavement surfaces. In the article, a new recurrent unit, gated recurrent multilayer perceptron (GRMLP), is proposed to recursively update the internal memory of CrackNet‐R. Unlike the widely used long short‐term memory (LSTM) and gated recurrent unit (GRU), GRMLP is intended for deeper abstractions on the inputs and hidden states by conducting multilayer nonlinear transforms at gating units. CrackNet‐R implements a two‐phase sequence processing: sequence generation and sequence modeling. Sequence generation is specifically developed in the study to find the best local paths that are most likely to form crack patterns. Sequence modeling predicts timely probabilities of the input sequence being a crack pattern. In terms of sequence modeling, GRMLP slightly outperforms LSTM and GRU by using only one more nonlinear layer at each gate. In addition to sequence processing, an output layer is proposed to produce pixel probabilities based on timely probabilities predicted for sequences. The proposed output layer is critical for pixel‐perfect accuracy, as it accomplishes the transition from sequence‐level learning to pixel‐level learning. Using 3,000 diverse 3D images, the training of CrackNet‐R is completed through optimizing sequence modeling, sequence generation, and the output layer serially. The experiment using 500 testing pavement images shows that CrackNet‐R can achieve high Precision (88.89%), Recall (95.00%), and F‐measure (91.84%) simultaneously. Compared with the original CrackNet, CrackNet‐R is about four times faster and introduces tangible improvements in detection accuracy.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network

Deep learning‐based structural damage detection methods overcome the limitation of inferior adaptability caused by extensively varying real‐world situations (e.g., lighting and shadow changes). However, most deep learning‐based methods detect structural damage at the image level and grid‐cell level. To provide pixel‐level detection of multiple damages, a Fully Convolutional Network (FCN)‐based multiple damages detection method for concrete structure is proposed. To realize this method, a database of 2,750 images (with 504 × 376 pixels) including crack, spalling, efflorescence, and hole images in concrete structure is built, and the four damages included in those images are labeled manually. Then, the architecture of the FCN is modified, trained, validated, and tested using this database. A strategy of model‐based transfer learning is used to initialize the parameters of the FCN during the training process. The results show 98.61% pixel accuracy (PA), 91.59% mean pixel accuracy (MPA), 84.53% mean intersection over union (MIoU), and 97.34% frequency weighted intersection over union (FWIoU). Subsequently, the robustness and adaptability of the trained FCN model is tested and the damage is extracted, where damage areas are provided according to a calibrated relation between the ratio (the pixel area and true area of the detected object) and the distance from the smartphone to the concrete surface using a laser range finder. A comparative study is conducted to examine the performance of the proposed FCN‐based approach using a SegNet‐based method. The results show that the proposed method substantiates quite better performance and can indeed detect multiple concrete damages at the pixel level in realistic situations.     

Automatic detection method of cracks from concrete surface imagery using two‐step light gradient boosting machine

Automated crack detection based on image processing is widely used when inspecting concrete structures. The existing methods for crack detection are not yet accurate enough due to the difficulty and complexity of the problem; thus, more accurate and practical methods should be developed. This paper proposes an automated crack detection method based on image processing using the light gradient boosting machine (LightGBM), one of the supervised machine learning methods. In supervised machine learning, appropriate features should be identified to obtain accurate results. In crack detection, the pixel values of the target pixels and geometric features of the cracks that occur when they are connected linearly should be considered. This paper proposes a methodology for generating features based on pixel values and geometric shapes in two stages. The accuracy of the proposed methodology is investigated using photos of concrete structures with adverse conditions, such as shadows and dirt. The proposed methodology achieves an accuracy of 99.7%, sensitivity of 75.71%, specificity of 99.9%, precision of 68.2%, and an F‐measure of 0.6952. The experimental results demonstrate that the proposed method can detect cracks with higher performance than the pix2pix‐based approach. Furthermore, the training time is 7.7 times shorter than that of the XGBoost and 2.3 times shorter than that of the pix2pix. The experimental results demonstrate that the proposed method can detect cracks with high accuracy.     

A Computational Methodology for Assessing the Time‐Dependent Structural Performance of Electric Road Infrastructures

An infrastructure adapted to dynamic wireless recharging of electric vehicles is often referred to generically as Electric Road (“e‐road”). E‐roads are deemed to become essential components of future grid environments and smart city strategies. Several technologies already exist that propose different ways to integrate dynamic inductive charging systems within the infrastructure. One e‐road solution uses a very thin rail with box‐section made of fibre‐reinforced polymer, inside which an electric current flows producing a magnetic field. In spite of the great interest and research generated by recharging technologies, the structural problems of e‐roads, including vibrations and structural integrity in the short and/or long period, have received relatively little attention to date. This article presents a novel computational methodology for assessing the time‐dependent structural performance of e‐roads, including a recursive strategy for the estimation of the lifetime of surface layers. The article also reports some numerical findings about e‐roads that will drive further numerical analyses and experimental studies on this novel type of infrastructure. Finally, numerical simulations have been conducted to compare an e‐road with a traditional road (“t‐road”), in terms of static, dynamic and fatigue behavior.     

Automated crack evaluation of a high‐rise bridge pier using a ring‐type climbing robot

This article proposes a deep learning‐based automated crack evaluation technique for a high‐rise bridge pier using a ring‐type climbing robot. First, a ring‐type climbing robot system composed of multiple vision cameras, climbing robot, and control computer is developed. By spatially moving the climbing robot system along a target bridge pier with close‐up scanning condition, high‐quality raw vision images are continuously obtained. The raw vision images are then processed through feature control‐based image stitching, deep learning‐based semantic segmentation, and Euclidean distance transform–based crack quantification algorithms. Finally, a digital crack map on the region of interest (ROI) of the target bridge pier can be automatically established. The proposed technique is experimentally validated using in situ test data obtained from Jang–Duck bridge in South Korea. The test results reveal that the proposed technique successfully evaluates cracks on the entire ROI of the bridge pier with precision of 90.92% and recall of 97.47%.     

A Pavement Crack Detection Method Combining 2D with 3D Information Based on Dempster‐Shafer Theory

Pavement cracking is one of the main distresses presented in the road surface. Objective and accurate detection or evaluation for these cracks is an important task in the pavement maintenance and management. In this work, a new pavement crack detection method is proposed by combining two‐dimensional (2D) gray‐scale images and three‐dimensional (3D) laser scanning data based on Dempster‐Shafer (D‐S) theory. In this proposed method, 2D gray‐scale image and 3D laser scanning data are modeled as a mass function in evidence theory, and 2D and 3D detection results for pavement cracks are fused at decision‐making level. The experimental results show that the proposed method takes advantage of the respective merits of 2D images and 3D laser scanning data and therefore improves the pavement crack detection accuracy and reduces recognition error rate compared to 2D image intensity‐based methods.     

Convolutional sparse coding‐based deep random vector functional link network for distress classification of road structures

This paper presents a convolutional sparse coding (CSC)‐based deep random vector functional link network (CSDRN) for distress classification of road structures. The main contribution of this paper is the introduction of CSC into a feature extraction scheme in the distress classification. CSC can extract visual features representing characteristics of target images because it can successfully estimate optimal convolutional dictionary filters and sparse features as visual features by training from a small number of distress images. The optimal dictionaries trained from distress images have basic components of visual characteristics such as edge and line information of distress images. Furthermore, sparse feature maps estimated on the basis of the dictionaries represent both strength of the basic components and location information of regions having their components, and these maps can represent distress images. That is, sparse feature maps can extract key components from distress images that have diverse visual characteristics. Therefore, CSC‐based feature extraction is effective for training from a limited number of distress images that have diverse visual characteristics. The construction of a novel neural network, CSDRN, by the use of a combination of CSC‐based feature extraction and the DRN classifier, which can also be trained from a small dataset, is shown in this paper. Accurate distress classification is realized via the CSDRN.     

A unified convolutional neural network integrated with conditional random field for pipe defect segmentation

Semantic segmentation of closed‐circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel on the image, from which defect types, locations, and geometric information can be obtained. In this study, a unified neural network, namely DilaSeg‐CRF, is proposed by fully integrating a deep convolutional neural network (CNN) with dense conditional random field (CRF) for improving the segmentation accuracy. First, DilaSeg is constructed with dilated convolution and multiscale techniques for producing feature maps with high resolution. The steps of the dense CRF inference algorithm are converted into CNN operations, which are then formulated as recurrent neural network (RNN) layers. The DilaSeg‐CRF is proposed by integrating DilaSeg with the RNN layers. Images containing three common types of sewer defects are collected from CCTV inspection videos and are annotated with ground truth labels, after which the proposed models are trained and evaluated. Experiments demonstrate that the end‐to‐end trainable DilaSeg‐CRF can improve the segmentation significantly, with an increase of 32% and 20% in mean intersection over union (mIoU) values compared with fully convolutional network (FCN‐8s) and DilaSeg, respectively. Our proposed DilaSeg‐CRF also achieves faster inference speed than FCN and eliminates the manual postprocessing for refining the segmentation results.