A unified convolutional neural network integrated with conditional random field for pipe defect segmentation

Semantic segmentation of closed‐circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel on the image, from which defect types, locations, and geometric information can be obtained. In this study, a unified neural network, namely DilaSeg‐CRF, is proposed by fully integrating a deep convolutional neural network (CNN) with dense conditional random field (CRF) for improving the segmentation accuracy. First, DilaSeg is constructed with dilated convolution and multiscale techniques for producing feature maps with high resolution. The steps of the dense CRF inference algorithm are converted into CNN operations, which are then formulated as recurrent neural network (RNN) layers. The DilaSeg‐CRF is proposed by integrating DilaSeg with the RNN layers. Images containing three common types of sewer defects are collected from CCTV inspection videos and are annotated with ground truth labels, after which the proposed models are trained and evaluated. Experiments demonstrate that the end‐to‐end trainable DilaSeg‐CRF can improve the segmentation significantly, with an increase of 32% and 20% in mean intersection over union (mIoU) values compared with fully convolutional network (FCN‐8s) and DilaSeg, respectively. Our proposed DilaSeg‐CRF also achieves faster inference speed than FCN and eliminates the manual postprocessing for refining the segmentation results.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

Encoder–decoder network for pixel‐level road crack detection in black‐box images

Timely monitoring of pavement cracks is essential for successful maintenance of road infrastructure. Accurate information concerning crack location and severity enables proactive management of the infrastructure. Black‐box cameras, which are becoming increasingly widespread at an affordable price, can be used as efficient road‐image collectors over a wide area. However, the cracks in these images are difficult to detect, because the images containing them often include objects other than roads. Thus, we propose a pixel‐level detection method for identifying road cracks in black‐box images using a deep convolutional encoder–decoder network. The encoder consists of convolutional layers of the residual network for extracting crack features, and the decoder consists of deconvolutional layers for localizing the cracks in an input image. The proposed network was trained on 427 out of 527 images extracted from black‐box videos and tested on the remaining 100 images. Compared with VGG‐16, ResNet‐50, ResNet‐101, ResNet‐200 with transfer learning, and ResNet‐152 without transfer learning, ResNet‐152 with transfer learning exhibited the best performance, achieving recall, precision, and intersection of union of 71.98%, 77.68%, and 59.65%, respectively. The experimental results prove that the proposed method is optimal for detecting cracks in black‐box images at the pixel level.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.     

Evidential transformer for pavement distress segmentation

Distress segmentation assigns each pixel of a pavement image to one distress class or background, which provides a simplified representation for distress detection and measurement. Even though remarkably benefiting from deep learning, distress segmentation still faces the problems of poor calibration and multimodel fusion. This study has proposed a deep neural network by combining the Dempster–Shafer theory (DST) and a transformer network for pavement distress segmentation. The network, called the evidential segmentation transformer, uses its transformer backbone to obtain pixel-wise features from input images. The features are then converted into pixel-wise mass functions by a DST-based evidence layer. The pixel-wise masses are utilized for performing distress segmentation based on the pignistic criterion. The proposed network is iteratively trained by a new learning strategy, which represents uncertain information of ambiguous pixels by mass functions. In addition, an evidential fusion strategy is proposed to fuse heterogeneous transformers with different distress classes. Experiments using three public data sets (Pavementscape, Crack500, and CrackDataset) show that the proposed networks achieve state-of-the-art accuracy and calibration on distress segmentation, which allows for measuring the distress shapes more accurately and stably. The proposed fusion strategy combines heterogeneous transformers while remaining a performance not less than those of the individual networks on their respective data sets. Thus, the fusion strategy makes it possible to use the existing networks to build a more general and accurate one for distress segmentation.     

A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects

Tunnel lining defects are an important indicator reflecting the safety status of shield tunnels. Inspired by the state‐of‐the‐art deep learning, a method for automatic intelligent classification and detection methodology of tunnel lining defects is presented. A fully convolutional network (FCN) model for classification is proposed. Information about defects, collected using charge‐coupled device cameras, was used to train the model. The model's performance was compared to those of GoogLeNet and VGG. The best‐set accuracy of the proposed model was over 95% at a test‐time speed of 48 ms per image. For defects detection, image features were computed from large‐scale images by the FCN and then detected using a region proposal network and position‐sensitive region of interest pooling. Some indices (detection rate, detection accuracy, and detection efficiency, locating accuracy) were used to evaluate the model. The comparisons with faster R‐CNN and a traditional method were conducted. The results show that the model is very fast and efficient, allowing automatic intelligent classification and detection of tunnel lining defects.     

Concrete crack detection using context‐aware deep semantic segmentation network

Computer‐vision and deep‐learning techniques are being increasingly applied to inspect, monitor, and assess infrastructure conditions including detection of cracks. Traditional vision‐based methods to detect cracks lack accuracy and generalization to work on complicated infrastructural conditions. This paper presents a novel context‐aware deep convolutional semantic segmentation network to effectively detect cracks in structural infrastructure under various conditions. The proposed method applies a pixel‐wise deep semantic segmentation network to segment the cracks on images with arbitrary sizes without retraining the prediction network. Meanwhile, a context‐aware fusion algorithm that leverages local cross‐state and cross‐space constraints is proposed to fuse the predictions of image patches. This method is evaluated on three datasets: CrackForest Dataset (CFD) and Tomorrows Road Infrastructure Monitoring, Management Dataset (TRIMMD) and a Customized Field Test Dataset (CFTD) and achieves Boundary F1 (BF) score of 0.8234, 0.8252, and 0.7937 under 2‐pixel error tolerance margin in CFD, TRIMMD, and CFTD, respectively. The proposed method advances the state‐of‐the‐art performance of BF score by approximately 2.71% in CFD, 1.47% in TRIMMD, and 4.14% in CFTD. Moreover, the averaged processing time of the proposed system is 0.7 s on a typical desktop with Intel^® Quad‐Core? i7‐7700 CPU@3.6 GHz Processor, 16GB RAM and NVIDIA GeForce GTX 1060 6GB GPU for an image of size 256 × 256 pixels.     

Unified Vision‐Based Methodology for Simultaneous Concrete Defect Detection and Geolocalization

Vision‐based autonomous inspection of concrete surface defects is crucial for efficient maintenance and rehabilitation of infrastructures and has become a research hot spot. However, most existing vision‐based inspection methods mainly focus on detecting one kind of defect in nearly uniform testing background where defects are relatively large and easily recognizable. But in the real‐world scenarios, multiple types of defects often occur simultaneously. And most of them occupy only small fractions of inspection images and are swamped in cluttered background, which easily leads to missed and false detections. In addition, the majority of the previous researches only focus on detecting defects but few of them pay attention to the geolocalization problem, which is indispensable for timely performing repair, protection, or reinforcement works. And most of them rely heavily on GPS for tracking the locations of the defects. However, this method is sometimes unreliable within infrastructures where the GPS signals are easily blocked, which causes a dramatic increase in searching costs. To address these limitations, we present a unified and purely vision‐based method denoted as defects detection and localization network, which can detect and classify various typical types of defects under challenging conditions while simultaneously geolocating the defects without requiring external localization sensors. We design a supervised deep convolutional neural network and propose novel training methods to optimize its performance on specific tasks. Extensive experiments show that the proposed method is effective with a detection accuracy of 80.7% and a localization accuracy of 86% at 0.41 s per image (at a scale of 1,200 pixels in the field test experiment), which is ideal for integration within intelligent autonomous inspection systems to provide support for practical applications.     

A Pavement Crack Detection Method Combining 2D with 3D Information Based on Dempster‐Shafer Theory

Pavement cracking is one of the main distresses presented in the road surface. Objective and accurate detection or evaluation for these cracks is an important task in the pavement maintenance and management. In this work, a new pavement crack detection method is proposed by combining two‐dimensional (2D) gray‐scale images and three‐dimensional (3D) laser scanning data based on Dempster‐Shafer (D‐S) theory. In this proposed method, 2D gray‐scale image and 3D laser scanning data are modeled as a mass function in evidence theory, and 2D and 3D detection results for pavement cracks are fused at decision‐making level. The experimental results show that the proposed method takes advantage of the respective merits of 2D images and 3D laser scanning data and therefore improves the pavement crack detection accuracy and reduces recognition error rate compared to 2D image intensity‐based methods.     

A spatio‐temporal ensemble method for large‐scale traffic state prediction

How to effectively ensemble multiple models while leveraging the spatio‐temporal information is a challenging but practical problem. However, there is no existing ensemble method explicitly designed for spatio‐temporal data. In this paper, a fully convolutional model based on semantic segmentation technology is proposed, termed as spatio‐temporal ensemble net. The proposed method is suitable for grid‐based spatio‐temporal prediction in dense urban areas. Experiments demonstrate that through spatio‐temporal ensemble net, multiple traffic state prediction base models can be combined to improve the prediction accuracy.     

Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques

Sanitary sewer systems are designed to collect and transport sanitary wastewater and stormwater. Pipe inspection is important in identifying both the type and location of pipe defects to maintain the normal sewer operations. Closed-circuit television (CCTV) has been commonly utilized for sewer pipe inspection. Currently, interpretation of the CCTV images is mostly conducted manually to identify the defect type and location, which is time-consuming, labor-intensive and inaccurate. Conventional computer vision techniques are explored for automated interpretation of CCTV images, but such process requires large amount of image pre-processing and the design of complex feature extractor for certain cases. In this study, an automated approach is developed for detecting sewer pipe defects based on a deep learning technique namely faster region-based convolutional neural network (faster R-CNN). The detection model is trained using 3000 images collected from CCTV inspection videos of sewer pipes. After training, the model is evaluated in terms of detection accuracy and computation cost using mean average precision (mAP), missing rate, detection speed and training time. The proposed approach is demonstrated to be applicable for detecting sewer pipe defects accurately with high accuracy and fast speed. In addition, a new model is constructed and several hyper-parameters are adjusted to study the influential factors of the proposed approach. The experiment results demonstrate that dataset size, initialization network type and training mode, and network hyper-parameters have influence on model performance. Specifically, the increase of dataset size and convolutional layers can improve the model accuracy. The adjustment of hyper-parameters such as filter dimensions or stride values contributes to higher detection accuracy, achieving an mAP of 83%. The study lays the foundation for applying deep learning techniques in sewer pipe defect detection as well as addressing similar issues for construction and facility management.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces with a Recurrent Neural Network

A recurrent neural network (RNN) called CrackNet‐R is proposed in the article for fully automated pixel‐level crack detection on three‐dimensional (3D) asphalt pavement surfaces. In the article, a new recurrent unit, gated recurrent multilayer perceptron (GRMLP), is proposed to recursively update the internal memory of CrackNet‐R. Unlike the widely used long short‐term memory (LSTM) and gated recurrent unit (GRU), GRMLP is intended for deeper abstractions on the inputs and hidden states by conducting multilayer nonlinear transforms at gating units. CrackNet‐R implements a two‐phase sequence processing: sequence generation and sequence modeling. Sequence generation is specifically developed in the study to find the best local paths that are most likely to form crack patterns. Sequence modeling predicts timely probabilities of the input sequence being a crack pattern. In terms of sequence modeling, GRMLP slightly outperforms LSTM and GRU by using only one more nonlinear layer at each gate. In addition to sequence processing, an output layer is proposed to produce pixel probabilities based on timely probabilities predicted for sequences. The proposed output layer is critical for pixel‐perfect accuracy, as it accomplishes the transition from sequence‐level learning to pixel‐level learning. Using 3,000 diverse 3D images, the training of CrackNet‐R is completed through optimizing sequence modeling, sequence generation, and the output layer serially. The experiment using 500 testing pavement images shows that CrackNet‐R can achieve high Precision (88.89%), Recall (95.00%), and F‐measure (91.84%) simultaneously. Compared with the original CrackNet, CrackNet‐R is about four times faster and introduces tangible improvements in detection accuracy.     

基于声呐成像的水下桩墩表观病害深度学习与智能检测

声呐成像检测水下桩墩表观病害时，其图像与光学图像的病害特征存在较大差异，病害的位置和类型需要人工识别且易出错。为解决这个问题，提出基于声呐成像的水下桩墩表观病害深度学习与智能检测方法。首先对水下实桥桩墩以及试验模型进行声呐扫描获取大量图像，并分析声呐图像中的病害特征；然后对Faster R-CNN框架下VGG16网络模型进行改进，采用水平、垂直等线性变换实现原始声呐图像的数据增强，对深度学习模型进行近似联合优化训练，用一定概率保证率的矩形识别框实现水下桩墩多类病害的分类定位；最后选取150幅未参与训练的声呐图像进行识别，验证所提出方法的有效性，并通过混淆矩阵、精确率、召回率、准确率以及F1值等评价指标对识别方法性能进行研究。研究结果发现，桩墩孔洞、剥落和位移等病害以及无病害类型的识别结果的总体准确率为88.3%，F1值分别为90.1%、84.9%、78.7%和94.6%，平均F1值为87%。这说明该方法在水下桩墩表观病害识别、定位以及自动化处理方面是可行、有效的，为桥梁水下桩墩表观病害的图像处理、智能化检测与桥梁安全评估提供技术支撑。     

Optimal CNN-based semantic segmentation model of cutting slope images

This paper utilizes three popular semantic segmentation networks, specifically DeepLab v3+, fully convolutional network (FCN), and U-Net to qualitively analyze and identify the key components of cutting slope images in complex scenes and achieve rapid image-based slope detection. The elements of cutting slope images are divided into 7 categories. In order to determine the best algorithm for pixel level classification of cutting slope images, the networks are compared from three aspects: a) different neural networks, b) different feature extractors, and c) 2 different optimization algorithms. It is found that DeepLab v3+ with Resnet18 and Sgdm performs best, FCN 32s with Sgdm takes the second, and U-Net with Adam ranks third. This paper also analyzes the segmentation strategies of the three networks in terms of feature map visualization. Results show that the contour generated by DeepLab v3+ (combined with Resnet18 and Sgdm) is closest to the ground truth, while the resulting contour of U-Net (combined with Adam) is closest to the input images.     

Vision-aided three-dimensional damage quantification and finite element model geometric updating for reinforced concrete structures

This article presents a vision-aided framework to achieve three-dimensional (3D) concrete damage quantification and finite element (FE) model geometric updating for reinforced concrete structures. The framework can process images and point clouds to extract damage information and update it in an FE model. First, a mask region convolutional neural network was used to realize highly precise damage detection and segmentation based on images. Second, a 3D point cloud was adopted in conjunction with the processed images for 3D damage qualification. The model-updating method enables an FE model to delete concrete elements to update the variations in volume caused by structural damage. This framework supports interaction with mainstream FE software for further analysis. To demonstrate the efficiency of the proposed framework, it was used in an experiment on a reinforced-concrete shear wall.     

A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision

Spatiotemporal information of the vehicles on a bridge is important evidence for reflecting the stress state and traffic density of the bridge. A methodology for obtaining the information is proposed based on computer vision technology, which contains the detection by Faster region‐based convolutional neural network (Faster R‐CNN), multiple object tracking, and image calibration. For minimizing the detection time, the ZF (Zeiler & Fergus) model with five convolutional layers is selected as the shared part between Region Proposal Network and Fast R‐CNN in Faster R‐CNN. An image data set including 1,694 images is established about eight types of vehicles for training Faster R‐CNN. Combined with the detection of each frame of the video, the methods of multiple object tracking and image calibration are developed for acquiring the vehicle parameters, including the length, number of axles, speed, and the lane that the vehicle is in. The method of tracking is mainly based on the judgment of the distances between the vehicle bounding boxes in virtual detection region. As for image calibration, it is based on the moving standard vehicles whose lengths are known, which can be regarded as the 3D template to calculate the vehicle parameters. After acquiring the vehicles' parameters, the spatiotemporal information of the vehicles can be obtained. The proposed system has a frame rate of 16 fps and only needs two cameras as the input device. The system is successfully applied on a double tower cable‐stayed bridge, and the identification accuracies of the types and number of axles are about 90 and 73% in the virtual detection region, and the speed errors of most vehicles are less than 6%.     

Severe rail wear detection with rail running band images

Rail wear occurs continuously owing to the rolling contact load of trains and is fundamental for railway operational safety. A point-based manual rail wear inspection cannot satisfy the increasing demand for rapid, low-cost, and continuous monitoring. This paper proposes a depth-plus-region fusion network for detecting rail wear on a running band, which is a collection of wheel–rail interaction traces. The following steps are involved in the proposed method. (i) A depth map estimated by a modified MiDaS model is utilized as guidance for exploiting the depth information of the running band for rail wear detection. (ii) The running band of a rail is segmented and extracted from images using an improved mask region-based convolutional neural network that uses the scale and ratio information to perform instance segmentation of the running band images. (iii) A two-channel attention–fusion network that classifies rail wear is constructed. In this study, we collected real-world running band images and rail wear-related data to validate our approach using a high-accuracy rail-profile measurement tool. The case-study results demonstrated that the proposed method can rapidly and accurately detect rail wear under different ambient light conditions. Moreover, the recall rate of severe wear detection was 84.21%.     

Automated classification of civil structure defects based on convolutional neural network

Today, the most commonly used civil infrastructure inspection method is based on a visual assessment conducted by certified inspectors following prescribed protocols. However, the increase in aggressive environmental and load conditions, coupled with the achievement of many structures of the life-cycle end, has highlighted the need to automate damage identification and satisfy the number of structures that need to be inspected. To overcome this challenge, this paper presents a method for automating concrete damage classification using a deep convolutional neural network. The convolutional neural network was designed after an experimental investigation of a wide number of pretrained networks, applying the transfer-learning technique. Training and validation were conducted using a database built with 1352 images balanced between “undamaged”, “cracked”, and “delaminated” concrete surfaces. To increase the network robustness compared to images in real-world situations, different image configurations have been collected from the Internet and on-field bridge inspections. The GoogLeNet model, with the highest validation accuracy of approximately 94%, was selected as the most suitable network for concrete damage classification. The results confirm that the proposed model can correctly classify images from real concrete surfaces of bridges, tunnels, and pavement, resulting in an effective alternative to the current visual inspection techniques.     

Transformer-optimized generation,detection, and tracking network for images with drainage pipeline defects

Regular detection of defects in drainage pipelines is crucial. However, some problems associated with pipeline defect detection, such as data scarcity and defect counting difficulty, need to be addressed. Therefore, a Transformer-optimized generation, detection, and counting method for drainage-pipeline defects was established in this paper. First, a generation network called Trans-GAN-Cla was developed for data augmentation. A classification network was trained to improve the quality of the generated images. Second, a detection and tracking model called Trans-Det-Tra was developed to track and count the number of defects. Third, the feature extraction capability of the proposed method was improved by leveraging Transformers. Compared with some well-known convolutional neural network-based methods, the proposed network achieved the best classification and detection accuracies of 87.2% and 87.57%, respectively. Furthermore, the F₁ scores were 87.7% and 91.9%. Finally, two pieces of onsite videos were detected and tracked, and the numbers of misalignments and obstacles were accurately counted. The results indicate that the established Transformer-optimized method can generate high-quality images and realize the high-accuracy detection and counting of drainage pipeline defects.