Pavement defect detection with fully convolutional network and an uncertainty framework

Image segmentation has been implemented for pavement defect detection, from which types, locations, and geometric information can be obtained. In this study, an integration of a fully convolutional network with a Gaussian‐conditional random field (G‐CRF), an uncertainty framework, and probability‐based rejection is proposed for detecting pavement defects. First, a fully convolutional network is designed to generate preliminary segmentation results, and a G‐CRF is used to refine the segmentation. Second, epistemic and aleatory uncertainties in the model and database are considered to overcome the disadvantages of traditional deep‐learning methods. Last, probability‐based rejection is conducted to remove unreasonable segmentations. The proposed method is evaluated on a data set of images that were obtained from 16 highways. The proposed integration segments pavement distresses from digital images with desirable performance. It also provides a satisfactory means to improve the accuracy and generalization performance of pavement defect detection without introducing a delay into the segmentation process.     

Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques

Sanitary sewer systems are designed to collect and transport sanitary wastewater and stormwater. Pipe inspection is important in identifying both the type and location of pipe defects to maintain the normal sewer operations. Closed-circuit television (CCTV) has been commonly utilized for sewer pipe inspection. Currently, interpretation of the CCTV images is mostly conducted manually to identify the defect type and location, which is time-consuming, labor-intensive and inaccurate. Conventional computer vision techniques are explored for automated interpretation of CCTV images, but such process requires large amount of image pre-processing and the design of complex feature extractor for certain cases. In this study, an automated approach is developed for detecting sewer pipe defects based on a deep learning technique namely faster region-based convolutional neural network (faster R-CNN). The detection model is trained using 3000 images collected from CCTV inspection videos of sewer pipes. After training, the model is evaluated in terms of detection accuracy and computation cost using mean average precision (mAP), missing rate, detection speed and training time. The proposed approach is demonstrated to be applicable for detecting sewer pipe defects accurately with high accuracy and fast speed. In addition, a new model is constructed and several hyper-parameters are adjusted to study the influential factors of the proposed approach. The experiment results demonstrate that dataset size, initialization network type and training mode, and network hyper-parameters have influence on model performance. Specifically, the increase of dataset size and convolutional layers can improve the model accuracy. The adjustment of hyper-parameters such as filter dimensions or stride values contributes to higher detection accuracy, achieving an mAP of 83%. The study lays the foundation for applying deep learning techniques in sewer pipe defect detection as well as addressing similar issues for construction and facility management.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects

Tunnel lining defects are an important indicator reflecting the safety status of shield tunnels. Inspired by the state‐of‐the‐art deep learning, a method for automatic intelligent classification and detection methodology of tunnel lining defects is presented. A fully convolutional network (FCN) model for classification is proposed. Information about defects, collected using charge‐coupled device cameras, was used to train the model. The model's performance was compared to those of GoogLeNet and VGG. The best‐set accuracy of the proposed model was over 95% at a test‐time speed of 48 ms per image. For defects detection, image features were computed from large‐scale images by the FCN and then detected using a region proposal network and position‐sensitive region of interest pooling. Some indices (detection rate, detection accuracy, and detection efficiency, locating accuracy) were used to evaluate the model. The comparisons with faster R‐CNN and a traditional method were conducted. The results show that the model is very fast and efficient, allowing automatic intelligent classification and detection of tunnel lining defects.     

Network‐wide traffic speed forecasting: 3D convolutional neural network with ensemble empirical mode decomposition

Accurate traffic speed forecasting is one of the most critical tasks in proactive traffic management and the deployment of advanced traveler information systems. This paper proposes a hybrid forecasting approach named DeepEnsemble by integrating the three‐dimensional convolutional neural network (3D CNN) with ensemble empirical mode decomposition (EEMD). There are four steps in this hybrid approach. First, EEMD is adopted to decompose the complex traffic speed time series data with noise into several intrinsic mode functions (IMFs) and a residue. Second, a three‐dimensional tensor is established and fed into 3D CNN for prediction. Third, the output of 3D CNN prediction is obtained by a linear combination of the results of all components. Finally, the 3D CNN prediction output, external features, and historical features are fused to predict the network‐wide traffic speed simultaneously. The proposed DeepEnsemble approach is tested on the three‐month traffic speed series data of a real‐world large‐scale urban expressway network with 308 traffic flow detectors in Beijing, China. The experimental results indicate that DeepEnsemble outperforms the state‐of‐the‐art network‐wide traffic speed forecasting models. 3D CNN learns temporal, spatial, and depth information better than 2D CNN. Moreover, forecasting accuracy can be improved by employing EEMD. DeepEnsemble is a promising model with scalability and portability for network‐wide traffic speed prediction and can be further extended to conduct traffic status monitoring and congestion mitigation strategies.     

Vibration‐based structural state identification by a 1‐dimensional convolutional neural network

Deep learning has ushered in many breakthroughs in vision‐based detection via convolutional neural networks (CNNs), but the vibration‐based structural damage detection by CNN remains being refined. Thus, this study proposes a simple one‐dimensional CNN that detects tiny local structural stiffness and mass changes, and validates the proposed CNN on actual structures. Three independent acceleration databases are established based on a T‐shaped steel beam, a short steel girder bridge (in test field), and a long steel girder bridge (in service). The raw acceleration data are not pre‐processed and are directly used as the training and validation data. The well‐trained CNN almost perfectly identifies the locations of small local changes in the structural mass and stiffness, demonstrating the high sensitivity of the proposed simple CNN to tiny structural state changes in actual structures. The convolutional kernels and outputs of the convolutional and max pooling layers are visualized and discussed as well.     

Optimal CNN-based semantic segmentation model of cutting slope images

This paper utilizes three popular semantic segmentation networks, specifically DeepLab v3+, fully convolutional network (FCN), and U-Net to qualitively analyze and identify the key components of cutting slope images in complex scenes and achieve rapid image-based slope detection. The elements of cutting slope images are divided into 7 categories. In order to determine the best algorithm for pixel level classification of cutting slope images, the networks are compared from three aspects: a) different neural networks, b) different feature extractors, and c) 2 different optimization algorithms. It is found that DeepLab v3+ with Resnet18 and Sgdm performs best, FCN 32s with Sgdm takes the second, and U-Net with Adam ranks third. This paper also analyzes the segmentation strategies of the three networks in terms of feature map visualization. Results show that the contour generated by DeepLab v3+ (combined with Resnet18 and Sgdm) is closest to the ground truth, while the resulting contour of U-Net (combined with Adam) is closest to the input images.     

Structural health monitoring using extremely compressed data through deep learning

This study introduces a novel convolutional neural network (CNN)‐based approach for structural health monitoring (SHM) that exploits a form of measured compressed response data through transfer learning (TL)‐based techniques. The implementation of the proposed methodology allows damage identification and localization within a realistic large‐scale system. To validate the proposed method, first, a well‐known benchmark model is numerically simulated. Using acceleration response histories, as well as compressed response data in terms of discrete histograms, CNN models are trained, and the robustness of the CNN architectures is evaluated. Finally, pretrained CNNs are fine‐tuned to be adaptable for three‐parameter, extremely compressed response data, based on the response mean, standard deviation, and a scale factor. The performance of each CNN implementation is assessed using training accuracy histories as well as confusion matrices, along with other performance metrics. In addition to the numerical study, the performance of the proposed method is demonstrated using experimental vibration response data for verification and validation. The results indicate that deep TL can be implemented effectively for SHM of similar structural systems with different types of sensors.     

Optical flow techniques for estimation of camera motion parameters in sewer closed circuit television inspection videos

This paper discusses a novel approach for automated analysis and tracking of camera motion in sewer inspection closed circuit television (CCTV) videos. This approach represents an important building block for any system that supports automated analysis and defect detection of CCTV videos. The proposed approach employs optical flow techniques to automatically identify, locate, and extract a limited set of video segments, called regions of interest (ROI), which likely include defects, thus reducing the time and computational requirements needed for video processing. Tracking the camera motion parameters is used to recover the operator actions during the inspection session, which would provide important clues about the location and severity of the ROI. Techniques for estimating the camera travelling distance, position inside the sewer, and direction of motion from optical flow vectors are discussed. The proposed techniques were validated using a representative set of sewer CCTV videos obtained from the cities of Regina and Calgary, Canada.     

Tunnel boring machine vibration-based deep learning for the ground identification of working faces

Tunnel boring machine (TBM) vibration induced by cutting complex ground contains essential information that can help engineers evaluate the interaction between a cutterhead and the ground itself. In this study, deep recurrent neural networks (RNNs) and convolutional neural networks (CNNs) were used for vibration-based working face ground identification. First, field monitoring was conducted to obtain the TBM vibration data when tunneling in changing geological conditions, including mixed-face, homogeneous, and transmission ground. Next, RNNs and CNNs were utilized to develop vibration-based prediction models, which were then validated using the testing dataset. The accuracy of the long short-term memory (LSTM) and bidirectional LSTM (Bi-LSTM) models was approximately 70% with raw data; however, with instantaneous frequency transmission, the accuracy increased to approximately 80%. Two types of deep CNNs, GoogLeNet and ResNet, were trained and tested with time-frequency scalar diagrams from continuous wavelet transformation. The CNN models, with an accuracy greater than 96%, performed significantly better than the RNN models. The ResNet-18, with an accuracy of 98.28%, performed the best. When the sample length was set as the cutterhead rotation period, the deep CNN and RNN models achieved the highest accuracy while the proposed deep CNN model simultaneously achieved high prediction accuracy and feedback efficiency. The proposed model could promptly identify the ground conditions at the working face without stopping the normal tunneling process, and the TBM working parameters could be adjusted and optimized in a timely manner based on the predicted results.     

3D convolutional neural network-based one-stage model for real-time action detection in video of construction equipment

This study aims to propose a three-dimensional convolutional neural network (3D CNN)-based one-stage model for real-time action detection in video of construction equipment (ADVICE). The 3D CNN-based single-stream feature extraction network and detection network are designed with the implementation of the 3D attention module and feature pyramid network developed in this study to improve performance. For model evaluation, 130 videos were collected from YouTube including videos of four types of construction equipment at various construction sites. Trained on 520 clips and tested on 260 clips, ADVICE achieved precision and recall of 82.1% and 83.1%, respectively, with an inference speed of 36.6 frames per second. The evaluation results indicate that the proposed method can implement the 3D CNN-based one-stage model for real-time action detection of construction equipment in videos of diverse, variable, and complex construction sites. The proposed method paved the way to improving safety, productivity, and environmental management of construction projects.     

卷积神经网络在结构损伤诊断中的应用

对卷积神经网络(CNN)在工程结构损伤诊断中的应用进行了深入探讨; 以多层框架结构节点损伤位置的识别问题为研究对象,构建了可以直接从结构动力反应信号中进行学习并完成分类诊断的基于原始信号和傅里叶频域信息的一维卷积神经网络模型和基于小波变换数据的二维卷积神经网络模型; 从输入数据样本类别、训练时间、预测准确率、浅层与深层卷积神经网络以及不同损伤程度的影响等多方面进行了研究。结果表明:卷积神经网络能从结构动力反应信息中有效提取结构的损伤特征,且具有很高的识别精度; 相比直接用加速度反应样本,使用傅里叶变换后的频域数据作为训练样本能使CNN的收敛速度更快、更稳定,并且深层CNN的性能要好于浅层CNN; 将卷积神经网络用于工程结构损伤诊断具有可行性,特别是在大数据处理和解决复杂问题能力方面与其他传统诊断方法相比有很大优势,应用前景广阔。     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision

Spatiotemporal information of the vehicles on a bridge is important evidence for reflecting the stress state and traffic density of the bridge. A methodology for obtaining the information is proposed based on computer vision technology, which contains the detection by Faster region‐based convolutional neural network (Faster R‐CNN), multiple object tracking, and image calibration. For minimizing the detection time, the ZF (Zeiler & Fergus) model with five convolutional layers is selected as the shared part between Region Proposal Network and Fast R‐CNN in Faster R‐CNN. An image data set including 1,694 images is established about eight types of vehicles for training Faster R‐CNN. Combined with the detection of each frame of the video, the methods of multiple object tracking and image calibration are developed for acquiring the vehicle parameters, including the length, number of axles, speed, and the lane that the vehicle is in. The method of tracking is mainly based on the judgment of the distances between the vehicle bounding boxes in virtual detection region. As for image calibration, it is based on the moving standard vehicles whose lengths are known, which can be regarded as the 3D template to calculate the vehicle parameters. After acquiring the vehicles' parameters, the spatiotemporal information of the vehicles can be obtained. The proposed system has a frame rate of 16 fps and only needs two cameras as the input device. The system is successfully applied on a double tower cable‐stayed bridge, and the identification accuracies of the types and number of axles are about 90 and 73% in the virtual detection region, and the speed errors of most vehicles are less than 6%.     

基于卷积神经网络的结构损伤识别研究进展

为系统梳理基于卷积神经网络的工程结构损伤识别方法的发展脉络和研究现状,分别从结构损伤的识别目的和在不同类型结构中的应用两方面进行了归类、分析和评价。介绍了卷积神经网络的基本结构和评价指标,回顾了卷积神经网络的研究和应用历程。在损伤的识别目的方面,主要针对混凝土结构损伤的分类、定位和分割,详细介绍了基于不同类型卷积神经网络的结构损伤识别方法,即基于分类的方法、基于回归的方法和像素级的图像分割算法; 分析了各类方法所使用的卷积神经网络模型的结构特点、计算流程、训练方法和损伤识别性能。在不同类型结构的损伤识别方面,分析了卷积神经网络在砌体结构、钢结构桥梁和古建筑木结构裂缝识别中的应用。最后,基于对卷积神经网络优缺点的思考,提出了发展建议和展望。结果表明:训练样本中结构损伤的多样性对模型的损伤识别效果影响较大; 现有基于卷积神经网络的损伤分割方法模型参数较多,计算量大; 采用数据增广和迁移学习方法可有效防止模型过拟合,提高模型训练效率; 针对微小损伤和不同类型结构损伤的识别,此类方法的性能有待提高。     

Structure-aware dehazing of sewer inspection images based on monocular depth cues

In sewer pipes, haze caused by the humid environment seriously impairs the quality of closed-circuit television (CCTV) images, which leads to poor performance of subsequent pipe defects detection. Meanwhile, the complexity of sewer images, such as steep depth change and extensive textureless regions, brings great challenges to the performance or application of general dehazing algorithms. Therefore, this study estimates sewer depth maps first with the help of the water–pipewall borderlines to produce the paired dehazing dataset. Then a structure-aware nonlocal network (SANL-Net) is proposed with the detected borderlines and the dehazing result as two supervisory signals. SANL-Net shows its superiority over other state-of-the-art approaches with 147 in mean square error (MSE), 27.28 in peak signal to noise ratio (PSNR), 0.8963 in structural similarity index measure (SSIM), and 15.47M in parameters. Also, the outstanding performance in real image dehazing implies the accuracy of depth estimation. Experimental results indicate that SANL-Net significantly improves the performance of defects detection tasks, such as an increase of 23.16% in mean intersection over union (mIoU) for semantic segmentation.     

Deep learning-based evaluation of factor of safety with confidence interval for tunnel deformation in spatially variable soil

The random finite difference method (RFDM) is a popular approach to quantitatively evaluate the influence of inherent spatial variability of soil on the deformation of embedded tunnels. However, the high computational cost is an ongoing challenge for its application in complex scenarios. To address this limitation, a deep learning-based method for efficient prediction of tunnel deformation in spatially variable soil is proposed. The proposed method uses one-dimensional convolutional neural network (CNN) to identify the pattern between random field input and factor of safety of tunnel deformation output. The mean squared error and correlation coefficient of the CNN model applied to the newly untrained dataset was less than 0.02 and larger than 0.96, respectively. It means that the trained CNN model can replace RFDM analysis for Monte Carlo simulations with a small but sufficient number of random field samples (about 40 samples for each case in this study). It is well known that the machine learning or deep learning model has a common limitation that the confidence of predicted result is unknown and only a deterministic outcome is given. This calls for an approach to gauge the model's confidence interval. It is achieved by applying dropout to all layers of the original model to retrain the model and using the dropout technique when performing inference. The excellent agreement between the CNN model prediction and the RFDM calculated results demonstrated that the proposed deep learning-based method has potential for tunnel performance analysis in spatially variable soils.     

Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine

Condition assessment of municipal sewer pipes using closed circuit television (CCTV) inspections is known to be time consuming, costly, and prone to errors primarily due to operator fatigue or novicity. Automated detection of defects can provide a valuable tool for ensuring the quality, accuracy, and consistency of condition data, while reducing the time and cost of the inspection process. This paper presents an efficient pattern recognition algorithm to support automated detection and classification of pipe defects in images obtained from conventional CCTV inspection videos. The algorithm employs the histograms of oriented gradients (HOG) and support vector machine (SVM) to identify pipe defects. The algorithm involves two main steps: (1) image segmentation to extract suspicious regions of interest (ROI) that represent candidate defect areas; and (2) classification of the ROI using SVM classifier that was trained using sets of HOG features extracted from positive and negative examples of the defect. Proposed algorithm is applied to the problem of detecting tree root intrusions. The performance of linear and radial basis function SVM classifiers evaluated. The algorithm was tested on a set of actual CCTV videos obtained from the cities of Regina and Calgary in Canada. Experimental results demonstrated the viability and robustness of the algorithm.     

Semantic Fire Segmentation Model Based on Convolutional Neural Network for Outdoor Image

Fire Technology - In this paper, we proposed a semantic fire image segmentation method using a convolutional neural network. The simple but powerful method proposed is middle skip connection...