Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

A unified convolutional neural network integrated with conditional random field for pipe defect segmentation

Semantic segmentation of closed‐circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel on the image, from which defect types, locations, and geometric information can be obtained. In this study, a unified neural network, namely DilaSeg‐CRF, is proposed by fully integrating a deep convolutional neural network (CNN) with dense conditional random field (CRF) for improving the segmentation accuracy. First, DilaSeg is constructed with dilated convolution and multiscale techniques for producing feature maps with high resolution. The steps of the dense CRF inference algorithm are converted into CNN operations, which are then formulated as recurrent neural network (RNN) layers. The DilaSeg‐CRF is proposed by integrating DilaSeg with the RNN layers. Images containing three common types of sewer defects are collected from CCTV inspection videos and are annotated with ground truth labels, after which the proposed models are trained and evaluated. Experiments demonstrate that the end‐to‐end trainable DilaSeg‐CRF can improve the segmentation significantly, with an increase of 32% and 20% in mean intersection over union (mIoU) values compared with fully convolutional network (FCN‐8s) and DilaSeg, respectively. Our proposed DilaSeg‐CRF also achieves faster inference speed than FCN and eliminates the manual postprocessing for refining the segmentation results.     

基于计算机视觉技术的结构表面裂缝检测方法研究

计算机视觉技术用于混凝土结构表面裂缝检测,具有现场检测方便、效率高、客观性强的特点,但图像数据分析是该技术的核心,其中裂缝提取与定量测量较为复杂。为提高裂缝图像处理效率和准确率,将深度学习和数字图像处理技术相结合,提出一种裂缝检测方法。建立基于深度卷积神经网络的裂缝识别模型,在图像上自动定位裂缝并结合图像局域阈值分割方法提取裂缝。在裂缝宽度定量测量方面,采用双边滤波算法和三段线性变换对裂缝图像进行预处理,提高了裂缝边缘识别的精确度。通过改进边缘梯度法,实现裂缝最大宽度的定位和裂缝最大宽度的自动获取。该研究为全自动识别裂缝图像及高精度测量裂缝宽度提供了一种解决方法。     

Encoder–decoder network for pixel‐level road crack detection in black‐box images

Timely monitoring of pavement cracks is essential for successful maintenance of road infrastructure. Accurate information concerning crack location and severity enables proactive management of the infrastructure. Black‐box cameras, which are becoming increasingly widespread at an affordable price, can be used as efficient road‐image collectors over a wide area. However, the cracks in these images are difficult to detect, because the images containing them often include objects other than roads. Thus, we propose a pixel‐level detection method for identifying road cracks in black‐box images using a deep convolutional encoder–decoder network. The encoder consists of convolutional layers of the residual network for extracting crack features, and the decoder consists of deconvolutional layers for localizing the cracks in an input image. The proposed network was trained on 427 out of 527 images extracted from black‐box videos and tested on the remaining 100 images. Compared with VGG‐16, ResNet‐50, ResNet‐101, ResNet‐200 with transfer learning, and ResNet‐152 without transfer learning, ResNet‐152 with transfer learning exhibited the best performance, achieving recall, precision, and intersection of union of 71.98%, 77.68%, and 59.65%, respectively. The experimental results prove that the proposed method is optimal for detecting cracks in black‐box images at the pixel level.     

Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques

Sanitary sewer systems are designed to collect and transport sanitary wastewater and stormwater. Pipe inspection is important in identifying both the type and location of pipe defects to maintain the normal sewer operations. Closed-circuit television (CCTV) has been commonly utilized for sewer pipe inspection. Currently, interpretation of the CCTV images is mostly conducted manually to identify the defect type and location, which is time-consuming, labor-intensive and inaccurate. Conventional computer vision techniques are explored for automated interpretation of CCTV images, but such process requires large amount of image pre-processing and the design of complex feature extractor for certain cases. In this study, an automated approach is developed for detecting sewer pipe defects based on a deep learning technique namely faster region-based convolutional neural network (faster R-CNN). The detection model is trained using 3000 images collected from CCTV inspection videos of sewer pipes. After training, the model is evaluated in terms of detection accuracy and computation cost using mean average precision (mAP), missing rate, detection speed and training time. The proposed approach is demonstrated to be applicable for detecting sewer pipe defects accurately with high accuracy and fast speed. In addition, a new model is constructed and several hyper-parameters are adjusted to study the influential factors of the proposed approach. The experiment results demonstrate that dataset size, initialization network type and training mode, and network hyper-parameters have influence on model performance. Specifically, the increase of dataset size and convolutional layers can improve the model accuracy. The adjustment of hyper-parameters such as filter dimensions or stride values contributes to higher detection accuracy, achieving an mAP of 83%. The study lays the foundation for applying deep learning techniques in sewer pipe defect detection as well as addressing similar issues for construction and facility management.     

Deep learning-based evaluation of factor of safety with confidence interval for tunnel deformation in spatially variable soil

The random finite difference method (RFDM) is a popular approach to quantitatively evaluate the influence of inherent spatial variability of soil on the deformation of embedded tunnels. However, the high computational cost is an ongoing challenge for its application in complex scenarios. To address this limitation, a deep learning-based method for efficient prediction of tunnel deformation in spatially variable soil is proposed. The proposed method uses one-dimensional convolutional neural network (CNN) to identify the pattern between random field input and factor of safety of tunnel deformation output. The mean squared error and correlation coefficient of the CNN model applied to the newly untrained dataset was less than 0.02 and larger than 0.96, respectively. It means that the trained CNN model can replace RFDM analysis for Monte Carlo simulations with a small but sufficient number of random field samples (about 40 samples for each case in this study). It is well known that the machine learning or deep learning model has a common limitation that the confidence of predicted result is unknown and only a deterministic outcome is given. This calls for an approach to gauge the model's confidence interval. It is achieved by applying dropout to all layers of the original model to retrain the model and using the dropout technique when performing inference. The excellent agreement between the CNN model prediction and the RFDM calculated results demonstrated that the proposed deep learning-based method has potential for tunnel performance analysis in spatially variable soils.     

Vision‐Based Automated Crack Detection for Bridge Inspection

Visual inspection of bridges is customarily used to identify and evaluate faults. However, current procedures followed by human inspectors demand long inspection times to examine large and difficult to access bridges. Also, highly relying on an inspector's subjective or empirical knowledge induces false evaluation. To address these limitations, a vision‐based visual inspection technique is proposed by automatically processing and analyzing a large volume of collected images. Images used in this technique are captured without controlling angles and positions of cameras and no need for preliminary calibration. As a pilot study, cracks near bolts on a steel structure are identified from images. Using images from many different angles and prior knowledge of the typical appearance and characteristics of this class of faults, the proposed technique can successfully detect cracks near bolts.     

A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects

Tunnel lining defects are an important indicator reflecting the safety status of shield tunnels. Inspired by the state‐of‐the‐art deep learning, a method for automatic intelligent classification and detection methodology of tunnel lining defects is presented. A fully convolutional network (FCN) model for classification is proposed. Information about defects, collected using charge‐coupled device cameras, was used to train the model. The model's performance was compared to those of GoogLeNet and VGG. The best‐set accuracy of the proposed model was over 95% at a test‐time speed of 48 ms per image. For defects detection, image features were computed from large‐scale images by the FCN and then detected using a region proposal network and position‐sensitive region of interest pooling. Some indices (detection rate, detection accuracy, and detection efficiency, locating accuracy) were used to evaluate the model. The comparisons with faster R‐CNN and a traditional method were conducted. The results show that the model is very fast and efficient, allowing automatic intelligent classification and detection of tunnel lining defects.     

Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images

Safety has been a concern for the construction industry for decades. Unsafe conditions and behaviors are considered as the major causes of construction accidents. The current safety inspection of conditions and behaviors heavily rely on human efforts which are limited onsite. To improve the safety performance of the industry, a more efficient approach to identify the unsafe conditions on site is required to supplement the current manual inspection practice. A promising way to supplement the current manual safety inspection is automated and intelligent monitoring/inspection through information and sensing technologies, including localization techniques, environment monitoring, image processing and etc. To assess the potential benefits of contemporary technologies for onsite safety inspection, the authors focused on real-time guardrail detection, as unprotected edges are the ones cause for workers falling from heights.In this paper, the authors developed a safety guardrail detection model based on convolutional neural network (CNN). An augmented data set is generated with the addition of background image to guardrail 3D models and used as training set. Transfer learning is utilized and the Visual Geometry Group architecture with 16 layers (VGG-16) model is adopted to construct the basic features extraction for the neural network. In the CNN implementation, 4000 augmented images were used to train the proposed model, while another 2000 images collected from real construction jobsites and 2000 images from Google were used to validate the proposed model. The proposed CNN-based guardrail detection model obtained a high accuracy of 96.5%. In addition, this study indicates that the synthetic images generated by augment technology can be used to create a large training dataset, and CNN-based image detection algorithm is a promising approach in construction jobsite safety monitoring.     

Deep domain adaptation based video smoke detection using synthetic smoke images

In this paper, a deep domain adaptation based method for video smoke detection is proposed to extract a powerful feature representation of smoke. Due to the smoke image samples limited in scale and diversity for deep CNN training, we systematically produced adequate synthetic smoke images with a wide variation in the smoke shape, background and lighting conditions. Considering that the appearance gap (dataset bias) between synthetic and real smoke images degrades significantly the performance of the trained model on the test set composed fully of real images, we build deep architectures based on domain adaptation to confuse the distributions of features extracted from synthetic and real smoke images. This approach expands the domain-invariant feature space for smoke image samples. With their approximate feature distribution separated from non-smoke images, the recognition rate of the trained model is improved significantly compared with the model trained directly on mixed dataset of synthetic and real images. Experimentally, several deep architectures with different design choices are applied to the smoke detector. The ultimate framework can get a satisfactory result on the test set. We believe that our method own strong robustness and may offer a new way for the video smoke detection.     

Structural health monitoring using extremely compressed data through deep learning

This study introduces a novel convolutional neural network (CNN)‐based approach for structural health monitoring (SHM) that exploits a form of measured compressed response data through transfer learning (TL)‐based techniques. The implementation of the proposed methodology allows damage identification and localization within a realistic large‐scale system. To validate the proposed method, first, a well‐known benchmark model is numerically simulated. Using acceleration response histories, as well as compressed response data in terms of discrete histograms, CNN models are trained, and the robustness of the CNN architectures is evaluated. Finally, pretrained CNNs are fine‐tuned to be adaptable for three‐parameter, extremely compressed response data, based on the response mean, standard deviation, and a scale factor. The performance of each CNN implementation is assessed using training accuracy histories as well as confusion matrices, along with other performance metrics. In addition to the numerical study, the performance of the proposed method is demonstrated using experimental vibration response data for verification and validation. The results indicate that deep TL can be implemented effectively for SHM of similar structural systems with different types of sensors.     

Slope stability analysis based on big data and convolutional neural network

The Limit Equilibrium Method (LEM) is commonly used in traditional slope stability analyses, but it is time-consuming and complicated. Due to its complexity and nonlinearity involved in the evaluation process, it cannot provide a quick stability estimation when facing a large number of slopes. In this case, the convolutional neural network (CNN) provides a better alternative. A CNN model can process data quickly and complete a large amount of data analysis in a specific situation, while it needs a large number of training samples. It is difficult to get enough slope data samples in practical engineering. This study proposes a slope database generation method based on the LEM. Samples were amplified from 40 typical slopes, and a sample database consisting of 20000 slope samples was established. The sample database for slopes covered a wide range of slope geometries and soil layers’ physical and mechanical properties. The CNN trained with this sample database was then applied to the stability prediction of 15 real slopes to test the accuracy of the CNN model. The results show that the slope stability prediction method based on the CNN does not need complex calculation but only needs to provide the slope coordinate information and physical and mechanical parameters of the soil layers, and it can quickly obtain the safety factor and stability state of the slopes. Moreover, the prediction accuracy of the CNN trained by the sample database for slope stability analysis reaches more than 99%, and the comparisons with the BP neural network show that the CNN has significant superiority in slope stability evaluation. Therefore, the CNN can predict the safety factor of real slopes. In particular, the combination of typical actual slopes and generated slope data provides enough training and testing samples for the CNN, which improves the prediction speed and practicability of the CNN-based evaluation method in engineering practice.     

A methodology for obtaining spatiotemporal information of the vehicles on bridges based on computer vision

Spatiotemporal information of the vehicles on a bridge is important evidence for reflecting the stress state and traffic density of the bridge. A methodology for obtaining the information is proposed based on computer vision technology, which contains the detection by Faster region‐based convolutional neural network (Faster R‐CNN), multiple object tracking, and image calibration. For minimizing the detection time, the ZF (Zeiler & Fergus) model with five convolutional layers is selected as the shared part between Region Proposal Network and Fast R‐CNN in Faster R‐CNN. An image data set including 1,694 images is established about eight types of vehicles for training Faster R‐CNN. Combined with the detection of each frame of the video, the methods of multiple object tracking and image calibration are developed for acquiring the vehicle parameters, including the length, number of axles, speed, and the lane that the vehicle is in. The method of tracking is mainly based on the judgment of the distances between the vehicle bounding boxes in virtual detection region. As for image calibration, it is based on the moving standard vehicles whose lengths are known, which can be regarded as the 3D template to calculate the vehicle parameters. After acquiring the vehicles' parameters, the spatiotemporal information of the vehicles can be obtained. The proposed system has a frame rate of 16 fps and only needs two cameras as the input device. The system is successfully applied on a double tower cable‐stayed bridge, and the identification accuracies of the types and number of axles are about 90 and 73% in the virtual detection region, and the speed errors of most vehicles are less than 6%.     

Automated crack evaluation of a high‐rise bridge pier using a ring‐type climbing robot

This article proposes a deep learning‐based automated crack evaluation technique for a high‐rise bridge pier using a ring‐type climbing robot. First, a ring‐type climbing robot system composed of multiple vision cameras, climbing robot, and control computer is developed. By spatially moving the climbing robot system along a target bridge pier with close‐up scanning condition, high‐quality raw vision images are continuously obtained. The raw vision images are then processed through feature control‐based image stitching, deep learning‐based semantic segmentation, and Euclidean distance transform–based crack quantification algorithms. Finally, a digital crack map on the region of interest (ROI) of the target bridge pier can be automatically established. The proposed technique is experimentally validated using in situ test data obtained from Jang–Duck bridge in South Korea. The test results reveal that the proposed technique successfully evaluates cracks on the entire ROI of the bridge pier with precision of 90.92% and recall of 97.47%.     

全景绿视率自动识别和计算研究

绿视率是用于绿色空间感知的直观评价标准，传统研究的绿视率多基于平面影像进行计算，不能完全反映三维空间中人对绿量的主观感受。基于全景影像，提出全景绿视率的概念，通过全景相机获取球面全景照片，将等距圆柱投影转换为等积圆柱投影，利用基于语义分割的卷积神经网络模型，自动识别植被区域面积以实现全景绿视率自动化识别和计量。通过比较5项卷积神经网络模型对绿视率的识别效果，显示出Dilated ResNet-105神经网络模型具有最高的识别准确度。以武汉市武昌区紫阳公园为例，对各级园路和广场的全景绿视率进行计算和分析。将卷积神经网络的识别结果同人工判别结果进行对比研究，结果显示：使用Dilated ResNet-105卷积神经网络对绿植范围识别的平均交并比（mIoU）为62.53%，与人工识别的平均差异为9.17%。全景绿视率自动识别和计算可以为相关研究提供新的思路，实现客观准确、快速便捷的绿视率测量评估。     

Network‐wide traffic speed forecasting: 3D convolutional neural network with ensemble empirical mode decomposition

Accurate traffic speed forecasting is one of the most critical tasks in proactive traffic management and the deployment of advanced traveler information systems. This paper proposes a hybrid forecasting approach named DeepEnsemble by integrating the three‐dimensional convolutional neural network (3D CNN) with ensemble empirical mode decomposition (EEMD). There are four steps in this hybrid approach. First, EEMD is adopted to decompose the complex traffic speed time series data with noise into several intrinsic mode functions (IMFs) and a residue. Second, a three‐dimensional tensor is established and fed into 3D CNN for prediction. Third, the output of 3D CNN prediction is obtained by a linear combination of the results of all components. Finally, the 3D CNN prediction output, external features, and historical features are fused to predict the network‐wide traffic speed simultaneously. The proposed DeepEnsemble approach is tested on the three‐month traffic speed series data of a real‐world large‐scale urban expressway network with 308 traffic flow detectors in Beijing, China. The experimental results indicate that DeepEnsemble outperforms the state‐of‐the‐art network‐wide traffic speed forecasting models. 3D CNN learns temporal, spatial, and depth information better than 2D CNN. Moreover, forecasting accuracy can be improved by employing EEMD. DeepEnsemble is a promising model with scalability and portability for network‐wide traffic speed prediction and can be further extended to conduct traffic status monitoring and congestion mitigation strategies.     

Convolutional sparse coding‐based deep random vector functional link network for distress classification of road structures

This paper presents a convolutional sparse coding (CSC)‐based deep random vector functional link network (CSDRN) for distress classification of road structures. The main contribution of this paper is the introduction of CSC into a feature extraction scheme in the distress classification. CSC can extract visual features representing characteristics of target images because it can successfully estimate optimal convolutional dictionary filters and sparse features as visual features by training from a small number of distress images. The optimal dictionaries trained from distress images have basic components of visual characteristics such as edge and line information of distress images. Furthermore, sparse feature maps estimated on the basis of the dictionaries represent both strength of the basic components and location information of regions having their components, and these maps can represent distress images. That is, sparse feature maps can extract key components from distress images that have diverse visual characteristics. Therefore, CSC‐based feature extraction is effective for training from a limited number of distress images that have diverse visual characteristics. The construction of a novel neural network, CSDRN, by the use of a combination of CSC‐based feature extraction and the DRN classifier, which can also be trained from a small dataset, is shown in this paper. Accurate distress classification is realized via the CSDRN.