Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network

Deep learning‐based structural damage detection methods overcome the limitation of inferior adaptability caused by extensively varying real‐world situations (e.g., lighting and shadow changes). However, most deep learning‐based methods detect structural damage at the image level and grid‐cell level. To provide pixel‐level detection of multiple damages, a Fully Convolutional Network (FCN)‐based multiple damages detection method for concrete structure is proposed. To realize this method, a database of 2,750 images (with 504 × 376 pixels) including crack, spalling, efflorescence, and hole images in concrete structure is built, and the four damages included in those images are labeled manually. Then, the architecture of the FCN is modified, trained, validated, and tested using this database. A strategy of model‐based transfer learning is used to initialize the parameters of the FCN during the training process. The results show 98.61% pixel accuracy (PA), 91.59% mean pixel accuracy (MPA), 84.53% mean intersection over union (MIoU), and 97.34% frequency weighted intersection over union (FWIoU). Subsequently, the robustness and adaptability of the trained FCN model is tested and the damage is extracted, where damage areas are provided according to a calibrated relation between the ratio (the pixel area and true area of the detected object) and the distance from the smartphone to the concrete surface using a laser range finder. A comparative study is conducted to examine the performance of the proposed FCN‐based approach using a SegNet‐based method. The results show that the proposed method substantiates quite better performance and can indeed detect multiple concrete damages at the pixel level in realistic situations.     

Encoder–decoder network for pixel‐level road crack detection in black‐box images

Timely monitoring of pavement cracks is essential for successful maintenance of road infrastructure. Accurate information concerning crack location and severity enables proactive management of the infrastructure. Black‐box cameras, which are becoming increasingly widespread at an affordable price, can be used as efficient road‐image collectors over a wide area. However, the cracks in these images are difficult to detect, because the images containing them often include objects other than roads. Thus, we propose a pixel‐level detection method for identifying road cracks in black‐box images using a deep convolutional encoder–decoder network. The encoder consists of convolutional layers of the residual network for extracting crack features, and the decoder consists of deconvolutional layers for localizing the cracks in an input image. The proposed network was trained on 427 out of 527 images extracted from black‐box videos and tested on the remaining 100 images. Compared with VGG‐16, ResNet‐50, ResNet‐101, ResNet‐200 with transfer learning, and ResNet‐152 without transfer learning, ResNet‐152 with transfer learning exhibited the best performance, achieving recall, precision, and intersection of union of 71.98%, 77.68%, and 59.65%, respectively. The experimental results prove that the proposed method is optimal for detecting cracks in black‐box images at the pixel level.     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.     

Concrete crack detection using context‐aware deep semantic segmentation network

Computer‐vision and deep‐learning techniques are being increasingly applied to inspect, monitor, and assess infrastructure conditions including detection of cracks. Traditional vision‐based methods to detect cracks lack accuracy and generalization to work on complicated infrastructural conditions. This paper presents a novel context‐aware deep convolutional semantic segmentation network to effectively detect cracks in structural infrastructure under various conditions. The proposed method applies a pixel‐wise deep semantic segmentation network to segment the cracks on images with arbitrary sizes without retraining the prediction network. Meanwhile, a context‐aware fusion algorithm that leverages local cross‐state and cross‐space constraints is proposed to fuse the predictions of image patches. This method is evaluated on three datasets: CrackForest Dataset (CFD) and Tomorrows Road Infrastructure Monitoring, Management Dataset (TRIMMD) and a Customized Field Test Dataset (CFTD) and achieves Boundary F1 (BF) score of 0.8234, 0.8252, and 0.7937 under 2‐pixel error tolerance margin in CFD, TRIMMD, and CFTD, respectively. The proposed method advances the state‐of‐the‐art performance of BF score by approximately 2.71% in CFD, 1.47% in TRIMMD, and 4.14% in CFTD. Moreover, the averaged processing time of the proposed system is 0.7 s on a typical desktop with Intel^® Quad‐Core? i7‐7700 CPU@3.6 GHz Processor, 16GB RAM and NVIDIA GeForce GTX 1060 6GB GPU for an image of size 256 × 256 pixels.     

A unified convolutional neural network integrated with conditional random field for pipe defect segmentation

Semantic segmentation of closed‐circuit television (CCTV) images can facilitate automatic severity assessment of sewer pipe defects by assigning defect labels to each pixel on the image, from which defect types, locations, and geometric information can be obtained. In this study, a unified neural network, namely DilaSeg‐CRF, is proposed by fully integrating a deep convolutional neural network (CNN) with dense conditional random field (CRF) for improving the segmentation accuracy. First, DilaSeg is constructed with dilated convolution and multiscale techniques for producing feature maps with high resolution. The steps of the dense CRF inference algorithm are converted into CNN operations, which are then formulated as recurrent neural network (RNN) layers. The DilaSeg‐CRF is proposed by integrating DilaSeg with the RNN layers. Images containing three common types of sewer defects are collected from CCTV inspection videos and are annotated with ground truth labels, after which the proposed models are trained and evaluated. Experiments demonstrate that the end‐to‐end trainable DilaSeg‐CRF can improve the segmentation significantly, with an increase of 32% and 20% in mean intersection over union (mIoU) values compared with fully convolutional network (FCN‐8s) and DilaSeg, respectively. Our proposed DilaSeg‐CRF also achieves faster inference speed than FCN and eliminates the manual postprocessing for refining the segmentation results.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

Automatic detection method of cracks from concrete surface imagery using two‐step light gradient boosting machine

Automated crack detection based on image processing is widely used when inspecting concrete structures. The existing methods for crack detection are not yet accurate enough due to the difficulty and complexity of the problem; thus, more accurate and practical methods should be developed. This paper proposes an automated crack detection method based on image processing using the light gradient boosting machine (LightGBM), one of the supervised machine learning methods. In supervised machine learning, appropriate features should be identified to obtain accurate results. In crack detection, the pixel values of the target pixels and geometric features of the cracks that occur when they are connected linearly should be considered. This paper proposes a methodology for generating features based on pixel values and geometric shapes in two stages. The accuracy of the proposed methodology is investigated using photos of concrete structures with adverse conditions, such as shadows and dirt. The proposed methodology achieves an accuracy of 99.7%, sensitivity of 75.71%, specificity of 99.9%, precision of 68.2%, and an F‐measure of 0.6952. The experimental results demonstrate that the proposed method can detect cracks with higher performance than the pix2pix‐based approach. Furthermore, the training time is 7.7 times shorter than that of the XGBoost and 2.3 times shorter than that of the pix2pix. The experimental results demonstrate that the proposed method can detect cracks with high accuracy.     

Vibration‐based structural state identification by a 1‐dimensional convolutional neural network

Deep learning has ushered in many breakthroughs in vision‐based detection via convolutional neural networks (CNNs), but the vibration‐based structural damage detection by CNN remains being refined. Thus, this study proposes a simple one‐dimensional CNN that detects tiny local structural stiffness and mass changes, and validates the proposed CNN on actual structures. Three independent acceleration databases are established based on a T‐shaped steel beam, a short steel girder bridge (in test field), and a long steel girder bridge (in service). The raw acceleration data are not pre‐processed and are directly used as the training and validation data. The well‐trained CNN almost perfectly identifies the locations of small local changes in the structural mass and stiffness, demonstrating the high sensitivity of the proposed simple CNN to tiny structural state changes in actual structures. The convolutional kernels and outputs of the convolutional and max pooling layers are visualized and discussed as well.     

Concrete Crack Detection by Multiple Sequential Image Filtering

Abstract: This article presents a new robust automated image processing method for detecting cracks in surface images of concrete structures. This method involves two steps: (1) development of an image filter for detecting major cracks using genetic programming (GP), and (2) elimination of residual noise after filtering and detection of indistinct cracks by iterative applications of the image filter to the local regions surrounding the cracks. The proposed method can be used for the accurate detection of cracks in surface images recorded under various conditions. Moreover, the widths of the detected cracks can be quantified on the basis of the spatial derivatives of the brightness patterns. The estimated crack widths are in good agreement with those measured manually.     

Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system

Crack information provides important evidence of structural degradation and safety in civil structures. Existing inspection methods are inefficient and difficult to rapidly deploy. A real‐time crack inspection method is proposed in this study to address this difficulty. Within this method, a wall‐climbing unmanned aerial system (UAS) is developed to acquire detailed crack images without distortion, then a wireless data transmission method is applied to fulfill real‐time detection requirements, allowing smartphones to receive real‐time video taken from the UAS. Next, an image data set including 1,330 crack images taken by the wall‐climbing UAS is established and used for training a deep‐learning model. For increasing detection speed, state‐of‐the‐art convolutional neural networks (CNNs) are compared and employed to train the crack detector; the selected model is transplanted into an android application so that the detection of cracks can be undertaken on a smartphone in real time. Following this, images with cracks are separated and crack width is calculated using an image processing method. The proposed method is then applied to a building where crack information is acquired and calculated accurately with high efficiency, thus verifying the practicability of the proposed method and system.     

Foreground–background separation technique for crack detection

Current level‐2 condition assessment methods for critical infrastructure assets mostly rely on human visual investigation of visible damages and patterns at the structure surface, which can be a costly, time‐consuming, and subjective exercise in reality. In this article, a novel method for crack detection is proposed via salient structure extraction from textured background. This method first extracts strong edges and distinguishes them from strong textures in a local neighborhood. Then, the spatial distribution of texture features is estimated to detect cracks as salient structures that are not widely spread across the whole image. The outputs from these two key steps are fused to calculate the final structure saliency map for generation of the crack masks. This method was validated on a data set with 704 images and the outcome revealed an average f‐measure of 75% in detecting the concrete cracks that is significantly higher than two other baseline methods.     

Image analysis method for crack distribution and width estimation for reinforced concrete structures

Crack observation is important for evaluating the structural performance and safety of reinforced concrete (RC) structures. Most of the existing image-based crack detection methods are based on edge detection algorithms, which detect cracks that are wide enough to present dark areas in the obtained images. Cracks initiate as thin cracks, generally having width less than the width of a pixel in images; such cracks are generally undetectable by edge detection-based methods.An image analysis method is proposed to observe the development and distribution of thin cracks on RC surfaces; it also allows estimation of crack widths. Image matching based on optical flow and subpixel is employed to analyze slight concrete surface displacements. Camera calibration is included to eliminate perspective effects and lens distortion. Geometric transformation is adopted so that cameras do not need to be perpendicular to the observed surface or specified positions. Formulas are proposed to estimate the width of shear crack opening. The proposed method was then applied to a cyclic test of an RC structure. The crack widths and their development analyzed by the image analysis were verified with human inspection in the test. In addition, concrete surface cracks that appeared at a very early stage of the test could be observed by the proposed method before they could be detected by the naked eye. The results thus demonstrate that the proposed image analysis method offers an efficient way applicable not only for structural tests but also for crack-based structural-health-monitoring applications.     

Structural health monitoring using extremely compressed data through deep learning

This study introduces a novel convolutional neural network (CNN)‐based approach for structural health monitoring (SHM) that exploits a form of measured compressed response data through transfer learning (TL)‐based techniques. The implementation of the proposed methodology allows damage identification and localization within a realistic large‐scale system. To validate the proposed method, first, a well‐known benchmark model is numerically simulated. Using acceleration response histories, as well as compressed response data in terms of discrete histograms, CNN models are trained, and the robustness of the CNN architectures is evaluated. Finally, pretrained CNNs are fine‐tuned to be adaptable for three‐parameter, extremely compressed response data, based on the response mean, standard deviation, and a scale factor. The performance of each CNN implementation is assessed using training accuracy histories as well as confusion matrices, along with other performance metrics. In addition to the numerical study, the performance of the proposed method is demonstrated using experimental vibration response data for verification and validation. The results indicate that deep TL can be implemented effectively for SHM of similar structural systems with different types of sensors.     

基于计算机视觉技术的结构表面裂缝检测方法研究

计算机视觉技术用于混凝土结构表面裂缝检测,具有现场检测方便、效率高、客观性强的特点,但图像数据分析是该技术的核心,其中裂缝提取与定量测量较为复杂。为提高裂缝图像处理效率和准确率,将深度学习和数字图像处理技术相结合,提出一种裂缝检测方法。建立基于深度卷积神经网络的裂缝识别模型,在图像上自动定位裂缝并结合图像局域阈值分割方法提取裂缝。在裂缝宽度定量测量方面,采用双边滤波算法和三段线性变换对裂缝图像进行预处理,提高了裂缝边缘识别的精确度。通过改进边缘梯度法,实现裂缝最大宽度的定位和裂缝最大宽度的自动获取。该研究为全自动识别裂缝图像及高精度测量裂缝宽度提供了一种解决方法。     

Deep domain adaptation based video smoke detection using synthetic smoke images

In this paper, a deep domain adaptation based method for video smoke detection is proposed to extract a powerful feature representation of smoke. Due to the smoke image samples limited in scale and diversity for deep CNN training, we systematically produced adequate synthetic smoke images with a wide variation in the smoke shape, background and lighting conditions. Considering that the appearance gap (dataset bias) between synthetic and real smoke images degrades significantly the performance of the trained model on the test set composed fully of real images, we build deep architectures based on domain adaptation to confuse the distributions of features extracted from synthetic and real smoke images. This approach expands the domain-invariant feature space for smoke image samples. With their approximate feature distribution separated from non-smoke images, the recognition rate of the trained model is improved significantly compared with the model trained directly on mixed dataset of synthetic and real images. Experimentally, several deep architectures with different design choices are applied to the smoke detector. The ultimate framework can get a satisfactory result on the test set. We believe that our method own strong robustness and may offer a new way for the video smoke detection.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

Beamlet Transform‐Based Technique for Pavement Crack Detection and Classification

Abstract: This article presents a Beamlet transform‐based approach to automatically detect and classify pavement cracks in digital images. The proposed method uses a pavement distress image enhancement algorithm to correct the nonuniform background illumination by calculating the multiplicative factors that eliminate the background lighting variation. To extract linear features such as surface cracks from the pavement images, the image is partitioned into small windows and a Beamlet transform‐based algorithm is applied. The crack segments are then linked together and classified into four types: vertical, horizontal, transversal, and block. Simulation results show that the method is effective and robust in the extraction of cracks on a variety of pavement images.     

Tunnel boring machine vibration-based deep learning for the ground identification of working faces

Tunnel boring machine (TBM) vibration induced by cutting complex ground contains essential information that can help engineers evaluate the interaction between a cutterhead and the ground itself. In this study, deep recurrent neural networks (RNNs) and convolutional neural networks (CNNs) were used for vibration-based working face ground identification. First, field monitoring was conducted to obtain the TBM vibration data when tunneling in changing geological conditions, including mixed-face, homogeneous, and transmission ground. Next, RNNs and CNNs were utilized to develop vibration-based prediction models, which were then validated using the testing dataset. The accuracy of the long short-term memory (LSTM) and bidirectional LSTM (Bi-LSTM) models was approximately 70% with raw data; however, with instantaneous frequency transmission, the accuracy increased to approximately 80%. Two types of deep CNNs, GoogLeNet and ResNet, were trained and tested with time-frequency scalar diagrams from continuous wavelet transformation. The CNN models, with an accuracy greater than 96%, performed significantly better than the RNN models. The ResNet-18, with an accuracy of 98.28%, performed the best. When the sample length was set as the cutterhead rotation period, the deep CNN and RNN models achieved the highest accuracy while the proposed deep CNN model simultaneously achieved high prediction accuracy and feedback efficiency. The proposed model could promptly identify the ground conditions at the working face without stopping the normal tunneling process, and the TBM working parameters could be adjusted and optimized in a timely manner based on the predicted results.     

A convolutional neural‐network‐based pedestrian counting model for various crowded scenes

Pedestrian counting from unconstrained images is an important task in various applications such as resource management, transportation engineering, urban design, and advertising, but it is greatly challenged by some factors such as interocclusion, cross‐scene, scale, and scene perspective distortion. Traditional image‐based methods suffer from them, and the performance of conventional sensor‐based methods such as Kinect and LASER degrades gradually with the increase in pedestrian count and distance from the device to pedestrians. Based on these challenges, this paper proposes a new network model making use of stacked multicolumn convolutional neural networks (CNNs) for pedestrian counting. The human's head features are used to replace the whole body for solving the problem of serious occlusion and choose multicolumn CNNs for dealing with scale and scene perspective distortion. Also, pretrained VGG‐16 is used to generate deeper detailed features and expand the receptive field of the model. Extensive analysis and experiments on current major pedestrian counting datasets show that the proposed network model has considerable advantages in pedestrian counting tasks compared to other state‐of‐the‐art models, and the proposed network model has an improvement effect for the training process. Moreover, the visual differences between the generated density map and ground‐truth density map are visualized and analyzed quantitatively to demonstrate the feasibility of the model.