Autonomous UAVs for Structural Health Monitoring Using Deep Learning and an Ultrasonic Beacon System with Geo‐Tagging

Visual inspection has traditionally been used for structural health monitoring. However, assessments conducted by trained inspectors or using contact sensors on structures for monitoring are costly and inefficient because of the number of inspectors and sensors required. To date, data acquisition using unmanned aerial vehicles (UAVs) equipped with cameras has become popular, but UAVs require skilled pilots or a global positioning system (GPS) for autonomous flight. Unfortunately, GPS cannot be used by a UAV for autonomous flight near some parts of certain structures (e.g., beneath a bridge), but these are the critical locations that should be inspected to monitor and maintain structural health. To address this difficulty, this article proposes an autonomous UAV method using ultrasonic beacons to replace the role of GPS, a deep convolutional neural network (CNN) for damage detection, and a geo‐tagging method for the localization of damage. Concrete cracks, as an example of structural damage, were successfully detected with 97.7% specificity and 91.9% sensitivity, by processing video data collected from an autonomous UAV.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization

Many bridge structures, one of the most critical components in transportation infrastructure systems, exhibit signs of deteriorations and are approaching or beyond the initial design service life. Therefore, structural health inspections of these bridges are becoming critically important, especially after extreme events. To enhance the efficiency of such an inspection, in recent years, autonomous damage detection based on computer vision has become a research hotspot. This article proposes a three‐level image‐based approach for post‐disaster inspection of the reinforced concrete bridge using deep learning with novel training strategies. The convolutional neural network for image classification, object detection, and semantic segmentation are, respectively, proposed to conduct system‐level failure classification, component‐level bridge column detection, and local damage‐level damage localization. To enable efficient training and prediction using a small data set, the model robustness is a crucial aspect to be taken into account, generally through its hyperparameters’ selection. This article, based on Bayesian optimization, proposes a principled manner of such selection, with which very promising results (well over 90% accuracies) and robustness are observed on all three‐level deep learning models.     

Cognitive infrastructure - a modern concept for resilient performance under extreme events

The increasing frequency and intensity of natural disasters, as well as escalation of manmade threats, are posing significant threats to the built environment. Further, much of civil infrastructure in developed countries, built after World War II, is experiencing age-related deterioration and thus are vulnerable to damage under extreme events. This vulnerability of infrastructure under severe loading conditions can be assessed through a coupled sensing-structural framework that extends principles of the recently developed “Internet of Things” (IoT) technology into civil infrastructure. This concept aims at monitoring key response parameters (i.e. temperature, strain, deformation, vibration levels etc.) by incorporating cognitive abilities into a structure through interaction of various sensing devices and socio-environmental factors. These response parameters can be utilized to trace performance of critical infrastructure during the course of a disaster so as to predict signs of imminent failure and to provide first responders and occupants with much needed situational awareness. The practicality of the proposed concept in enhancing resilience of new and existing infrastructure is illustrated through two case studies.     

Automated classification of civil structure defects based on convolutional neural network

Today, the most commonly used civil infrastructure inspection method is based on a visual assessment conducted by certified inspectors following prescribed protocols. However, the increase in aggressive environmental and load conditions, coupled with the achievement of many structures of the life-cycle end, has highlighted the need to automate damage identification and satisfy the number of structures that need to be inspected. To overcome this challenge, this paper presents a method for automating concrete damage classification using a deep convolutional neural network. The convolutional neural network was designed after an experimental investigation of a wide number of pretrained networks, applying the transfer-learning technique. Training and validation were conducted using a database built with 1352 images balanced between “undamaged”, “cracked”, and “delaminated” concrete surfaces. To increase the network robustness compared to images in real-world situations, different image configurations have been collected from the Internet and on-field bridge inspections. The GoogLeNet model, with the highest validation accuracy of approximately 94%, was selected as the most suitable network for concrete damage classification. The results confirm that the proposed model can correctly classify images from real concrete surfaces of bridges, tunnels, and pavement, resulting in an effective alternative to the current visual inspection techniques.     

Prompt engineering for zero-shot and few-shot defect detection and classification using a visual-language pretrained model

Zero-shot learning, applied with vision-language pretrained (VLP) models, is expected to be an alternative to existing deep learning models for defect detection, under insufficient dataset. However, VLP models, including contrastive language-image pretraining (CLIP), showed fluctuated performance on prompts (inputs), resulting in research on prompt engineering—optimization of prompts for improving performance. Therefore, this study aims to identify the features of a prompt that can yield the best performance in classifying and detecting building defects using the zero-shot and few-shot capabilities of CLIP. The results reveal the following: (1) domain-specific definitions are better than general definitions and images; (2) a complete sentence is better than a set of core terms; and (3) multimodal information is better than single-modal information. The resulting detection performance using the proposed prompting method outperformed that of existing supervised models.     

Automatic recognition of asphalt pavement cracks using metaheuristic optimized edge detection algorithms and convolution neural network

Crack detection is a crucial task in periodic pavement survey. This study establishes and compares the performance of two intelligent approaches for automatic recognition of pavement cracks. The first model relies on edge detection approaches of the Sobel and Canny algorithms. Since the implementation of the two edge detectors require the setting of threshold values, Differential Flower Pollination, as a metaheuristic, is employed to fine-tune the model parameters. The second model is constructed by the implementation of the Convolution Neural Network (CNN) – a deep learning algorithm. CNN has the advantage of performing the feature extraction and the prediction of crack/non-crack condition in an integrated and fully automated manner. Experimental results show that the model based on CNN achieves a good prediction performance of Classification Accuracy Rate (CAR) = 92.08%. This performance is significantly better than the method based on the edge detection algorithms (CAR = 79.99%). Accordingly, the proposed CNN based crack detection model is a promising alternative to support transportation agencies in the task of periodic pavement inspection.     

Integrating IoT into operational workflows for real-time and automated decision-making in repetitive construction operations

Construction operations are typically spread across large areas and require remote collaboration between multiple disparate resources—characteristics that create logistical challenges for making and automating decisions on the worksite. This paper provides a framework for leveraging the growing ubiquity of devices that can be considered part of the internet of things (IoT) to inform real-time decision-making on the construction site. Specifically, systems and control theory concepts are implemented by first synthesizing sensor information at their point of origin into resource state, and then using this state as feedback into a process model of the operation. Decisions that are programed into the process model are then made automatically based on real-time status of the operation and relayed back to the entities on the construction site through the IoT infrastructure. The real-time decision-making capabilities enabled by the presented methodology and its associated benefits to construction performance are demonstrated through the use of a virtual experimental platform that simulates a potential implementation of IoT-enabled control on the construction worksite for an earthmoving operation. This research provides a practical and sensor-agnostic implementation of operation-level decision-making by utilizing IoT networks along with advancements in modeling and simulation tools. This paper illustrates the types of insights that can be synthesized from an operations-level IoT network that gathers and transmits information in real time from various locations of the worksite. Currently and without the use of the prescribed methodology, such actionable insight would be impractical, cost-prohibitive, and even impossible to obtain.     

Towards autonomous sewer robots: the MAKRO project

We expose current research and development in information technology that deals with building and controlling autonomous service robots for performing inspection tasks in sewerage systems that are inaccessible for humans. The problem is explained, the physical and legal boundary conditions for operating sewer robots are described, the state of current sewer inspection technology is sketched, and the need for using advanced technology in this area motivated. Respective work that we have been doing for the last four years, and that is currently being pursued in the MAKRO project, is presented. The main technological and control problems are described, and how Artificial Intelligence research and technology may be employed to solve the latter.     

Unified Vision‐Based Methodology for Simultaneous Concrete Defect Detection and Geolocalization

Vision‐based autonomous inspection of concrete surface defects is crucial for efficient maintenance and rehabilitation of infrastructures and has become a research hot spot. However, most existing vision‐based inspection methods mainly focus on detecting one kind of defect in nearly uniform testing background where defects are relatively large and easily recognizable. But in the real‐world scenarios, multiple types of defects often occur simultaneously. And most of them occupy only small fractions of inspection images and are swamped in cluttered background, which easily leads to missed and false detections. In addition, the majority of the previous researches only focus on detecting defects but few of them pay attention to the geolocalization problem, which is indispensable for timely performing repair, protection, or reinforcement works. And most of them rely heavily on GPS for tracking the locations of the defects. However, this method is sometimes unreliable within infrastructures where the GPS signals are easily blocked, which causes a dramatic increase in searching costs. To address these limitations, we present a unified and purely vision‐based method denoted as defects detection and localization network, which can detect and classify various typical types of defects under challenging conditions while simultaneously geolocating the defects without requiring external localization sensors. We design a supervised deep convolutional neural network and propose novel training methods to optimize its performance on specific tasks. Extensive experiments show that the proposed method is effective with a detection accuracy of 80.7% and a localization accuracy of 86% at 0.41 s per image (at a scale of 1,200 pixels in the field test experiment), which is ideal for integration within intelligent autonomous inspection systems to provide support for practical applications.     

Recording of bridge damage areas by 3D integration of multiple images and reduction of the variability in detected results

Machine learning models have been developed to perform damage detection using images to improve bridge inspection efficiency. However, in damage detection using images alone, the 3D coordinates of the damage cannot be recorded. Furthermore, the accuracy of the detection depends on the quality of the images. This paper proposes a method to integrate and record the damage detected from multiple images into a 3D model using deep learning to detect the damage from bridge images and structure from motion to identify the shooting position. The proposed method reduces the variability of the detection results between images and can assess the scale of damage or, conversely, where there is no damage and the extent of inspection omissions. The proposed method has been applied to a real bridge, and it has been shown that the actual damage locations can be recorded as a 3D model.     

Segment identification in water distribution systems

This paper presents a new method for identifying the segments that are formed after the installation and closure of isolation valves in a water distribution network. This method is able to identify segments also when one-way devices are installed in the network. Thanks to its short computing times, the method enables the analysis of real networks which always comprise a large number of nodes and pipes.

The numerical examples presented in this paper refer to two real water distribution networks. The first network is a part of a provincial network where two one-way devices are present; the second is a complex urban network without one-way devices. The method was first used to analyse the existing situation in both networks, i.e. the set of segments that are formed as a consequence of the present valve system. The method was subsequently used for the problem of the hypothetic redesign of the isolation valve system in the second urban network, i.e. the search for the optimal positions of the isolation valves in the network; in the redesign phase it provided solutions which are more cost-effective than the configuration of isolation valves currently present, the level of water service reliability being the same.     

Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network

Deep learning‐based structural damage detection methods overcome the limitation of inferior adaptability caused by extensively varying real‐world situations (e.g., lighting and shadow changes). However, most deep learning‐based methods detect structural damage at the image level and grid‐cell level. To provide pixel‐level detection of multiple damages, a Fully Convolutional Network (FCN)‐based multiple damages detection method for concrete structure is proposed. To realize this method, a database of 2,750 images (with 504 × 376 pixels) including crack, spalling, efflorescence, and hole images in concrete structure is built, and the four damages included in those images are labeled manually. Then, the architecture of the FCN is modified, trained, validated, and tested using this database. A strategy of model‐based transfer learning is used to initialize the parameters of the FCN during the training process. The results show 98.61% pixel accuracy (PA), 91.59% mean pixel accuracy (MPA), 84.53% mean intersection over union (MIoU), and 97.34% frequency weighted intersection over union (FWIoU). Subsequently, the robustness and adaptability of the trained FCN model is tested and the damage is extracted, where damage areas are provided according to a calibrated relation between the ratio (the pixel area and true area of the detected object) and the distance from the smartphone to the concrete surface using a laser range finder. A comparative study is conducted to examine the performance of the proposed FCN‐based approach using a SegNet‐based method. The results show that the proposed method substantiates quite better performance and can indeed detect multiple concrete damages at the pixel level in realistic situations.     

The last two decades of computer vision technologies in water resource management: A bibliometric analysis

Efficient management of water resources is an important task given the significance of water in daily lives and economic growth. Water resource management is a specific field of study which deals with the efficient management of water resources towards fulfilling the needs of society and preventing from water-related disasters. Many activities within this domain are getting benefitted with the recent technological advancements. Within many others, computer vision-based solutions have emerged as disruptive technologies to address complex real-world problems within the water resource management domain (e.g., flood detection and mapping, satellite-based water bodies monitoring, monitoring and inspection of hydraulic structures, blockage detection and assessment, drainage inspection and sewer monitoring). However, there are still many aspects within the water resource management domain which can be explored using computer vision technologies. Therefore, it is important to investigate the trends in current research related to these technologies to inform the new researchers in this domain. In this context, this paper presents the bibliometric analysis of the literature from the last two decades where computer vision technologies have been used for addressing problems within the water resource management domain. The analysis is presented in two categories: (a) performance analysis demonstrating highlighted trends in the number of publications, number of citations, top contributing countries, top publishing journals, top contributing institutions and top publishers and (b) science mapping to demonstrate the relation between the bibliographic records based on the co-occurrence of keywords, co-authorship analysis, co-citation analysis and bibliographic coupling analysis. Bibliographic records (i.e., 1059) are exported from the Web of Science (WoS) core collection database using a comprehensive query of keywords. VOSviewer opensource tool is used to generate the network and overlay maps for the science mapping of bibliographic records. Results highlighted important trends and valuable insights related to the use of computer vision technologies in water resource management. An increasing trend in the number of publications and focus on deep learning/artificial intelligence (AI)-based approaches has been reported from the analysis. Further, flood mapping, crack/fracture detection, coastal flood detection, blockage detection and drainage inspections are highlighted as active areas of research.     

High-resolution model reconstruction and bridge damage detection based on data fusion of unmanned aerial vehicles light detection and ranging data imagery

Damage detection is essential for the maintenance of transportation infrastructure that experiences high daily traffic levels in potentially extreme environments and changes in use patterns. However, traditional physical inspection is always labor-intensive, subjective, and biased, lacking the objective perspective required for a comprehensive and reliable assessment. Recently, unmanned aerial vehicles (UAVs) combined with emerging high-performance sensor(s) have been intensively researched. Here, we present an aerial bridge surface survey method that can be used to assess damage. Existing damage detection methods focus on single types of damage and are limited in locating global damage, whereas our method detects two types of damage on the surface and marks them in a panoramic image. The workflow involves three steps: data acquisition using a meticulous UAV flight strategy that covers the entire surface, data processing using image-based and point-cloud models after polynomial rectification, and data output (i.e., damage detected by the combined models). To verify the method, a field test detected damage to two real bridges. A UAV equipped with a camera and light detection and ranging (LiDAR) equipment was employed. Experiments demonstrate the effectiveness of the proposed method, which is capable of producing accurate outputs and detecting damage with an average position error of 13.37 mm and a relative size error of 25.25%. Owing to the data fusion model taking advantage of two-dimensional (2D) images and 3D LiDAR data, it outputs a high-resolution 3D model and avoids environmental disturbances. After decision-making-level damage fusion, all position and size properties of damage information are computed into a panoramic damage image. This panoramic image showing all detecting damage helps technicians perform maintenance; the image can be zoomed to focus on any issue individually.     

Parametric instability of edge cracked plates

The presented paper deals with the parametric instability behavior of a simply supported rectangular plate with a crack emanating from one edge, subjected to in-plane compressive periodic edge loading. The problem is reduced to computing the free vibration frequencies and the corresponding mode shapes and substituting them into an integral equation based formula, which leads to a compact matrix form. Once the components of this matrix are found, the rest of the computation, i.e., establishing regions of instability, buckling loads and modified frequencies, is straightforward and fast. Several plates, each with a different dimension and crack length size are analyzed using this approach. The comparison of results with those of finite element models is found to be in close agreement.     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.     

建筑物外立面损伤检测智能化发展趋势

为梳理建筑外立面主要损伤检测方法的优缺点并探究其发展趋势,通过文献调研与分析,总结了建筑物外立面损伤常见表现形式并分析了主要原因,回顾了建筑物外立面损伤检测技术发展过程,阐述了基于数字化和智能化的新型检测方法,分析了其适用条件和优缺点。结果表明:目前外立面损伤检测主要采用目测法、锤击法等传统方法,检测效率低,成本高,需7 d以上出具检测报告; 通过数字化和智能化手段进行检测在一定程度上提高了效率,但最终决策仍需有经验的专业人员进行判断; 外立面损伤检测的发展趋势是软件上开发能够快速进行损伤识别与判断的人工智能算法,将出具检测报告时间缩短至几小时甚至即时出具,硬件上开发适用于特定目标的机器人和检测模块。     

A Bayesian Probabilistic Approach for Crack Characterization in Plate Structures

Abstract: This article explores the possibility of using a Bayesian probabilistic approach for the detection of cracks in thin plate structures, utilizing measured dynamic responses at only a few points on the plate. Existing laser scanning or shearography based crack detection methods are applicable only when measurement at the region near the defect is possible. These types of techniques are important in providing information, in addition to that obtained through visual inspection, for the purpose of structural health monitoring. Because of the global nature of the vibration characteristics of structural systems, this article puts forward a crack detection approach that can be applicable with only a few sensors and when the sensor locations are not close to the crack. This kind of method is particularly valuable as it can be applied when visual inspection is not possible (e.g., part of the plate is obstructed and is not assessable by inspectors). Owing to the problems of measurement noise and incomplete measurement (i.e., only a limited number of measurement points are employed and high‐mode information is lost because of the digitization of the signal and measurement noise), the results of crack detection as an inverse problem contain uncertainties. To explicitly handle such uncertainty, the proposed crack detection method follows the Bayesian statistical system identification framework. Rather than pinpointing the crack parameters (i.e., the crack location, length, and depth), the posterior probability density function (PDF) of the crack parameters is calculated to quantify the confidence level of the identified results, which is extremely important for engineers when they make judgments about remedial works. This article reports the theoretical development of the modeling of a cracked plate and a crack detection method together with numerical verification of the proposed method. The case study results are very encouraging, and indicate that the proposed method is feasible.     

Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.