Autonomous damage segmentation and measurement of glazed tiles in historic buildings via deep learning

Detecting and measuring the damage on historic glazed tiles plays an important role in the maintenance and protection of historic buildings. However, the current visual inspection method for identifying and assessing superficial damage on historic buildings is time and labor intensive. In this article, a novel two‐level object detection, segmentation, and measurement strategy for large‐scale structures based on a deep‐learning technique is proposed. The data in this study are from the roof images of the Palace Museum in China. The first level of the model, which is based on the Faster region‐based convolutional neural network (Faster R‐CNN), automatically detects and crops two types of glazed tile photographs from 100 roof images (2,488 × 3,264 pixels). The average precision values (AP) for roll roofing and pan tiles are 0.910 and 0.890, respectively. The cropped images are used to form a dataset for training a Mask R‐CNN model. The second level of the model, which is based on Mask R‐CNN, automatically segments and measures the damage based on the cropped historic tile images; the AP for the damage segmentation is 0.975. Based on Mask R‐CNN, the predicted pixel‐level damage segmentation result is used to quantitatively measure the morphological features of the damage, such as the damage topology, area, and ratio. To verify the performance of the proposed method, a comparative study was conducted with Mask R‐CNN and a fully convolutional network. This is the first attempt at employing a two‐level strategy to automatically detect, segment, and measure large‐scale superficial damage on historic buildings based on deep learning, and it achieved good results.     

Vibration‐based structural state identification by a 1‐dimensional convolutional neural network

Deep learning has ushered in many breakthroughs in vision‐based detection via convolutional neural networks (CNNs), but the vibration‐based structural damage detection by CNN remains being refined. Thus, this study proposes a simple one‐dimensional CNN that detects tiny local structural stiffness and mass changes, and validates the proposed CNN on actual structures. Three independent acceleration databases are established based on a T‐shaped steel beam, a short steel girder bridge (in test field), and a long steel girder bridge (in service). The raw acceleration data are not pre‐processed and are directly used as the training and validation data. The well‐trained CNN almost perfectly identifies the locations of small local changes in the structural mass and stiffness, demonstrating the high sensitivity of the proposed simple CNN to tiny structural state changes in actual structures. The convolutional kernels and outputs of the convolutional and max pooling layers are visualized and discussed as well.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects

Tunnel lining defects are an important indicator reflecting the safety status of shield tunnels. Inspired by the state‐of‐the‐art deep learning, a method for automatic intelligent classification and detection methodology of tunnel lining defects is presented. A fully convolutional network (FCN) model for classification is proposed. Information about defects, collected using charge‐coupled device cameras, was used to train the model. The model's performance was compared to those of GoogLeNet and VGG. The best‐set accuracy of the proposed model was over 95% at a test‐time speed of 48 ms per image. For defects detection, image features were computed from large‐scale images by the FCN and then detected using a region proposal network and position‐sensitive region of interest pooling. Some indices (detection rate, detection accuracy, and detection efficiency, locating accuracy) were used to evaluate the model. The comparisons with faster R‐CNN and a traditional method were conducted. The results show that the model is very fast and efficient, allowing automatic intelligent classification and detection of tunnel lining defects.     

基于WIM的典型城市桥梁车辆荷载状况分析

基于安装在杭州石祥路留石高架上的动态称重系统（WIM）采集的交通流和车辆荷载数据,通过对交通流量、车辆构成、轴重、车辆总重等进行分析,获取了其概率统计特性及分布规律,建立了轴载谱和车辆总重谱,并与相关文献、规范进行了比较。结果表明：一天中各时段的交通流量具有很强的潮汐规律性;2.86%的车辆轴重超过了10 t的超载标准,最大的单轴重量达39.5 t,最大车辆总重达115 t,后半夜车辆恶性超载严重;2轴车车辆总重表现为单峰分布,3轴以上车辆的总重均表现为多峰分布,5轴车和6轴车的总重表现为3峰分布;轴重大的车辆主要是超载的2轴、3轴车辆,这部分车辆对桥梁等基础设施的危害较大。这些车辆荷载特征的获取有助于区域既有桥梁安全性的合理评估,为超重车辆治理和桥梁安全管理提供帮助。     

基于时空马尔可夫随机场模型的车辆跟踪算法研究

鲁棒的车辆跟踪是实现交通事件自动检测的重要前提,车辆跟踪中的车辆相互遮挡则是影响车辆跟踪结果的关键因素。针对这一难题,设计自适应的车辆跟踪算法,并依据交通图像序列的时空相关性,根据马尔可夫的基本理论和贝叶斯方法,应用MRF-MAP理论分析框架,并结合了彩色图像序列的纹理信息建立了图像序列的时空马尔可夫随机场模型。采用随机松弛算法中的Metropolis算法来求解时空马尔可夫随机场模型,对车辆跟踪得到的目标标号图进行优化,从而解决车辆跟踪中的遮挡问题。初步实验结果,跟踪不遮挡的车辆时达到的跟踪成功率为95%。遮挡情况时成功率也可达到83%。实验结果表明,该跟踪算法在不遮挡时效果非常理想,在遮挡情况下跟踪鲁棒性也较好。     

Vehicle models for fatigue loading on steel box-girder bridges based on weigh-in-motion data

On the basis of massive data collected by the Weigh-in-motion (WIM) system of XiHouMen Bridge, analysis of fatigue-loaded vehicle models and theoretical fatigue life of U ribs butt weld of steel box girder are discussed in this paper. Firstly, basic vehicle information can be obtained from preliminary statistics of various types of vehicles data and the vehicles passing XiHouMen Bridge are divided into seven types based on the number of axles and axle groups. Secondly, the statistical distribution functions of gross vehicle weight under different loading conditions and wheelbase for each type of vehicle are developed to obtain the equivalent fatigue vehicle load. Then, the parameters of various types of fatigue-loaded vehicles are determined by combining the mathematic expectation of gross vehicle weight distribution with Palmgren-Miner fatigue damage accumulation theory. Finally, the seven types of fatigue-loaded vehicles are applied to the component-coupled finite element model of the steel box girders in XiHouMen Bridge and the theoretical fatigue life of the U ribs butt weld can be estimated. This research provides a reference to define the vehicle models for fatigue loading on steel box-girder bridges based on WIM data.     

Detecting and tracking vehicles in traffic by unmanned aerial vehicles

Using unmanned aerial vehicles (UAV) as devices for traffic data collection exhibits many advantages in collecting traffic information. This paper introduces a new vehicle detecting and tracking system based on image data collected by UAV. This system uses consecutive frames to generate vehicle's dynamic information, such as positions and velocities. Four major modules have been developed: image registration, image feature extraction, vehicle shape detecting, and vehicle tracking. Some unique features have been introduced into this system to customize the vehicle and traffic flow and to jointly use them in multiple consecutive images to increase the system accuracy of detecting and tracking vehicles. Field tests demonstrate that the present system exhibits high accuracy in traffic information acquisition at different UAV altitudes with different view scopes, which can be used in future traffic monitoring and control in metropolitan areas.     

Graph‐Based Modeling of Information Flow Evolution and Propagation under V2V Communications‐Based Advanced Traveler Information Systems

A vehicle equipped with a vehicle‐to‐vehicle (V2V) communications capability can continuously update its knowledge on traffic conditions using its own experience and anonymously obtained travel experience data from other such equipped vehicles without any central coordination. In such a V2V communications‐based advanced traveler information system (ATIS), the dynamics of traffic flow and intervehicle communication lead to the time‐dependent vehicle knowledge on the traffic network conditions. In this context, this study proposes a graph‐based multilayer network framework to model the V2V‐based ATIS as a complex system which is composed of three coupled network layers: a physical traffic flow network, and virtual intervehicle communication and information flow networks. To determine the occurrence of V2V communication, the intervehicle communication layer is first constructed using the time‐dependent locations of vehicles in the traffic flow layer and intervehicle communication‐related constraints. Then an information flow network is constructed based on events in the traffic and intervehicle communication networks. The graph structure of this information flow network enables the efficient tracking of the time‐dependent vehicle knowledge of the traffic network conditions using a simple graph‐based reverse search algorithm and the storage of the information flow network as a single graph database. Further, the proposed framework provides a retrospective modeling capability to articulate explicitly how information flow evolves and propagates. These capabilities are critical to develop strategies for the rapid flow of useful information and traffic routing to enhance network performance. It also serves as a basic building block for the design of V2V‐based route guidance strategies to manage traffic conditions in congested networks. Synthetic experiments are used to compare the graph‐based approach to a simulation‐based approach, and illustrate both memory usage and computational time efficiencies.     

基于卷积神经网络的结构损伤识别研究进展

为系统梳理基于卷积神经网络的工程结构损伤识别方法的发展脉络和研究现状,分别从结构损伤的识别目的和在不同类型结构中的应用两方面进行了归类、分析和评价。介绍了卷积神经网络的基本结构和评价指标,回顾了卷积神经网络的研究和应用历程。在损伤的识别目的方面,主要针对混凝土结构损伤的分类、定位和分割,详细介绍了基于不同类型卷积神经网络的结构损伤识别方法,即基于分类的方法、基于回归的方法和像素级的图像分割算法; 分析了各类方法所使用的卷积神经网络模型的结构特点、计算流程、训练方法和损伤识别性能。在不同类型结构的损伤识别方面,分析了卷积神经网络在砌体结构、钢结构桥梁和古建筑木结构裂缝识别中的应用。最后,基于对卷积神经网络优缺点的思考,提出了发展建议和展望。结果表明:训练样本中结构损伤的多样性对模型的损伤识别效果影响较大; 现有基于卷积神经网络的损伤分割方法模型参数较多,计算量大; 采用数据增广和迁移学习方法可有效防止模型过拟合,提高模型训练效率; 针对微小损伤和不同类型结构损伤的识别,此类方法的性能有待提高。     

Automated Pixel‐Level Pavement Crack Detection on 3D Asphalt Surfaces Using a Deep‐Learning Network

The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.     

A deep learning algorithm for simulating autonomous driving considering prior knowledge and temporal information

Autonomous vehicle (AV) stakeholders continue to seek assurance of the safety performance of this new technology through AV testing on in‐service roads, AV‐dedicated road networks, and AV test tracks. However, recent AV‐related fatalities on in‐service roads have exacerbated public skepticism and eroded some public trust in the safety of AV operations. Further, test tracks are unable to characterize adequately the real‐world driving environment. For this reason, driving simulators continue to serve as an attractive means of AV testing. However, in most AV driving simulators, the AV operation is based on commands external to the vehicle and embedded in the code for the driving environment. To address the simulation shortfalls associated with this approach, this paper develops a deep convolutional neural network–long short‐term memory (CNN–LSTM) algorithm for self‐driving simulation. This algorithm observes and characterizes the AV's driving environment, and controls the AV movement in the driving simulation. The CNN part extracts features that use transfer learning to introduce human prior knowledge, and the LSTM part uses temporal information to process the extracted features, and incorporates temporal dynamics to predict driving decisions. The AV may also use an external server with a database containing road environment data as an additional source of information. It is acknowledged that different driving simulators differ in their functions and their capabilities to access driving‐environment data. Therefore, to make it sufficiently flexible to facilitate replication by other researchers that use driving simulators, the algorithm has been designed and demonstrated using only image data of the driving environment as input. This is because roadway image data are easily and readily accessible from the screen of any driving simulator. The proposed algorithm was tested using the open racing car simulator test track platform and was found to be able to mimic human driving decisions with a high degree of accuracy.     

Effects of bridge motion and crosswind on ride comfort of road vehicles

Ride comfort of road vehicles running on the ground has been extensively investigated as one of the main criteria in the design of road vehicles. The information on the ride comfort of road vehicles running on a long span cable-supported bridge under crosswind is, however, very limited. This paper presents an evaluation of ride comfort of road vehicles running on a long span cable-stayed bridge under crosswind. The ride comfort criteria for road vehicles are briefly reviewed first. The principle of mathematical modeling and the equation of motion of a coupled road vehicle–bridge system under crosswind are then introduced. The framework proposed is finally applied to a group of high-sided road vehicles running on a real cable-stayed bridge under crosswind as a case study. Ride comfort of the high-sided road vehicle under various conditions of road roughness, vehicle speed, and crosswind speed is investigated for the vehicle running on either the ground or the cable-stayed bridge with and without crosswind. The results show that the crosswind influences the ride comfort of the vehicle in the lateral direction while the bridge motion affects the ride comfort of the vehicle in the vertical direction.     

Identification of parameters of vehicles moving on bridges

This paper presents a method for identifying the parameters of vehicles moving on bridges. Two vehicle models, a single-degree-of-freedom model and a full-scale vehicle model, are used. The vehicle–bridge coupling equations are established by combining the equations of motion of both the bridge and the vehicle using the displacement relationship and the interaction force relationship at the contact point. Bridge responses including displacement, acceleration, and strain are used in the identification process. The parameters of vehicles moving on the bridge are then identified by optimizing an objective function, which is built up using the residual between the measured response time history and predicted response time history using the Genetic Algorithm. A series of case studies have been carried out and the identified results demonstrate that the proposed method is able to identify vehicle parameters very accurately. Field tests have also been performed on an existing bridge in Louisiana, and the parameters of a real truck are predicted. Since it is able to identify the parameters of moving vehicles, the methodology can be applied to improve the current weigh-in-motion techniques that usually require a smooth road surface and slow vehicle movement to minimize the dynamic effects. The methodology can also be implemented in routine traffic monitoring and control.     

Automated Road Sign Inventory System Based on Stereo Vision and Tracking

Abstract: Detection, recognition, and positioning of road signs are critical components of a roadway asset management system. In this research, a stereo vision‐based system is developed to conduct automated road sign inventory. The system in real time integrates and synchronizes the data streams from multiple sensors of high‐resolution cameras, Differential Global Positioning System receivers, Distance Measurement Instrument, and Inertial Measurement Unit. Algorithms are developed based on data sets from the multiple positioning sensors to determine the positions of the moving vehicle and the orientation of the cameras. The key findings from the research include feature extraction and analysis that are applied for automated sign detection and recognition in the Right‐of‐Way (ROW) images, implementing a tracking algorithm of the candidate sign region among the image frames so the same signs are not counted more than once in an image sequence, and implementing stereo vision technique to compute the world coordinates of the road sign from the stereo‐paired ROW images. Particular techniques are employed to conduct all data acquisition and analysis in real time onboard the vehicle. This system is an advanced alternative to traditional inventory methods in terms of safety and efficiency.     

Evaluation of Information Applications of a Self‐Organizing Distributed Traffic Information System for a Large‐Scale Real‐World Traffic Network

Abstract: This article presents an evaluation of the system performance of a proposed self‐organizing, distributed traffic information system based on vehicle‐to‐vehicle information‐sharing architecture. Using microsimulation, several information applications derived from this system are analyzed relative to the effectiveness and efficiency of the system to estimate traffic conditions along each individual path in the network, to identify possible incidents in the traffic network, and to provide rerouting strategies for vehicles to escape congested spots in the network. A subset of vehicles in the traffic network is equipped with specific intervehicle communication devices capable of autonomous traffic surveillance, peer‐to‐peer information sharing, and self‐data processing. A self‐organizing traffic information overlay on the existing vehicular roadway network assists their independent evaluation of route information, detection of traffic incidents, and dynamic rerouting in the network based both on historical information stored in an in‐vehicle database and on real‐time information disseminated through intervehicle communications. A path‐based microsimulation model is developed for these information applications and the proposed distributed traffic information system is tested in a large‐scale real‐world network. Based on simulation study results, potential benefits both for travelers with such equipment as well as for the traffic system as a whole are demonstrated.     

Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization

Many bridge structures, one of the most critical components in transportation infrastructure systems, exhibit signs of deteriorations and are approaching or beyond the initial design service life. Therefore, structural health inspections of these bridges are becoming critically important, especially after extreme events. To enhance the efficiency of such an inspection, in recent years, autonomous damage detection based on computer vision has become a research hotspot. This article proposes a three‐level image‐based approach for post‐disaster inspection of the reinforced concrete bridge using deep learning with novel training strategies. The convolutional neural network for image classification, object detection, and semantic segmentation are, respectively, proposed to conduct system‐level failure classification, component‐level bridge column detection, and local damage‐level damage localization. To enable efficient training and prediction using a small data set, the model robustness is a crucial aspect to be taken into account, generally through its hyperparameters’ selection. This article, based on Bayesian optimization, proposes a principled manner of such selection, with which very promising results (well over 90% accuracies) and robustness are observed on all three‐level deep learning models.     

Autonomous UAVs for Structural Health Monitoring Using Deep Learning and an Ultrasonic Beacon System with Geo‐Tagging

Visual inspection has traditionally been used for structural health monitoring. However, assessments conducted by trained inspectors or using contact sensors on structures for monitoring are costly and inefficient because of the number of inspectors and sensors required. To date, data acquisition using unmanned aerial vehicles (UAVs) equipped with cameras has become popular, but UAVs require skilled pilots or a global positioning system (GPS) for autonomous flight. Unfortunately, GPS cannot be used by a UAV for autonomous flight near some parts of certain structures (e.g., beneath a bridge), but these are the critical locations that should be inspected to monitor and maintain structural health. To address this difficulty, this article proposes an autonomous UAV method using ultrasonic beacons to replace the role of GPS, a deep convolutional neural network (CNN) for damage detection, and a geo‐tagging method for the localization of damage. Concrete cracks, as an example of structural damage, were successfully detected with 97.7% specificity and 91.9% sensitivity, by processing video data collected from an autonomous UAV.     

Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks

A number of image processing techniques (IPTs) have been implemented for detecting civil infrastructure defects to partially replace human‐conducted onsite inspections. These IPTs are primarily used to manipulate images to extract defect features, such as cracks in concrete and steel surfaces. However, the extensively varying real‐world situations (e.g., lighting and shadow changes) can lead to challenges to the wide adoption of IPTs. To overcome these challenges, this article proposes a vision‐based method using a deep architecture of convolutional neural networks (CNNs) for detecting concrete cracks without calculating the defect features. As CNNs are capable of learning image features automatically, the proposed method works without the conjugation of IPTs for extracting features. The designed CNN is trained on 40 K images of 256 × 256 pixel resolutions and, consequently, records with about 98% accuracy. The trained CNN is combined with a sliding window technique to scan any image size larger than 256 × 256 pixel resolutions. The robustness and adaptability of the proposed approach are tested on 55 images of 5,888 × 3,584 pixel resolutions taken from a different structure which is not used for training and validation processes under various conditions (e.g., strong light spot, shadows, and very thin cracks). Comparative studies are conducted to examine the performance of the proposed CNN using traditional Canny and Sobel edge detection methods. The results show that the proposed method shows quite better performances and can indeed find concrete cracks in realistic situations.