Image‐based post‐disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization

Many bridge structures, one of the most critical components in transportation infrastructure systems, exhibit signs of deteriorations and are approaching or beyond the initial design service life. Therefore, structural health inspections of these bridges are becoming critically important, especially after extreme events. To enhance the efficiency of such an inspection, in recent years, autonomous damage detection based on computer vision has become a research hotspot. This article proposes a three‐level image‐based approach for post‐disaster inspection of the reinforced concrete bridge using deep learning with novel training strategies. The convolutional neural network for image classification, object detection, and semantic segmentation are, respectively, proposed to conduct system‐level failure classification, component‐level bridge column detection, and local damage‐level damage localization. To enable efficient training and prediction using a small data set, the model robustness is a crucial aspect to be taken into account, generally through its hyperparameters’ selection. This article, based on Bayesian optimization, proposes a principled manner of such selection, with which very promising results (well over 90% accuracies) and robustness are observed on all three‐level deep learning models.     

Zernike‐moment measurement of thin‐crack width in images enabled by dual‐scale deep learning

Although crack inspection is a routine practice in civil infrastructure management (especially for highway bridge structures), it is time‐consuming and safety‐concerning to trained engineers and costly to the stakeholders. To automate this in the near future, the algorithmic challenge at the onset is to detect and localize cracks in imagery data with complex scenes. The rise of deep learning (DL) sheds light on overcoming this challenge through learning from imagery big data. However, how to exploit DL techniques is yet to be fully explored. One primary component of practical crack inspection is that it is not merely detection via visual recognition. To evaluate the potential risk of structural failure, it entails quantitative characterization, which usually includes crack width measurement. To further facilitate the automation of machine‐vision‐based concrete crack inspection, this article proposes a DL‐enabled quantitative crack width measurement method. In the detection and mapping phase, dual‐scale convolutional neural networks are designed to detect cracks in complex scene images with validated high accuracy. Subsequently, a novel crack width estimation method based on the use of Zernike moment operator is further developed for thin cracks. The experimental results based on a laboratory loading test agree well with the direct measurements, which substantiates the effectiveness of the proposed method for quantitative crack detection.     

Concrete bridge surface damage detection using a single‐stage detector

Early and timely detection of surface damages is important for maintaining the functionality, reliability, and safety of concrete bridges. Recent advancement in convolution neural network has enabled the development of deep learning‐based visual inspection techniques for detecting multiple structural damages. However, most deep learning‐based techniques are built on two‐stage, proposal‐driven detectors using less complex image data, which could be restricted for practical applications and possible integration within intelligent autonomous inspection systems. In this study, a faster, simpler single‐stage detector is proposed based on a real‐time object detection technique, You Only Look Once (YOLOv3), for detecting multiple concrete bridge damages. A field inspection images dataset labeled with four types of concrete damages (crack, pop‐out, spalling, and exposed rebar) is used for training and testing of YOLOv3. To enhance the detection accuracy, the original YOLOv3 is further improved by introducing a novel transfer learning method with fully pretrained weights from a geometrically similar dataset. Batch renormalization and focal loss are also incorporated to increase the accuracy. Testing results show that the improved YOLOv3 has a detection accuracy of up to 80% and 47% at the Intersection‐over‐Union (IoU) metrics of 0.5 and 0.75, respectively. It outperforms the original YOLOv3 and the two‐stage detector Faster Region‐based Convolutional Neural Network (Faster R‐CNN) with ResNet‐101, especially for the IoU metric of 0.75.     

A sematic and prior‐knowledge‐aided monocular localization method for construction‐related entities

The real‐time location of construction‐related entities is some of the most useful basic information for automated construction management. However, the implementation of most existing localization methods is limited due to the weak adaptability to construction sites. In this paper, we enhance the monocular vision technique for the localization of construction‐related entities by a sematic and prior knowledge‐based method. A deep learning algorithm is employed to segment the sematic instance in the images, and then the prior knowledge model specifies projection strategies for entities corresponding to various scenarios. Results show that the proposed method achieves satisfying positioning accuracy, is robust in low‐ratio occlusions, and can help facilitate safety early warning, activity recognition, and productivity analysis.     

Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system

Crack information provides important evidence of structural degradation and safety in civil structures. Existing inspection methods are inefficient and difficult to rapidly deploy. A real‐time crack inspection method is proposed in this study to address this difficulty. Within this method, a wall‐climbing unmanned aerial system (UAS) is developed to acquire detailed crack images without distortion, then a wireless data transmission method is applied to fulfill real‐time detection requirements, allowing smartphones to receive real‐time video taken from the UAS. Next, an image data set including 1,330 crack images taken by the wall‐climbing UAS is established and used for training a deep‐learning model. For increasing detection speed, state‐of‐the‐art convolutional neural networks (CNNs) are compared and employed to train the crack detector; the selected model is transplanted into an android application so that the detection of cracks can be undertaken on a smartphone in real time. Following this, images with cracks are separated and crack width is calculated using an image processing method. The proposed method is then applied to a building where crack information is acquired and calculated accurately with high efficiency, thus verifying the practicability of the proposed method and system.     

End‐to‐End Deep Learning Methodology for Real‐Time Traffic Network Management

This article presents a novel real‐time traffic network management system using an end‐to‐end deep learning (E2EDL) methodology. A computational learning model is trained, which allows the system to identify the time‐varying traffic congestion pattern in the network, and recommend integrated traffic management schemes to reduce this congestion. The proposed model structure captures the temporal and spatial congestion pattern correlations exhibited in the network, and associates these patterns with efficient traffic management schemes. The E2EDL traffic management system is trained using a laboratory‐generated data set consisting of pairings of prevailing traffic network conditions and efficient traffic management schemes designed to cope with these conditions. The system is applied for the US‐75 corridor in Dallas, Texas. Several experiments are conducted to examine the system performance under different traffic operational conditions. The results show that the E2EDL system achieves travel time savings comparable to those recorded for an optimization‐based traffic management system.     

A Fast Detection Method via Region‐Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects

Tunnel lining defects are an important indicator reflecting the safety status of shield tunnels. Inspired by the state‐of‐the‐art deep learning, a method for automatic intelligent classification and detection methodology of tunnel lining defects is presented. A fully convolutional network (FCN) model for classification is proposed. Information about defects, collected using charge‐coupled device cameras, was used to train the model. The model's performance was compared to those of GoogLeNet and VGG. The best‐set accuracy of the proposed model was over 95% at a test‐time speed of 48 ms per image. For defects detection, image features were computed from large‐scale images by the FCN and then detected using a region proposal network and position‐sensitive region of interest pooling. Some indices (detection rate, detection accuracy, and detection efficiency, locating accuracy) were used to evaluate the model. The comparisons with faster R‐CNN and a traditional method were conducted. The results show that the model is very fast and efficient, allowing automatic intelligent classification and detection of tunnel lining defects.     

Capturing and Understanding Workers’ Activities in Far‐Field Surveillance Videos with Deep Action Recognition and Bayesian Nonparametric Learning

Recording workers’ activities is an important, but burdensome, management task for site supervisors. The last decade has seen a growing trend toward vision‐based activity recognition. However, recognizing workers’ activities in far‐field surveillance videos is understudied. This study proposes a hierarchical statistical method for recognizing workers’ activities in far‐field surveillance videos. The method consists of two steps. First, a deep action recognition method was used to recognize workers’ actions, and a new fusion strategy was proposed to consider the characteristics of far‐field surveillance videos. The deep action recognition method with the new fusion strategy has achieved the comparable performance (0.84 average accuracy) on the far‐field surveillance data set in contrast to the original method on the public data sets. Second, a Bayesian nonparametric hidden semi‐Markov model was innovatively used to model and infer workers’ activities based on action sequences. The latent states of the fitted Bayesian model captured workers’ activities in terms of state duration distributions and state transition distributions, which are indispensable for understanding workers’ time allocation. It has been preliminarily illustrated that the activity information learned by the Bayesian model possesses the potential to implement objective work sampling, personal physical fatigue monitoring, trade‐level health risk assessment, and process‐based quality control. Also, the limitations of this study are discussed.     

Structure‐aware 3D reconstruction for cable‐stayed bridges: A learning‐based method

A powerful deep learning‐based three‐dimensional (3D) reconstruction method for reconstructing structure‐aware semantic 3D models of cable‐stayed bridges is proposed herein. Typically, conventional bridge semantic 3D model reconstruction methods are not robust when low‐quality point clouds are used. Furthermore, they are suited particularly for their respective fields and less generalized for cable‐stayed bridges. Hence, a structure‐aware learning‐based cable‐stayed bridge 3D reconstruction framework is proposed. The encoder part of the network uses both multiview images and a photogrammetric point cloud as input, whereas the decoder part uses a recursive binary tree network to model a high‐level structural relation graph and low‐level 3D geometric shapes. Two actual cable‐stayed bridges are employed as examples to evaluate the proposed method. Test results demonstrate that the proposed method successfully reconstructs the bridge model with structural components and their relations. Quantitative results indicate that the predicted models achieved an average F₁ score of 99.01%, a Chamfer distance of 0.0259, and a mesh‐to‐cloud distance of 1.78 m. The achieved result is similar to that obtained using the manual reconstruction approach in terms of component‐wise accuracy, and it is considerably better than that obtained using the manual approach in terms of spatial accuracy. In addition, the proposed recursive binary tree network is robust to noise and partial scans. The potential applications of the obtained 3D bridge models are discussed.     

Automatic pixel‐level multiple damage detection of concrete structure using fully convolutional network

Deep learning‐based structural damage detection methods overcome the limitation of inferior adaptability caused by extensively varying real‐world situations (e.g., lighting and shadow changes). However, most deep learning‐based methods detect structural damage at the image level and grid‐cell level. To provide pixel‐level detection of multiple damages, a Fully Convolutional Network (FCN)‐based multiple damages detection method for concrete structure is proposed. To realize this method, a database of 2,750 images (with 504 × 376 pixels) including crack, spalling, efflorescence, and hole images in concrete structure is built, and the four damages included in those images are labeled manually. Then, the architecture of the FCN is modified, trained, validated, and tested using this database. A strategy of model‐based transfer learning is used to initialize the parameters of the FCN during the training process. The results show 98.61% pixel accuracy (PA), 91.59% mean pixel accuracy (MPA), 84.53% mean intersection over union (MIoU), and 97.34% frequency weighted intersection over union (FWIoU). Subsequently, the robustness and adaptability of the trained FCN model is tested and the damage is extracted, where damage areas are provided according to a calibrated relation between the ratio (the pixel area and true area of the detected object) and the distance from the smartphone to the concrete surface using a laser range finder. A comparative study is conducted to examine the performance of the proposed FCN‐based approach using a SegNet‐based method. The results show that the proposed method substantiates quite better performance and can indeed detect multiple concrete damages at the pixel level in realistic situations.     

Encoder–decoder network for pixel‐level road crack detection in black‐box images

Timely monitoring of pavement cracks is essential for successful maintenance of road infrastructure. Accurate information concerning crack location and severity enables proactive management of the infrastructure. Black‐box cameras, which are becoming increasingly widespread at an affordable price, can be used as efficient road‐image collectors over a wide area. However, the cracks in these images are difficult to detect, because the images containing them often include objects other than roads. Thus, we propose a pixel‐level detection method for identifying road cracks in black‐box images using a deep convolutional encoder–decoder network. The encoder consists of convolutional layers of the residual network for extracting crack features, and the decoder consists of deconvolutional layers for localizing the cracks in an input image. The proposed network was trained on 427 out of 527 images extracted from black‐box videos and tested on the remaining 100 images. Compared with VGG‐16, ResNet‐50, ResNet‐101, ResNet‐200 with transfer learning, and ResNet‐152 without transfer learning, ResNet‐152 with transfer learning exhibited the best performance, achieving recall, precision, and intersection of union of 71.98%, 77.68%, and 59.65%, respectively. The experimental results prove that the proposed method is optimal for detecting cracks in black‐box images at the pixel level.     

Concrete crack detection using context‐aware deep semantic segmentation network

Computer‐vision and deep‐learning techniques are being increasingly applied to inspect, monitor, and assess infrastructure conditions including detection of cracks. Traditional vision‐based methods to detect cracks lack accuracy and generalization to work on complicated infrastructural conditions. This paper presents a novel context‐aware deep convolutional semantic segmentation network to effectively detect cracks in structural infrastructure under various conditions. The proposed method applies a pixel‐wise deep semantic segmentation network to segment the cracks on images with arbitrary sizes without retraining the prediction network. Meanwhile, a context‐aware fusion algorithm that leverages local cross‐state and cross‐space constraints is proposed to fuse the predictions of image patches. This method is evaluated on three datasets: CrackForest Dataset (CFD) and Tomorrows Road Infrastructure Monitoring, Management Dataset (TRIMMD) and a Customized Field Test Dataset (CFTD) and achieves Boundary F1 (BF) score of 0.8234, 0.8252, and 0.7937 under 2‐pixel error tolerance margin in CFD, TRIMMD, and CFTD, respectively. The proposed method advances the state‐of‐the‐art performance of BF score by approximately 2.71% in CFD, 1.47% in TRIMMD, and 4.14% in CFTD. Moreover, the averaged processing time of the proposed system is 0.7 s on a typical desktop with Intel^® Quad‐Core? i7‐7700 CPU@3.6 GHz Processor, 16GB RAM and NVIDIA GeForce GTX 1060 6GB GPU for an image of size 256 × 256 pixels.     

A Cell‐Based Distributed‐Coordinated Approach for Network‐Level Signal Timing Optimization

This article develops an efficient methodology to optimize the timing of signalized intersections in urban street networks. Our approach distributes a network‐level mixed‐integer linear program (MILP) to intersection level. This distribution significantly reduces the complexity of the MILP and makes it real‐time and scalable. We create coordination between MILPs to reduce the probability of finding locally optimal solutions. The formulation accounts for oversaturated conditions by using an appropriate objective function and explicit constraints on queue length. We develop a rolling‐horizon solution algorithm and apply it to several case‐study networks under various demand patterns. The objective function of the optimization program is to maximize intersection throughput. The comparison of the obtained solutions to an optimal solution found by a central optimization approach (whenever possible) shows a maximum of 1% gap on a number of performance measures over different conditions.     

A Clustering Algorithm for Bi‐Criteria Stop Location Design with Elastic Demand

This article proposes a bi‐criteria formulation to find the optimal location of light rapid transit stations in a network where demand is elastic and budget is constrained. Our model is composed of two competing objective functions seeking to maximize the total ridership and minimize the total budget allocated. In this research, demand is formulated using the random utility maximization method with variables including access time and travel time. The transit station location problem of this study is formulated using mixed integer programming and we propose a heuristic solution algorithm to solve large‐scale instances which is inspired by the problem context. The elastic demand is integrated with the optimization problem in an innovative way which facilitates the solution process. The performance of our model is evaluated on two test problems and we carry out its implementation on a real‐world instance. Due to the special shape of the Pareto front function, significant practical policy implications, in particular budget allocation, are discussed to emphasize the fact that the trade‐off between cost and benefit may result in large investments with little outcomes and vice versa.     

Self‐centering steel–timber hybrid shear wall with slip friction dampers: Theoretical analysis and experimental investigation

An innovative self‐centering steel–timber hybrid shear wall (SC‐STHSW) system is proposed as a promising structural solution for earthquake‐resilient buildings. The SC‐STHSW is composed of posttensioned (PT) steel rocking frame and infill light‐frame wood shear wall. The PT steel frame provides self‐centering capability, whereas the infill wood shear wall improves the lateral stiffness and the load resistance. Meanwhile, friction dampers are assembled into the connections between the steel frame and the infill wall to provide energy dissipation. Theoretical analysis and cyclic loading test were conducted to comprehend the load‐resisting behavior of the proposed SC‐STHSW system, and closed‐form solutions of the moment, shear, and axial force distribution along the length of the steel beam were formulated. Moreover, a nonlinear finite element model was developed, and the model was further used to verify the derived theoretical formulas. Results showed that the SC‐STHSW system was able to undergo large interstory drift without the development of plastic zones in the steel frame members, which resulted in very small residual deformation. The presented experimental and numerical results aim to provide a practical structural solution for high‐performance earthquake‐resilient buildings.     

Sense‐Think‐Act Framework for Intelligent Building Energy Management

The realization of smart and energetically efficient buildings is contingent upon the successful implementation of two tasks that occur on distinct phases of the building life cycle: in the design and subsequent retrofitting phases, the selection and implementation of an effective energy concept, and, during the operation phase, the actuation of energy systems to ensure parsimonious energy use while retaining acceptable end‐user thermal comfort. Operational efficiencies are achieved through the use of Building Energy Management Systems tasked to deliver core Sense, Think, Act (STA) functionalities: Sense, using sensing modalities installed in the building; Think, utilizing, typically a rule‐based decision system; and Act, by sending actuation commands to controllable building elements. Providing the intelligence in this STA process can be a formidable task due to the complex interplay of many systems and occurrence of disturbances. In this article, an architectural and algorithmic framework is presented to provide streamlined implementation of this process. Important ingredients in this framework are: (S) a data access component capable of collecting and aggregating information from a number of heterogeneous sources (sensors, weather stations, weather forecasts); (T) a model‐based optimization methodology to generate intelligent operational decisions; and (A) an assessment and actuation component. An illustrative application of the proposed methodology in an office building is provided.     

Automated crack evaluation of a high‐rise bridge pier using a ring‐type climbing robot

This article proposes a deep learning‐based automated crack evaluation technique for a high‐rise bridge pier using a ring‐type climbing robot. First, a ring‐type climbing robot system composed of multiple vision cameras, climbing robot, and control computer is developed. By spatially moving the climbing robot system along a target bridge pier with close‐up scanning condition, high‐quality raw vision images are continuously obtained. The raw vision images are then processed through feature control‐based image stitching, deep learning‐based semantic segmentation, and Euclidean distance transform–based crack quantification algorithms. Finally, a digital crack map on the region of interest (ROI) of the target bridge pier can be automatically established. The proposed technique is experimentally validated using in situ test data obtained from Jang–Duck bridge in South Korea. The test results reveal that the proposed technique successfully evaluates cracks on the entire ROI of the bridge pier with precision of 90.92% and recall of 97.47%.     

Real‐Time System Identification: An Algorithm for Simultaneous Model Class Selection and Parametric Identification

In this article, a novel Bayesian real‐time system identification algorithm using response measurement is proposed for dynamical systems. In contrast to most existing structural identification methods which focus solely on parametric identification, the proposed algorithm emphasizes also model class selection. By embedding the novel model class selection component into the extended Kalman filter, the proposed algorithm is applicable to simultaneous model class selection and parametric identification in the real‐time manner. Furthermore, parametric identification using the proposed algorithm is based on multiple model classes. Examples are presented with application to damage detection for degrading structures using noisy dynamic response measurement.     

Self‐calibrating Bayesian real‐time system identification

In this article, a novel Bayesian framework is proposed for real‐time system identification with calibratable model classes. This self‐calibrating scheme adaptively reconfigures the model classes to achieve reliable real‐time estimation for the system state and model parameters. At each time step, the plausibilities of the model classes are computed and they serve as the cue for calibration. Once calibration is triggered, all model classes will be reconfigured. Thereafter, identification will continue to propagate with the calibrated model classes until the next recalibration. Consequently, the model classes will evolve and their deficiencies can be corrected adaptively. This remarkable feature of the proposed framework stimulates the accessibility of reliable real‐time system identification. Examples are presented to demonstrate the efficacy of the proposed approach using noisy response measurement of linear and nonlinear time‐varying dynamical systems under stationary condition.     

Neuro‐Fuzzy Cost Estimation Model Enhanced by Fast Messy Genetic Algorithms for Semiconductor Hookup Construction

Abstract: Semiconductor hookup construction (i.e., constructing process tool piping systems) is critical to semiconductor fabrication plant completion. During the conceptual project phase, it is difficult to conduct an accurate cost estimate due to the great amount of uncertain cost items. This study proposes a new model for estimating semiconductor hookup construction project costs. The developed model, called FALCON‐COST, integrates the component ratios method, fuzzy adaptive learning control network (FALCON), fast messy genetic algorithm (fmGA), and three‐point cost estimation method to systematically deal with a cost‐estimating environment involving limited and uncertain data. In addition, the proposed model improves the current FALCON by devising a new algorithm to conduct building block selection and random gene deletion so that fmGA operations can be implemented in FALCON. The results of 54 case studies demonstrate that the proposed model has estimation accuracy of 83.82%, meaning it is approximately 22.74%, 23.08%, and 21.95% more accurate than the conventional average cost method, component ratios method, and modified FALCON‐COST method, respectively. Providing project managers with reliable cost estimates is essential for effectively controlling project costs.