A large quantity of civil infrastructure in North America is near the end of their design life. Consequently, the routine visual structural inspection is increasingly necessary to ensure the safety and efficient management of the infrastructure stock. The increasing need for inspections and the laborious nature of the work has caused strain on the inspection industry. To improve inspection efficacy, various researchers have proposed novel deep learning methodologies to automatically classify, detect, and segment structural defects from images. After the defects are identified, it is often desirable to quantify the size of the defect, for severity classification and repair cost estimation. Yet, the measurement from a single image for quantification is not a trivial task, requiring supplementary data or sensor inputs, which may not be practical or economical in the current inspection process. In this study, we propose to recover the three-dimensional geometry of a scene from a single image, by using deep learning-based monocular depth estimation. The monocular depth estimation field has made great progress by leveraging deep learning and a plethora of open red, green, blue, and depth (RGB-D) datasets. However, there has not been a publicly available in situ Light Detection and Ranging (LiDAR) RGB-D dataset for the civil engineering domain, which is a barrier for researchers to develop and evaluate spatial computer vision methods in the civil engineering context. To bridge this gap, we build a LiDAR-based RGB-D dataset for training monocular depth estimators. Then using the civil RGB-D dataset, we test a solution for the real-world application of monocular depth estimation to quantify defects in civil infrastructure.

Routine visual structural inspection is a tedious but vital part of structural health monitoring. Many researchers have proposed novel methodologies to automatically classify, detect, and segment structural defects (i.e., crack, spall, etc.) from images. Yet, the scale from single images is ambiguous, which is an important prior for severity classification. While monocular depth estimation is an ill-posed problem, deep learning methods have made great progress field, greatly spurred by a plethora of open RGB-D datasets. However, to the best knowledge of the authors, there is no RGB-D dataset in the civil engineering domain. In this work, the authors seek to develop an efficient method to build an RGB-D dataset for the civil research community. The authors review popular RGB-D data collection paradigms and propose a LiDAR-based data collection method. Finally, the authors use the collected data to create a deep convolutional monocular depth estimation model for defect quantification. The authors hope this work can help other researchers incorporate depth information in their projects and to create a community to share civil RGB-D data to help advance the state-of-the-art in the automated visual structural inspection.

Single shot, semantic bounding box detectors, trained in a supervised manner are popular in computer vision‐aided visual inspections. These methods have several key limitations: (1) bounding boxes capture too much background, especially when images experience perspective transformation; (2) insufficient domain‐specific data and cost to label; and (3) redundant or incorrect detection results on videos or multi‐frame data; where it is a nontrivial task to select the best detection and check for outliers. Recent developments in commercial augmented reality and robotic hardware can be leveraged to support inspection tasks. A common capability of the previous is the ability to obtain image sequences and camera poses. In this work, the authors leverage pose information as “prior” to address the limitations of existing supervised learned, single‐shot, semantic detectors for the application of visual inspection. The authors propose an unsupervised semantic segmentation method (USP), based on unsupervised learning for image segmentation inspired by differentiable feature clustering coupled with a novel outlier rejection and stochastic consensus mechanism for mask refinement. USP was experimentally validated for a spalling quantification task using a mixed reality headset (Microsoft HoloLens 2). Also, a sensitivity study was conducted to evaluate the performance of USP under environmental or operational variations.

Seismic vulnerability assessment empowers decision-makers to develop effective earthquake mitigation strategies. In the past few decades, numerous empirical and analytical seismic vulnerability methods have been proposed; among which is the Hassan-Sozen Priority Index (PI). The PI is a very simple and popular metric for the seismic vulnerability of low-rise reinforced concrete buildings based on a linear combination of only the geometric dimensions of a building such as the floor area and the size of the lateral force resisting elements. This simplicity makes the PI a useful preliminary screening tool, but its accuracy and reliability are limited and questionable since the index oversimplifies the relationship between the dimensions of the building and its potential damage rating. Moreover, the performance of the PI has only been validated using a small set of post-earthquake reconnaissance data. State-of-the-art machine learning techniques make it possible to efficiently fit large and complex datasets which opens the door for the development of more advanced seismic vulnerability assessment methodologies. In this study, a large volume of post-earthquake reconnaissance data is gathered and an ensemble supervised learning approach (Random Forest) is implemented to develop a robust model that can relate the PI input parameters to building damage rating. Additionally, a Shapley Additive explanations (SHAP) method is exploited to validate and explain the models and conduct model transparency analysis.

After hazard events, large numbers of images are collected by reconnaissance teams to document the post-event state of structures, and to assess their performance and improve design procedures and codes. The majority of these data are captured as images and manually labeled. This highly repetitive task requires considerable domain expertise and time. Advances in deep learning have enabled researchers to rapidly classify reconnaissance images. Thus far, these classification methods are limited to a simple classification schema in which the classes are all either mutually exclusive or independent. To date, an efficient classification system of a complex schema containing many classes arranged in a multi-level hierarchical structure is not available to support earthquake reconnaissance. To address this gap, this paper introduces a comprehensive classification schema and a multi-output deep convolutional neural network (DCNN) model for rapid postearthquake image classification. In contrast to past work, herein a single multi-output DCNN classification model with a hierarchy-aware prediction was trained to enable the rapid organization of images. The performance of the proposed multi-output model was validated through comparisons with multi-label and multi-class models using an F1-score. As result, the multi-output model outperformed other models. Then, the multi-output model was deployed to a web-based platform called the Automated Reconnaissance Image Organizer, which can be used to easily organize earthquake reconnaissance images. (C) 2022 American Society of Civil Engineers.