Autonomous vehicles (AVs) are a promising technology to reduce automotive fatalities and crashes dramatically. However, AV deployment to date has been restricted to limited operational design domains, largely because of the wide-range complexity of driving. For instance, AVs will encounter work zones while driving, because work zones are necessary to maintain, repair, and upgrade roadways. Work zones and road incidents such as crash sites are complex and safety-critical, raising significant challenges for AV perception and behaviors. Bad weather, poor lighting, weak cloud connectivity, and degraded road conditions aggravate matters further. We study how V2X communications can assist AVs to make driving safer for road users. V2X communications provide reliable and detailed information that can augment perception. We introduce a range of realistic yet complex driving environments, discuss how V2X communications improve AV safety, and capture deployment requirements for V2X. We propose three new V2X message protocols called Lane Blockage Alert (LBA), Basic Contour Message (BCM), and Moving Obstacle Alert (MOA) to address some of these requirements. We present our experiences and results from a set of experiments conducted in test-track and real-world work zones to demonstrate the benefits of AVs using V2X in work zones.

Work zones are essential to maintain, repair and upgrade our roadways. However, they introduce complex, dynamic and challenging environments for autonomous vehicles to navigate safely. To help address this challenge, we introduce the first publicly available, large-scale, multimodal work zone dataset collected with an autonomous vehicle consisting of multiple synchronized lidars and high-resolution cameras. Our WorkZone3D dataset covers various work zone elements such as cones, barrels, and channelizers, and provides 3D annotation boxes for these objects. We also propose a auto-annotation pipeline that can produce high-quality 3D labels, assimilating data across frames, even for rare classes which do not have pre-trained 3D object detection models to start with. We evaluate unimodal and multimodal models on our dataset, showing the critical role of sensor fusion in accurate 3D localization of such small objects at a distance, often having very few lidar points on them. Our results demonstrate the usefulness of WorkZone3D for generalization to real-world scenarios. Our code and dataset are available at https://github.com/ssuralcmu/WorkZone3D.git.

Autonomous vehicles (AVs) promise significant advances in transportation safety and efficiency. However, navigating roadway work zones, which can be rather complex and dynamic, remains a significant challenge. This paper presents the results of a large study that addresses the challenges, requirements, solutions and practical experiences of AVs driving safely through work zones. We begin by proposing a taxonomy of work zone scenarios and analyzing their structures and attributes. We next discuss the perception, routing, behavioral and path-planning requirements for AVs to safely navigate these scenarios. We then offer methods to meet these requirements and investigate the impact of range and AV speed on perception confidence levels for work zone detection. We evaluate our solutions in a co-simulation environment, on a closed test-track and on public roadways across more than 20 work zone scenarios specified by the Pennsylvania Department of Transportation (PennDoT). Video demonstrations illustrate the feasibility of safe and reliable navigation of AVs in a wide variety of work zones.

A very important use case of V2X communications is the enhancement of roadway safety by utilizing the transmission of road information among vehicles. The SAE Basic Safety Message (BSM) is the most common standard used to transmit road event information and their locations based on global latitudinal and longitudinal coordinates of transmitting vehicles. In practice, however, global coordinate estimations are inherently limited by the accuracy of Global Navigation Satellite Systems (GNSS) such as GPS. GNSS signals can also be unavailable in urban canyons and tunnels, be spoofed to force incorrect localization, or be restricted to low accuracy due to the sparsity of available groundbased corrections (such as RTK base stations) in rural and remote areas. In this paper, we introduce Enhanced Safety Messages (ESMs), a backward-compatible BSM replacement that adopts the well-established foundations of information redundancy in safety-critical systems to avoid catastrophic failure by providing both absolute and relative coordinate frames to robustly describe vehicle locations. This position information redundancy in two different coordinate systems, one from external sources and another from local sensing, effectively addresses the BSM drawbacks of relying entirely on GNSS signals. Specifically, ESM includes LaneContext and MapContext to address two of the most common driving environments of open spaces and urban roadways. Location communication over ESM is enhanced by additionally specifying the connected vehicle's driving lane, its offset from the center of the lane, and its longitudinal position along the road segment. Our experimental evaluation confirms that ESM's inclusion of relative positioning rectifies the core BSM weaknesses in relying only on global GNSS coordinates.

Locating and reading traffic signs is an integral aspect of driving in general and is considered safety-critical for autonomous vehicles (AVs). However, the sheer number of disparate traffic signs and a lack of public datasets pose unique challenges for designing computer vision algorithms to recognize a wide variety of signs. In this paper, we propose TrafficSignReader, a three-stage framework for recognizing textbased traffic signs without the need for category-specific sign data. TrafficSignReader comprises an object detection stage, an optical character recognition step, and a language-based matching stage for individual sign recognition. Our approach is robust against motion blur, occlusion, glare from streetlights as well as weather conditions. We also create the Textual Traffic Sign Dataset, consisting of about 10,000 textual traffic sign images across 190 classes, covering most of the textual traffic sign categories found in the United States. TrafficSignReader recognizes textual traffic signs across all these classes with an F1-Score of 95.1 %, comparing very favorably with respect to the current state of the art in terms of accuracy, breadth of coverage, generalizability and run-time efficiency. It runs at 10 fps using a lightweight GPU on our CMU AV, reading traffic signs over 50 m away, and making it feasible for real-world deployment. Our code and dataset are available at https://github.com/ssuralcmu/TrafficSignReader.git.

The fusion of multimodal sensor data streams such as camera images and lidar point clouds plays an important role in the operation of autonomous vehicles (AVs). Robust perception across a range of adverse weather and lighting conditions is specifically required for AVs to be deployed widely. While multi-sensor fusion networks have been previously developed for perception in sunny and clear weather conditions, these methods show a significant degradation in performance under night-time and poor weather conditions. In this paper, we propose a simple yet effective technique called ContextualFusion to incorporate the domain knowledge about cameras and lidars behaving differently across lighting and weather variations into 3D object detection models. Specifically, we design a Gated Convolutional Fusion (GatedConv) approach for the fusion of sensor streams based on the operational context. To aid in our evaluation, we use the open-source simulator CARLA to create a multimodal adverse-condition dataset called AdverseOp3D to address the shortcomings of existing datasets being biased towards daytime and good-weather conditions. Our ContextualFusion approach yields an mAP improvement of 6.2% over state-of-the-art methods on our context-balanced synthetic dataset. Finally, our method enhances state-of-the-art 3D objection performance at night on the real-world NuScenes dataset with a significant mAP improvement of 11.7%.

Autonomous vehicles (AVs) can operate more safely with vehicular communication technology. A Vehicle-to-Everything (V2X) communication application domain of particular interest is work zones, which may require significant amounts of information to fully describe their shapes and contents. However, existing V2X protocols are limited in how much data can be transmitted in a single packet. In this paper, we propose and evaluate the Multi-Packet Memo Protocol (MPMP), a V2X communication protocol for broadcasting long messages, called memos. MPMP can transmit up to 1 MB per memo and correctly assemble packets received in an out-of-order sequence. Multiple receivers can simultaneously begin receiving packets at different points in the sequence. Latency and reliability measurements show that long messages can be transmitted reliably with acceptable delays for safe CAV operations in work zones.

In recent years, there has been a notable increase in the development of autonomous vehicle (AV) technologies aimed at improving safety in transportation systems. While AVs have been deployed in the real-world to some extent, a full-scale deployment requires AVs to robustly navigate through challenges like heavy rain, snow, low lighting, construction zones and GPS signal loss in tunnels. To be able to handle these specific challenges, an AV must reliably recognize the physical attributes of the environment in which it operates. In this paper, we define context recognition as the task of accurately identifying environmental attributes for an AV to appropriately deal with them. Specifically, we define 24 environmental contexts capturing a variety of weather, lighting, traffic and road conditions that an AV must be aware of. Motivated by the need to recognize environmental contexts, we create a context recognition dataset called DrivingContexts with more than 1.6 million context-query pairs relevant for an AV. Since traditional supervised computer vision approaches do not scale well to a variety of contexts, we propose a framework called ContextVLM that uses vision-language models to detect contexts using zero- and few-shot approaches. ContextVLM is capable of reliably detecting relevant driving contexts with an accuracy of more than 95% on our dataset, while running in real-time on a 4GB Nvidia GeForce GTX 1050 Ti GPU on an AV with a latency of 10.5 ms per query.

Autonomous vehicles (AVs) have the potential to mitigate automotive crashes, lower fatalities and optimize energy consumption [1]. Still, the widespread deployment of AVs faces limitations due to practical constraints placed on the Operational Design Domain (ODD) that often excludes challenging scenarios such as work zones and inclement weather conditions. These restrictions pose a considerable obstacle to broader AV deployment. For instance, robotaxis deployed by GM Cruise and Waymo in San Francisco have demonstrated unreliable behavior in work zones. In this paper, we introduce a cooperative route-planning and behavioral framework named SafeRoute designed to mitigate operational safety risks along the routes taken by AVs. SafeRoute explicitly considers high-risk conditions including work zones, low-visibility environments and adverse weather conditions to minimize associated risks while avoiding excessively long detours. The weights assigned to various ODD features can be customized as the perception capabilities of AVs improve. Efficient lane-changing mechanisms are designed to handle complex work zones dynamically. SafeRoute has been integrated into CMU's AV software stack. Extensive testing has been done using an array of work zone scenarios specified by the Pennsylvania Department of Transportation (PennDoT), encompassing simulations, test tracks and real-world public roads (video demos provided). Operating in real-time, SafeRoute effectively handles a diverse set of challenging work zone scenarios.

Autonomous driving has immense potential to improve the safety of vehicles and pedestrians. However, safety assurances for Autonomous Vehicles (AVs) are lacking under adverse weather conditions and in unseen road environments. Unfortunately, real-world testing of AVs in such situations can be unsafe and even infeasible. Reproducing any failure cases with identical external operating conditions is practically impossible. Specifically, situations like heavy rain, low-lighting conditions, and work zones require considerable time and effort. To address this problem, we propose CoSim, a co-simulation architecture for testing of AV perception in adverse operating conditions. We specifically use the general framework of CoSim to interface CMU's autonomous software stack called CADRE with an open-source simulator named CARLA. CoSim is designed such that the AV software stack interacts with realistic simulated driving scenarios as in the real world. Using CoSim, we mimic the use of cameras and LIDARs for the real-time detection of road objects and work zones under adverse operating conditions. CoSim supports time virtualization on both CADRE and CARLA to accommodate heavy processing demands. Using CoSim, we evaluate the performance of our AV in adverse operating conditions, dramatically reducing the need for expensive real-world testing.

Selecting period values for tasks is a very important step in the design process of a real-time system, especially due to the significance of its impact on system schedulability. It is well known that, under RMS, the utilization bound for a harmonic task set is 100%. Also, polynomial-time algorithms have been developed for response-time analysis of harmonic task sets. In practice, the largest acceptable value for the period of a task is determined by the performance and safety requirements of the application. In this paper, we address the problem of assigning harmonic periods to a task set such that every task gets assigned an integer period less than or equal to its application specified upper bound and the task utilization of every task is less than 1. We focus on integer solutions given the discrete nature of time in real-time computer systems. We first express this problem of assigning harmonic periods to a task set as a discrete piecewise optimization problem. We then present the 'Discrete Piecewise Harmonic Search' (DPHS) algorithm that outputs an optimal harmonic task assignment. We then define conditions for a metric to be rational for harmonization. We show that commonly used metrics like, the total percentage error (TPE), total system utilization (TSU), first order error (FOE), and maximum percentage error (MPE), are rational. We next prove that the DPHS algorithm finds the optimal feasible assignment, if one exists, for these rational metrics. We apply the DPHS algorithm to harmonize task sets used in real-world applications to highlight its benefits. We compare the performance of the DPHS algorithm against a brute-force search and find that the DPHS searches up to 94\% fewer task sets than the brute-force search that obtains the optimal solution.

Autonomous vehicles (AVs) have garnered immense interest and investments for more than a decade and a half. Nevertheless, large-scale AV deployments do not seem viable in the near future. In this talk, the speaker will address questions like "What went wrong?", "Is AI the answer?", "Can (and how do) we course-correct?" and "Which future contributions will matter?". Finally, challenges that must be addressed by the research and engineering communities will be discussed.

Cooperative perception is a prospective application to improve road safety by having connected autonomous vehicles (CAVs) exchange their raw or processed sensor data over vehicular communications. Since CAVs heavily rely on sensor-based perception, including vision cameras, LiDARs, and radars, cooperative perception has an immense potential to improve road safety. At the same time, a variety of sensors and edge servers have been widely deployed in smart cities, and such sensors and servers might be able to empower CAVs on public roads. In this article, we comprehensively study such cooperative perception for overview, technical challenges, practical requirements, prospective system designs, current approaches, and future research directions. In particular, we focus on the time sensitivity of the cooperative perception frameworks, in which delays of computing and communications lead to detection errors. In addition, we comprehensively study sensor fusion frameworks and models to improve the detection accuracy and reliability of CAVs while avoiding information flooding and/or rumor spreading. Since network congestion might lead to packet collisions and/or delays, congestion control might be essential to use cooperative perception in practice. Finally, we discuss the technical and ethical challenges of using cooperative perception on public roads and conclude with future research directions.

Autonomous driving promises a safer and more efficient means of transportation. However, one of many challenges it faces is understanding the complex driving environment. Modern perception systems often utilize neural networks and tackle complicated driving scenarios in a data-driven manner. Unfortunately, such systems require a large amount of labeled data, which can be prohibitively expensive to collect. In this paper, we investigate self-supervised learning as a method to reduce the reliance on labeled data in the context of autonomous driving. We specifically focus on point cloud recognition and apply contrastive and geometric pretext tasks to pretrain neural networks using unlabeled point cloud data. We conduct experiments in the nuScenes autonomous driving dataset with various amounts of labeled data. Our experiments reveal three insights: (1) pretraining with contrastive loss alone improves the average precision (AP) but negatively impacts the object heading accuracy, (2) combining contrastive and geometric pretext tasks benefits both the average precision and heading accuracy, and (3) the improvement by self-supervised pretraining remains even with an increased amount of labeled data and training steps.

Autonomous driving systems are becoming increasingly feasible and highly expected to be the heart of intelligent transportation systems. To deploy the autonomous driving vehicles on public roads, one of the practical challenges might be safe cooperation and collaboration among multiple vehicles, in particular when conflicts arise on shared road segments, such as road intersections, merge points, construction zones, single-track lanes, and center turn lane. In the current traffic systems, human drivers navigate these regions using a combination of traffic rules, social norms, courtesy, hand signals, and common sense. In this paper, we identify and classify such Dynamic Intersections that might lead to vehicle accidents and/or deadlocks and that might appear almost anytime and anywhere on public roads. In addition, we present a cooperative dynamic intersection protocol that uses on-board perception systems and vehicular communications for peer-to-peer negotiation. Under our protocol, autonomous driving vehicles can create a vehicular communication-based traffic manager named Cyber Traffic Light when congestion arises. Cyber Traffic Light works as a self-organizing, self-planning, and self-optimizing traffic manager, and it allocates the green period for vehicles coming from the multiple directions. Finally, we showed that our decentralized protocol has much higher traffic throughput, compared to two simple protocols while guaranteeing road safety.

This letter presents LaneMatch, a localization method for use in autonomous vehicles (AVs). It utilizes lane matching to obtain an AV's lane occupancy and current pose estimation. Matching is performed on a compact low-resolution road map generated from satellite images. Our approach addresses the following challenges for AVs using this map: (1) misalignment between roads on the satellite images and their global coordinates, and (2) incomplete or incorrect lane detection outputs. First, LaneMatch estimates the offset between the AV's global pose in the global coordinate system and its local map pose on the map. Secondly, LaneMatch utilizes a spatio-temporal integration of a particle filter and a factor graph to resolve lane-matching ambiguities. It strategically constrains the dimensionality of variables to obtain real-time performance. We use highway experiments to evaluate the processing time, occupancy accuracy, lateral/longitudinal and position/heading errors. These experiments show that LaneMatch localizes our AV rather precisely on the road map in real-time and can be used for navigation and planning purposes even in GNSS-unfriendly areas.

Connected and Automated Vehicles (CAVs) are highly expected to improve traffic throughput and safety at road intersections, single-track lanes, and construction zones. However, multiple CAVs can block each other and create a mutual deadlock around these road segments (i) when vehicle systems have a failure, such as a communication failure, control failure, or localization failure and/or (ii) when vehicles use a long shared road segment. In this paper, we present an Autonomous Deadlock Detection and Recovery Protocol at Intersections for Automated Vehicles named A-DRIVE that is a decentralized and time-sensitive technique to improve traffic throughput and shorten worst-case recovery time. To enable the deadlock recovery with automated vehicles and with human-driven vehicles, A-DRIVE includes two components: V2V communication-based A-DRIVE and Local perception-based A-DRIVE. V2V communication-based A-DRIVE is designed for homogeneous traffic environments in which all the vehicles are connected and automated. Local perception-based A-DRIVE is for mixed traffic, where CAVs, non-connected automated vehicles, and human-driven vehicles co-exist and cooperate with one another. Since these two components are not exclusive, CAVs inclusively and seamlessly use them in practice. Finally, our simulation results show that A-DRIVE improves traffic throughput compared to a baseline protocol.

Path planning is a crucial task in autonomous vehicles for which real-time road segmentation is very important. Most existing road segmentation techniques are supervised but, in many cases, their performance may be limited by the availability of and variety in a large training dataset. In contrast, we propose a research direction on unsupervised road segmentation that does not need any training or adaption and can be utilized widely. We use K-means clustering and image processing techniques to segment roads in RGB images. The scheme works well on the KITTI Road dataset (urban), giving a maximum, mean, and minimum IoU score of 93.75 %, 66.64% and 32.21% respectively. The minimum, mean and maximum time taken for segmentation were 1.084 s, 1.999 s and 3.794 s respectively on an Intel Core i5-8th Gen.(8GB RAM) CPU. A major reason for low values of minimum accuracy is that the scheme may segment the sidewalk also as a road. Although the mean IoU score is lower and the processing time higher relative to existing schemes, the results are very promising as our scheme is completely unsupervised and the processing time can be reduced by leveraging the capabilities of GPUs, parallel execution, hardware acceleration and the like.

A major challenge for autonomous vehicles is when an AV encounters a flagman who regulates traffic near a construction workzone area or at an automotive crash site. In such real-world scenarios, recognizing the flagman's gesture is an essential AV function that has not received much attention. A key piece of earlier work used a chroma-key (green background) screen to generate datasets and evaluate the accuracy of flagman gesture recognition. We find that this background-agnostic approach drops significantly in performance when the chroma-key background is replaced with images from real-world traffic scenarios. In this paper, we extend that baseline approach by adding contextual information and boosting the baseline's accuracy and robustness. First, we replace the chroma-key background to create virtually augmented (VA) images to capture real-world contexts. Next, we propose and evaluate three approaches: (1) Extract CNN features directly from the “raw” VA images, (2) Extracting CNN features after embedding flagman skeleton and prop information in the VA images, and (3) Utilize an attention mechanism via a transformer with padding masks. All three approaches outperform the baseline method. Notably, our Two-Stage Classifier with Transformer approach boosts the baseline's performance from an F-score of 32% to 80% in challenging VA traffic scenarios.

Deep neural network (deepnet) applications play a crucial role in safety-critical systems such as autonomous vehicles (AVs). An AV must drive safely towards its destination, avoiding obstacles, and respond quickly when the vehicle must stop. Any transient errors in software calculations or hardware memory in these deepnet applications can potentially lead to dramatically incorrect results. Therefore, assessing and mitigating any transient errors and providing robust results are important for safety-critical systems. Previous research on this subject focused on detecting errors and then recovering from the errors by re-running the network. Other approaches were based on the extent of full network duplication such as the ensemble learning-based approach to boost system fault-tolerance by leveraging each model’s advantages. However, it is hard to detect errors in a deep neural network, and the computational overhead of full redundancy can be substantial.We first study the impact of the error types and locations in deepnets. We next focus on selecting which part should be duplicated using multiple ranking methods to measure the order of importance among neurons. We find that the duplication overhead for computation and memory is a trade-off between algorithmic performance and robustness. To achieve higher robustness with less system overhead, we present two error protection mechanisms that only duplicate parts of the network from critical neurons. Finally, we substantiate the practical feasibility of our approach and evaluate the improvement in the accuracy of a deepnet in the presence of errors. We demonstrate these results using a case study with real-world applications on an Nvidia GeForce RTX 2070Ti GPU and an Nvidia Xavier embedded platform used by automotive OEMs.