End-to-end deep neural networks have achieved remarkable success across various domains but are often criticized for their lack of interpretability. While post hoc explanation methods attempt to address this issue, they often fail to accurately represent these black-box models, resulting in misleading or incomplete explanations. To overcome these challenges, we propose an inherently transparent model architecture called Neural Probabilistic Circuits (NPCs), which enable compositional and interpretable predictions through logical reasoning. In particular, an NPC consists of two modules: an attribute recognition model, which predicts probabilities for various attributes, and a task predictor built on a probabilistic circuit, which enables logical reasoning over recognized attributes to make class predictions. To train NPCs, we introduce a three-stage training algorithm comprising attribute recognition, circuit construction, and joint optimization. Moreover, we theoretically demonstrate that an NPC's error is upper-bounded by a linear combination of the errors from its modules. To further demonstrate the interpretability of NPC, we provide both the most probable explanations and the counterfactual explanations. Empirical results on four benchmark datasets show that NPCs strike a balance between interpretability and performance, achieving results competitive even with those of end-to-end black-box models while providing enhanced interpretability.

LLM-based agents are becoming increasingly capable, yet their safety lags behind. This creates a gap between what agents can do and should do. This gap widens as agents engage in multi-turn interactions and employ diverse tools, introducing new risks overlooked by existing benchmarks. To systematically scale safety testing in multi-turn, tool-realistic settings, we propose a principled taxonomy that transforms single-turn harmful tasks into multi-turn attack sequences. Using this taxonomy, we construct MT-AgentRisk (Multi-Turn Agent Risk Benchmark), the first benchmark to evaluate multi-turn tool-agent safety. Our experiments reveal substantial safety degradation: the Attack Success Rate (ASR) increases by 16.1\% on average across open and closed models in multi-turn settings. To close this gap, we propose ToolShield, a training-free, tool-agnostic, self-exploration defense: when encountering a new tool, the agent autonomously generates test cases, executes them to observe downstream effects, and distills safety experiences for deployment. Experiments show that ToolShield effectively reduces ASR by 30\% on average in multi-turn interactions.

We present VISAT, a novel open dataset and benchmarking suite for evaluating model robustness in the task of traffic sign recognition with the presence of visual attributes. Built upon the Mapillary Traffic Sign Dataset (MTSD), our dataset introduces two benchmarks that respectively emphasize robustness against adversarial attacks and distribution shifts. For our adversarial attack benchmark, we employ the state-of-the-art Projected Gradient Descent (PGD) method to generate adversarial inputs and evaluate their impact on popular models. Additionally, we investigate the effect of adversarial attacks on attribute-specific multi-task learning (MTL) networks, revealing spurious correlations among MTL tasks. The MTL networks leverage visual attributes (color, shape, symbol, and text) that we have created for each traffic sign in our dataset. For our distribution shift benchmark, we utilize ImageNet-C's realistic data corruption and natural variation techniques to perform evaluations on the robustness of both base and MTL models. Moreover, we further explore spurious correlations among MTL tasks through synthetic alterations of traffic sign colors using color quantization techniques. Our experiments focus on two major backbones, ResNet-152 and ViT-B/32, and compare the performance between base and MTL models. The VISAT dataset and benchmarking framework contribute to the understanding of model robustness for traffic sign recognition, shedding light on the challenges posed by adversarial attacks and distribution shifts. We believe this work will facilitate advancements in developing more robust models for real-world applications in autonomous driving and cyber-physical systems.

Advances in deep learning have revolutionized cyber-physical applications, including the development of autonomous vehicles. However, real-world collisions involving autonomous control of vehicles have raised significant safety concerns regarding the use of deep neural networks (DNNs) in safety-critical tasks, particularly perception. The inherent unverifiability of DNNs poses a key challenge in ensuring their safe and reliable operation. In this work, we propose perception simplex (PS$$ \mathcal{PS} $$), a fault-tolerant application architecture designed for obstacle detection and collision avoidance. We analyse an existing LiDAR-based classical obstacle detection algorithm to establish strict bounds on its capabilities and limitations. Such analysis and verification have not been possible for deep learning-based perception systems yet. By employing verifiable obstacle detection algorithms, PS$$ \mathcal{PS} $$ identifies obstacle existence detection faults in the output of unverifiable DNN-based object detectors. When faults with potential collision risks are detected, appropriate corrective actions are initiated. Through extensive analysis and software-in-the-loop simulations, we demonstrate that PS$$ \mathcal{PS} $$ provides deterministic fault tolerance against obstacle existence detection faults, establishing a robust safety guarantee. The traditional autonomous driving system (mission layer) is monitored at runtime by the safety layer, which includes verifiable algorithms only. Utilizing the detectability model for such perception algorithms, safety layer provides deterministic fault detection and collision avoidance properties. This framework presents a promising approach towards the end goal of verifiable end-to-end safety in autonomous vehicles. Detectability model provides deterministic translation between safety policies, sensors and algorithm parameters and safety guarantees. Such a model can only be devised for fully analysable and verifiable solutions, a requirement not met by deep neural networks. image

The paper discusses algorithmic priority inversion in mission-critical machine inference pipelines used in modern neural-network-based perception subsystems and describes a solution to mitigate its effect. In general, priority inversion occurs in computing systems when computations that are "less important" are performed together with or ahead of those that are "more important." Significant priority inversion occurs in existing machine inference pipelines when they do not differentiate between critical and less critical data. We describe a framework to resolve this problem and demonstrate that it improves a perception system's ability to react to critical inputs, while at the same time reducing platform cost.

Perception of obstacles remains a critical safety concern for autonomous vehicles. Real-world collisions have shown that the autonomy faults leading to fatal collisions originate from obstacle existence detection. Open source autonomous driving implementations show a perception pipeline with complex interdependent Deep Neural Networks. These networks are not fully verifiable, making them unsuitable for safety-critical tasks. In this work, we present a safety verification of an existing LiDAR based classical obstacle detection algorithm. We establish strict bounds on the capabilities of this obstacle detection algorithm. Given safety standards, such bounds allow for determining LiDAR sensor properties that would reliably satisfy the standards. Such analysis has as yet been unattainable for neural network based perception systems. We provide a rigorous analysis of the obstacle detection system with empirical results based on real-world sensor data.

Advances in deep learning have revolutionized cyber-physical applications, including the development of Autonomous Vehicles. However, real-world collisions involving autonomous control of vehicles have raised significant safety concerns regarding the use of Deep Neural Networks (DNN) in safety-critical tasks, particularly Perception. The inherent unverifiability of DNNs poses a key challenge in ensuring their safe and reliable operation. In this work, we propose Perception Simplex (PS), a fault-tolerant application architecture designed for obstacle detection and collision avoidance. We analyze an existing LiDAR-based classical obstacle detection algorithm to establish strict bounds on its capabilities and limitations. Such analysis and verification have not been possible for deep learning-based perception systems yet. By employing verifiable obstacle detection algorithms, PS identifies obstacle existence detection faults in the output of unverifiable DNN-based object detectors. When faults with potential collision risks are detected, appropriate corrective actions are initiated. Through extensive analysis and software-in-the-loop simulations, we demonstrate that PS provides predictable and deterministic fault tolerance against obstacle existence detection faults, establishing a robust safety guarantee.

This paper explores criticality-based real-time scheduling of neural-network-based machine inference pipelines in cyber-physical systems (CPS) to mitigate the effect of algorithmic priority inversion. We specifically focus on the perception subsystem, an important subsystem feeding other components (e.g., planning and control). In general, priority inversion occurs in real-time systems when computations that are of lower priority are performed together with or ahead of those that are of higher priority. In current machine perception software, significant priority inversion occurs because resource allocation to the underlying neural network models does not differentiate between critical and less critical data within a scene. To remedy this problem, in recent work, we proposed an architecture to partition the input data into regions of different criticality, then formulated a utility-based optimization problem to batch and schedule their processing in a manner that maximizes confidence in perception results, subject to criticality-based time constraints. This journal extension matures the work in several directions: (i) We extend confidence maximization to a generalized utility optimization formulation that accounts for criticality in the utility function itself, offering finer-grained control over resource allocation within the perception pipeline; (ii) we further instantiate and compare two different criticality metrics (distance-based and relative velocity-based) to understand their relative advantages; and (iii) we explore the limitations of the approach, specifically how inaccuracies in criticality-based attention cueing affect performance. All experiments are conducted on the NVIDIA Jetson AGX Xavier platform with a real-world driving dataset.

Object detection in state-of-the-art Autonomous Vehicles (AV) framework relies heavily on deep neural networks. Typically, these networks perform object detection uniformly on the entire camera LiDAR frames. However, this uniformity jeopardizes the safety of the AV by giving the same priority to all objects in the scenes regardless of their risk of collision to the AV. In this paper, we present a new end-to-end pipeline for AV that introduces the concept of LiDAR cluster first and camera inference later to detect and classify objects. The benefits of our proposed framework are twofold. First, our pipeline prioritizes detecting objects that pose a higher risk of collision to the AV, giving more time for the AV to react to unsafe conditions. Second, it also provides, on average, faster inference speeds compared to popular deep neural network pipelines. We design our framework using the real-world datasets, the Waymo Open Dataset, solving challenges arising from the limitations of LiDAR sensors and object detection algorithms. We show that our novel object detection pipeline prioritizes the detection of higher risk objects while simultaneously achieving comparable accuracy and a 25% higher average speed compared to camera inference only.

The paper discusses algorithmic priority inversion in mission-critical machine inference pipelines used in modern neural-network-based cyber-physical applications, and develops a scheduling solution to mitigate its effect. In general, priority inversion occurs in real-time systems when computations that are of lower priority are performed together with or ahead of those that are of higher priority.1 In current machine intelligence software, significant priority inversion occurs on the path from perception to decision-making, where the execution of underlying neural network algorithms does not differentiate between critical and less critical data. We describe a scheduling framework to resolve this problem, and demonstrate that it improves the system’s ability to react to critical inputs, while at the same time reducing platform cost.

The growing application space of Unmanned Aerial Vehicles (UAVs) is creating the need for aircraft capable of autonomous, long-distance, and long-endurance flights. The two main challenges are the limited power capacity of UAVs, as well as the adaptation to real-time detected stimuli, changing the course of the mission. This paper describes the continuous development of the UIUC-TUM Solar Flyer, which is equipped with low-power, high-performance computing capabilities. The Solar Flyer addresses the aforementioned challenges by balancing power consumption and solar power generation, and by performing on-board data processing, to enable real-time mission adaptation. The Solar Flyer was developed from commercial-of-the-shelf components, making it affordable for a wide variety of applications. Current efforts have built on previous work in terms of airframe, avionics, and flight software. Significant effort has been allocated to the re-design of the propulsion and energy systems as well as the re-organization of the aircraft system layout. The avionics and the open-source uavAP flight software were integrated into the aircraft, which was recently flight tested in a variety of conditions, confirming aircraft power consumption and production values.

Being applied to many fields of research and industry, UAVs require reliable but modular autopilot software. An autopilot task can range from simple waypoint following to complex maneuvering or adaptive mission tracking. The developed and presented autopilot, uavAP, aims to be fully modular in a decentralized manner, embracing an object-oriented design in C++. It implements a typical control stack comprising of a mission planner, global planner, local planner, and controller. To facilitate its modularity, uavAP makes use of its core, cpsCore, for module management as well as core utilities. cpsCore administers the configuration, aggregation, and synchronization of all the modules in uavAP. With the emulation environment uavEE, uavAP forms an ecosystem for rapid prototyping and testing of modules for various research directions, ranging from scheduling and memory management, through planning and control system design, to flight profile and configuration optimization. The uavAP-uavEE ecosystem has facilitated the design of an accurate UAV power model based on the aircraft’s physical model, flight maneuver automation for aircraft system identification and dynamics parametrization, and an algorithm for geo-fencing of fixed-wing UAVs. This paper describes the control stack of uavAP, its core, cpsCore, as well as application examples highlighting the framework’s modularity and flexibility.

The steadily increasing popularity of Unmanned Aerial Vehicles (UAVs) is creating new opportunities in diverse fields of technology and business. However, this increase of popularity also raises safety concerns. To tackle the primary concern of keeping the UAV inside a designated region, a novel trajectory estimation algorithm for geo-fencing applications is proposed. We derive the Beta-Trajectory that takes into account constraints in curvature as well as constraints in the change of curvature which is bounded by the maximum roll-rate of the aircraft. We incorporate the Beta-Trajectory into a geo-fencing algorithm. By using our open-source uavAP autopilot, the applicability and necessity of accurate trajectory estimation algorithms for geo-fencing applications are shown on small fixed-wing aircraft. The model and algorithm are validated in high-fidelity simulations as well as in real flight testing.

In recent years, we have seen an uptrend in the popularity of unmanned aerial vehicles (UAVs) driven by the desire to apply these aircraft to a variety of civilian, commercial, education, government, and military applications. With the rapid increase in UAV application, significant effort has been put forth into research and development, which culminates with flight testing of the vehicle. Flight testing occurs using either a partialscale or a full-scale prototype and includes a series of maneuvers used to measure and verify the aircraft’s aerodynamics and control behavior. Most often, UAVs are manually piloted through at least the initial flight testing stage where flight qualities are recorded and translated into control tuning. This paper describes a flight testing automation process to streamline the parameterization of an unmanned aircraft’s flight dynamics. The developed flight testing automator commands the aircraft through a predetermined, conditional set of motions and states to induce certain maneuver sets, which allow for dynamics to more easily be parameterized. The desired maneuver sets follow the standards and generally accepted practices for full-scale flight testing. Specifically, the maneuvers of interest presented in this paper include: idle descent, stall, phugoid, doublets, and singlets, which provide the basis for determining the aircraft aerodynamics, longitudinal stability, and control effectiveness, respectively. The flight testing automator was implemented and demonstrated using software-in-the-loop simulation, including a comparison with manually-piloted flight, followed by flight testing using a fixed-wing trainer-type UAV. Automating the data collection process, as opposed to the previous status quo of manual piloting, would allow for more efficient aircraft parametrization and modelling by minimizing trial-and-error and, more importantly, reducing the flight time required.