This paper presents a lidar‐only state estimation and tracking framework, along with a roadside sensing unit for integration with existing urban infrastructure. Urban deployments demand scalable, real‐time tracking solutions, yet traditional remote sensing remains costly and computationally intensive, especially under perceptually degraded conditions. Our sensor node couples a single lidar with an edge computing unit and runs a computationally efficient, GPU‐free observer that simultaneously estimates object state, class, dimensions, and existence probability. The pipeline performs (i) state updates via an extended Kalman filter, (ii) dimension estimation using a 1D grid map/Bayesian update, (iii) class updates via a lookup table driven by the most probable object footprint, and (iv) existence estimation from track age and bounding box consistency. Experiments in dynamic urban‐like scenes with an instrumented passenger vehicle and additional road users demonstrate real‐time performance. Quantitative tracking accuracy is reported for the vehicle, while detection and classification performance are evaluated across all 48 annotated tracks. The complete end‐to‐end pipeline finishes within 100 ms for 99.88% of messages, with an excellent detection rate. Robustness is further confirmed under simulated wind‐induced sensor vibration. These results indicate that reliable, real‐time roadside tracking is feasible on CPU‐only edge hardware, enabling scalable, privacy‐friendly deployments within existing city infrastructure. The framework integrates with existing poles, traffic lights, and buildings, reducing deployment costs and simplifying large‐scale urban rollouts and maintenance efforts.

A monocular vision-based pose estimation framework is developed for autonomous real-time navigation in dynamic environments without using the global navigation satellite system (GNSS). The framework models distances from landmarks and static objects as a function of object segmentation information obtained from monocular visual inputs using Gaussian process regression (GPR), and utilizes a nonlinear optimization and an optimal variance filter with covariance adaptation to estimate the mobile robot/vehicle pose, subject to various initial state and error covariance conditions (even without intermitted updates from GNSS). The stability of the developed state observer and the boundedness of the estimation error covariance are also investigated. The real-time performance of the designed framework is experimentally evaluated using a vehicle test platform in various dynamic environments and under perceptually degraded conditions, which include intermittent visual inputs and different navigational scenarios. Consistent and accurate pose estimation, bounded estimation error, and computational efficiency of the proposed method are confirmed by comprehensive tests in highly-dynamic urban settings. The maximum processing time for the state estimator is less than 10 ms, making the framework computationally efficient for any visual input stream lower than 30-50 Hz. Furthermore, a visual perception system to customize essential map elements, including static landmarks and road features, are provided and experimentally evaluated in busy downtown and university campus areas.

Reliable learning-based high-level decision making for lane changes and speed control in automated driving must accommodate dynamically sized inputs due to varying scene traffic flow. DeepSet and its variants represent the state of the art among shared-encoder approaches; however, they neglect explicit traffic interaction modeling, limiting performance in negotiation-intensive scenarios such as intersections. Attention-based methods capture interactions among static and dynamic agents, but incur quadratic memory and computational complexity and provide limited control over representation granularity. Inspired by Perceiver IO, an attention-based architecture, DecisionPerceiver, is proposed to project dynamic agent features into a fixed-size latent space, where feature granularity is regulated by the number of latent queries, improving scalability for larger networks. A finer discretization of the action set is further proposed to increase the performance gain due to interaction awareness. Extensive evaluations across three driving scenarios that require different levels of interaction awareness demonstrate consistent performance gains and generalization across various navigation objectives. In addition, the proposed architecture is assessed in scenarios with an increasing number of vehicles to demonstrate scalability.

The growing deployment of multiple robots in dynamic scenes and unstructured settings requires scalable and robust estimation techniques, particularly those that leverage distributed sensor networks (DSN). This paper introduces a distributed observer framework designed for state estimation and tracking of autonomous mobile robots in LiDAR sensor networks. The framework incorporates remote state estimation and intermittent onboard inertial data to improve tracking accuracy and scalability. It reduces communication overhead and minimizes computational burden on DSN through a reliable clustering approach, addressing the challenges posed by large data volumes. In addition, a 3D bounding box detection module is developed to accommodate various sensor configurations, enabling robust and efficient object tracking in unstructured environments. The consensus properties of the distributed state estimators is analyzed, and the asymptotic stability of the corresponding estimation error dynamics is formally established. The proposed framework is validated through extensive experimental studies involving occlusions in dynamic scenes, demonstrating high estimation accuracy, consistency, and computational efficiency across a broad range of operating conditions.

Estimating vehicle velocity in the full-range of vehicle operation across various driving scenarios is a challenge for most model-based state observers, especially for vehicles operating at the limits of handling due to highly nonlinear vehicle dynamics, particularly on slippery roads. To address this challenge, this paper proposes a machine learning (ML) based method for vehicle velocity estimation. Two novel vehicle velocity estimators based on multi-layer feedforward neural network (FNN) are designed to estimate vehicle longitudinal and lateral velocities, respectively. The inputs of the proposed FNN estimators are selected based on a sensitivity analysis of vehicle dynamics models and are normalized by vehicle parameters to ensure that the trained FNN estimators have generic formulas transferable to vehicles of different specifications. Experimental and simulation results—including tests on vehicles not seen during training—demonstrate that the proposed velocity estimators provide accurate estimations in various challenging scenarios and are robust to differences in vehicle specifications. Moreover, the computational cost is low enough for real-time implementation.

A high-definition (HD) map development framework providing real-time visualization of multimodal perception data for state estimation, motion planning, and decision-making in autonomous navigation is presented and experimentally validated. The proposed framework integrates synchronized visual and LiDAR data and generates consistent frame transformations to construct accurate and interpretable HD maps suitable for navigation in dynamic environments. In addition, the framework enables flexible customization of essential map elements, including road features and static landmarks, facilitating efficient map generation and visualization. Building upon the developed HD map visualizer, a semantic-aware visual odometry (VO)-based pose estimation module is designed and verified through extensive evaluations and under perceptually degraded conditions. To ensure the reliability of synchronized multimodal data used by downstream perception and pose estimation modules, a sensor health monitoring system is also developed and validated in urban canyon scenarios with intermittent or unavailable global navigation satellite system (GNSS) measurements. Experimental results demonstrate that the proposed HD map visualizer and associated perception modules are transferable for autonomous navigation and can be effectively employed as benchmarking tools for state estimation and motion planning algorithms in autonomous driving.

Inertial navigation systems augmented with visual and wheel odometry measurements have emerged as a robust solution to address uncertainties in robot localization and odometry. This paper introduces a novel data-driven approach to compensate for wheel slippage in visual-inertial-wheel odometry (VIWO). The proposed method leverages Gaussian process regression (GPR) with deep kernel design and long short-term memory (LSTM) layers to model and mitigate slippage-induced errors effectively. Furthermore, a feature confidence estimator is incorporated to address the impact of dynamic feature points on visual measurements, ensuring reliable data integration. By refining these measurements, the system utilizes a multi-state constraint Kalman filter (MSCKF) to achieve accurate state estimation and enhanced navigation performance. The effectiveness of the proposed approach is demonstrated through extensive simulations and experimental validations using real-world datasets. The results highlight the ability of the method to handle challenging terrains and dynamic environments by compensating for wheel slippage and mitigating the influence of dynamic objects. Compared to conventional VIWO systems, the integration of GPR and LSTM layers significantly improves localization accuracy and robustness. This work paves the way for deploying VIWO systems in diverse and unpredictable environments, contributing to advancements in autonomous navigation and multi-sensor fusion technologies.

The accurate and reliable localization is crucial for the safe performance of autonomous vehicles in uncertain and complex environments such as urban areas. Thus, this paper contributes by presenting a feature-based localization framework for autonomous vehicles where segmented map features are used for designing soft and adaptive constraints for Particle Filtering. After obtaining features of landmarks (light poles) in instance- based segmented images acquired from a monocular camera, vehicle-to-landmark distances are predicted using Gaussian Process Regression models (GPR). Both mean and variance (uncertainty) outputs of Gaussian Process models were used to implement adaptive soft constraints. Experimental results over the WATonoBus from the University of Waterloo confirm that the use of the proposed constraints improves the vehicle states estimation. Using instance segmentation increases the vehicle-to-landmark distance prediction accuracy by 27% and keeps the real-time processing capability of the proposed approach. Besides, the proposed approach reliably localizes the vehicle in very noisy conditions using only a monocular camera, even with a reduced number of landmarks, reducing the localization Euclidean error from 2.55 m to 0.97 m when compared to an unconstrained Particle Filter.

Conventional visual navigation methods presume scene stability and encounter challenges due to moving objects in highly dynamic environments. We propose DynaNav-SVO, a stereo visual odometry (VO) framework, which semantically detects and constructs a region-of-interest (ROI) by focusing on a-priori urban fixed elements for reliable feature extraction and subsequently estimates vehicle pose. The outcome is a static map with minimal outliers and is used for state estimation in dynamic scenes and perceptually degraded conditions. This map enhances computational efficiency due to the reduced size of the new static mask (as confirmed in several experiments), compared to the existing visual simultaneous localization and mapping (vSLAM) solutions. To refine the estimated pose, a back-end module selects a moving horizon of frames, generates a covisibility graph for data association, and optimizes a structure-from-motion program using local bundle adjustment. Finally, the performance of the framework is experimentally evaluated using a test vehicle in highly-dynamic urban settings and under adverse weather conditions with degraded visual perception with varying sequence lengths. The experiments confirm excellent performance in terms of estimation accuracy and computational efficiency for autonomous navigation compared to existing vSLAM methods.

Ensuring accurate and efficient perception and motion planning is critical for the safety of autonomous vehicles. Addressing these pivotal challenges, this paper introduces a novel motion planning method employing a Lidar point cloud-based potential field (PF). Our approach innovatively extracts the drivable area boundary from point cloud, enhancing computational efficiency and reducing common perception errors, such as missed detections and inaccurate obstacle shape estimation. Built upon this drivable area boundary, the PF effectively represents the cost of traversing diverse areas. The PF is integrated into a model predictive control (MPC) framework to generate control commands considering vehicle dynamics, constraints, collision avoidance, and passenger comfort. Given the highly nonlinear nature of simultaneous longitudinal and lateral motion planning, an efficient Frenet frame-based trajectory sampling method is developed to provide an initial guess of the optimal trajectory for this complex motion planning task. The perception module has been validated in real bus tests, confirming its reliability and efficiency, and the entire motion planning methodology has been rigorously tested through simulations. These simulations show that our method efficiently generates smooth and safe control commands, even in challenging scenarios where the obstacle vehicle suddenly changes its lane, and remains robust under considerable state observation noise.

While demand for assistive technology has risen with aging populations and concomitant increase in mobility disabilities, conventional (passive) walker designs have demonstrated safety and usability limitations. Robotic rollators (or 4-wheeled walkers) have been proposed to address concerns, including slip, fall, and collision risks. To develop control systems for robotic rollators, accurate estimation of the states is required. While model-based estimation approaches have been widely investigated for mobile robots, robotic rollators present unique challenges due to model parameter changes and uncertainties. In contrast, data-driven estimation approaches require sufficient excitation modes during learning to address corner cases. The proposed learning-aided state estimation (L-ASE) method augments an unscented transformation observer with a long short term memory (LSTM) based learning algorithm to estimate rollator states by using on-board inertial measurement unit data and wheel speeds. The stability and boundedness of the error covariance is investigated. The developed learning-aided estimation method is also experimentally verified for the walker-assisted gait and demonstrates superior performance using a robotic rollator platform in rigorous testing conditions.

This article introduces a slip-aware networked vehicle model and proposes an adaptive time-delay control framework for connected autonomous driving systems’ cooperative adaptive cruise control and safety of the intended functionality. In order to improve the vehicular network safety by conveying the amount of longitudinal slip ratio along with the vehicle kinematic states, the innovative slip-aware model makes use of an auxiliary state variable representation that includes wheel slips. The proposed framework enables the formation of platoons consisting of vehicles from multiple manufacturers with similar dynamics, with each vehicle requiring state measurements and longitudinal slip information only from its preceding vehicle. To ensure robustness against external disturbances and model uncertainties, an artificial time-delayed control-based technique is implemented to control the entire networked vehicle system. In order to achieve disturbance rejection along the string of vehicles, control protocols have to be designed to ensure string stability of the whole vehicle platoon. The robust law is augmented with a dual-rate adaption law in order to tackle the overestimation and underestimation problem of switching gain. Subsequently, Lyapunov stability analysis is conducted to show that the inter-vehicular states are steered within a small region in the neighborhood of the origin, under the proposed adaptive-robust control scheme. Numerical simulations are also carried out to validate the robustness of the developed distributed control framework.

Autonomous driving systems must operate smoothly in human-populated indoor environments, where challenges arise including limited perception and occlusions when relying only on onboard sensors, as well as the need for socially compliant motion planning that accounts for human psychological comfort zones. These factors complicate accurate recognition of human intentions and the generation of comfortable, socially aware trajectories. To address these challenges, we propose SAP-CoPE, an indoor navigation system that integrates cooperative infrastructure with a novel 3D human pose estimation method and a socially-aware model predictive control (MPC)-based motion planner. In the perception module, an optimization problem is formulated to account for uncertainty propagation in the camera projection matrix while enforcing human joint coherence. The proposed method is adaptable to both single- and multi-camera configurations and can incorporate sparse LiDAR point-cloud data. For motion planning, we integrate a psychology inspired personal-space field using the information from estimated human poses into an MPC framework to enhance socially comfort in human-populated environments. Extensive real-world evaluations demonstrate the effectiveness of the proposed approach in generating socially aware trajectories for autonomous systems.

Autonomous mobility systems increasingly operate in dense and dynamic environments where perception occlusions, limited sensing coverage, and multi-agent interactions pose major challenges. While onboard sensors provide essential local perception, they often struggle to maintain reliable situational awareness in crowded urban or indoor settings. This article presents the Cloud-based Autonomous Mobility (CAM) framework, a generalized architecture that integrates infrastructure-based intelligent sensing with cloud-level coordination to enhance autonomous operations. The system deploys distributed Intelligent Sensor Nodes (ISNs) equipped with cameras, LiDAR, and edge computing to perform multi-modal perception and transmit structured information to a cloud platform via high-speed wireless communication. The cloud aggregates observations from multiple nodes to generate a global scene representation for other autonomous modules, such as decision making, motion planning, etc. Real-world deployments in an urban roundabout and a hospital-like indoor environment demonstrate improved perception robustness, safety, and coordination for future intelligent mobility systems.

An interaction-aware control and motion planning framework is proposed and experimentally verified for time-critical merging scenarios. The framework considers interaction between the automated driving system and other vehicles, including human-driven vehicles, by monitoring lateral and longitudinal response of the neighbor vehicle without communication or having access to their control and safety objective functions. This has not been accounted for safe motion planning and controls in existing merging solutions in mixed traffic. The framework includes a novel inverse differential game based on a long short-term memory network for estimation of the possible path tracking objective function of the human-driven vehicle in real-time. Then, a game-theoretic receding horizon controller is devised for the automated driving system by predicting the trajectory of the human-driven vehicle. The developed framework is validated in several merging scenarios and road surface conditions using CarSim high-fidelity simulations including human-in-the-loop case studies with different test subjects.

Autonomous vehicles must negotiate with pedestrians in ways that are both safe and socially compliant. We present an interaction-aware model predictive decision-making (IAMPDM) framework that integrates a gap-acceptance-inspired intention model with MPC to jointly reason about human intent and vehicle control in real time. The pedestrian module produces a continuous crossing-propensity signal - driven by time-to-collision (TTC) with an intention discounting mechanism - that modulates MPC safety terms and minimum-distance constraints. We implement IAMPDM in a projection-based, motion-tracked simulator and compare it against a rule-based intention-aware controller (RBDM) and a conservative non-interactive baseline (NIA). In a human-in-the-decision-loop study with 25 participants, intention-aware methods shortened negotiation and completion time relative to NIA across scenarios, at the expense of tighter TTC/DST margins, with no significant difference between IAMPDM and RBDM except for TTC in one scenario. Results indicate that intention-aware decision-making algorithms reduce pedestrian crossing time and improve subjective ratings of comfort, safety, and trust relative to a non-cooperative decision-making algorithm. We discuss implications for real-world deployment of interaction-aware autonomous vehicles. We detail decision-making calibration and real-time implementation (CasADi/IPOPT) and propose deployment guardrails - minimum surrogate-safety margins, deadlock prevention - to balance efficiency with safety.

Accurate state estimation of dynamic objects is critical for safe and reliable autonomous navigation in dynamic urban settings. Real-world scenarios may include occlusions, drifting in pose estimation, data sparsity, or abrupt appearances, representing complexity in multi-object tracking. This paper addresses the challenges of estimating the states of non-ego vehicles during autonomous navigation using multimodal visual-LiDAR perception, without any need to global navigation systems. The proposed estimation framework integrates detection and tracking through an optimal variance filter. The first stage incorporates instance segmentation and point cloud association, followed by an L-shape fitting method to estimate 3D Bounding Boxes. Object tracking is then performed using a modified multiobject tracking algorithm augmented by the initialization of motion models. This enables estimation of consistent surrounding vehicle velocities and positions while keeping track of them in the 3D space, enhancing situational awareness for motion planning, and supporting collision avoidance using predictive models. Comprehensive experiments on the KITTI dataset demonstrate the effectiveness and accuracy of the proposed framework compared to current benchmarks and state-of-the-art methods.

Vehicle-infrastructure (V2I) cooperative perception can substantially extend the range, coverage, and robustness of autonomous driving systems beyond the limits of onboard-only sensing, particularly in occluded and adverse-weather environments. However, its practical value is still difficult to quantify because existing benchmarks do not adequately capture large-scale multi-node deployments, realistic communication conditions, and adverse-weather operation. This paper presents CoInfra, a deployable cooperative infrastructure perception platform comprising 14 roadside sensor nodes connected through a commercial 5G network, together with a large-scale dataset and an open-source system stack for V2I cooperation research. The system supports synchronized multi-node sensing and delay-aware fusion under real 5G communication constraints. The released dataset covers an eight-node urban roundabout under four weather conditions (sunny, rainy, heavy snow, and freezing rain) and contains 294k LiDAR frames, 589k camera images, and 332k globally consistent 3D bounding boxes. It also includes a synchronized V2I subset collected with an autonomous vehicle. Beyond standard perception benchmarks, we further evaluate whether infrastructure sensing improves awareness of safety-critical traffic participants during roundabout interactions. In structured conflict scenarios, V2I cooperation increases critical-frame completeness from 33

Indoor perception is a field that has gained traction in recent years. While there has been a significant amount of research done on outdoor perception and motion planning, the indoor environment has yet to receive similar treatment. In an indoor environment, various sensor systems have been developed to track and localize objects, each tackling a different set of challenges. In this article, we introduce a novel infrastructure sensor node (ISN) consisting of a light detection and ranging (LiDAR) along with two monocular cameras mounted on the ceiling of the hallways of our laboratory to obtain relevant information. We present a perception pipeline that uses prior 3-D point cloud registration to localize objects in real time in dynamic indoor environments. We provided a complete case study to present a work that successfully detects, registers, and localizes objects through a dynamic environment with a high degree of occlusion.