We propose a unified strategy for fast goal inference in human–robot interaction. The core idea is to drive the human toward Critical Decision Points (CDPs)–states where competing human strategies prescribe different next actions and thus maximally reveal the goal. We formalise CDPs using a goal-conditioned policy divergence measure and incorporate them into a Receding-Horizon Planner that explores future action sequences while optimizing a cost function balancing task progress and information gain. We evaluate this approach in both a collaborative, fully observable cooking task and a competitive, partially observable hide-and-seek game, each in simulation and on real robots. In both scenarios, our method infers human goals more accurately and earlier than baseline strategies.
Unmanned aerial vehicle (UAV) swarm agents operating in radio-frequency (RF)-degraded environments require coordination mechanisms that connect directly observable signals to executable responses. This work presents an embodied visual-communication approach in which bio-inspired motion–LED glyphs are represented by a reduced six-parameter semantic chart embedded within a full 24-dimensional hybrid execution manifold. A learned translator large language model (LLM) maps a perceived glyph to a response that is instantiated as an executable trajectory and propagated through closed-loop quadrotor dynamics. The system is evaluated in 200 three-UAV search-and-rescue trials with receiver-specific degradation from sensing range, field of view, occlusion, and relative motion. Under clean observations, the quantized small model and rule-based translator both achieve 100% semantic correctness, but under single- and two-parameter corruption, the quantized small model achieves 64.7% and 64.1% , compared with 44.8% and 35.2% for rule-based, while reducing mean multi-agent trajectory error from 2.902 m to 0.993 m . All methods maintain 100% rotor-allocation and finite-horizon feasibility. The quantized small model also has comparable overall semantic correctness with the large model ( 83.3% vs. 86.3% ) while dropping latency from 5.115 s to 2.789 s and is validated onboard a Jetson–Pixhawk UAV platform, where airborne inference and bounded command execution produce measurable physical motion. These results establish a unified pathway from degraded visual observation to semantically meaningful, dynamically grounded UAV coordination.
In swarm robotics, stigmergy (indirect communication through changes to the environment) offers an alternative to direct communication and centralised control when these are unavailable. We investigate whether stigmergy alone can solve a group-formation task, namely controlling the number of robots that gather into a group. To this end, a leader robot lays a phosphorescent trail by exciting the arena surface with ultraviolet light; the brightness of the trail decays over time, and its initial intensity encodes the target group size. Following robots join the trail and leave it once the measured brightness falls below a threshold, so that the group size is regulated without any direct inter-robot communication. We demonstrate this mechanism as a proof of concept on real hardware, using a swarm of small Dezibot robots, and show that groups of a specified size form and are maintained over an extended period of time.
Robotic assembly in semi-structured environments represents a critical challenge in flexible manufacturing. While existing hybrid methods benefit from knowledge-based hierarchies, model-based planning, and data-driven learning, they still struggle with small-batch, precise assembly tasks due to loose coupling between components and high training costs. To address these limitations, we introduce Cognition-inspired Robot Learning and Manipulation (CRLM), a neuro-symbolic framework that tightly couples structured task knowledge, semi-supervised perception, and residual reinforcement learning for efficient skill acquisition and robust execution. Inspired by cognitive apprenticeship theory, CRLM operationalizes three developmental phases—modeling, scaffolding, and fading—through a hierarchical manipulation architecture. Specifically, explicit symbolic stage graphs and Object-Embodiment-Centric (OEC) waypoints provide interpretable task decomposition and safety bounds; semi-supervised visual learning bootstraps global and local object detectors from sparse embodied data; and a residual fine policy trained via structured-to-semi (STS) transfer learns contact-aware corrections from multimodal visual and force feedback. By integrating these components into a coherent developmental pipeline rather than treating them as isolated sequential blocks, CRLM achieves robust contact-rich assembly with minimal human engineering. Simulation experiments demonstrate substantial improvements of 13
Significant advances in robot sensing and mobility have enabled the use of multi-robot networks to track multiple targets moving in an environment. Tracking moving targets that outnumber the size of the robot network involves the network to collaboratively assume a subset of targets to be tracked and plan the actions of the robots. This paper focuses on the scenario when the targets outnumber the robots in the network. The goal of the network is to assign targets to the robots and plan robot actions to track these assigned targets. This has to be done consistently as the robots and targets move in the environment to ensure the tracking performance is maximized. However, this problem, as shown in this paper, is NP-hard. This paper leverages decomposition theory to solve this problem efficiently in real-time in two stages. The first stage leverages inter-robot communication in the network to assign robots to targets, and the second stage, solved on each robot locally, optimizes the control to track the targets assigned to it. A novel decentralized approach, called bundle-based assignment, is presented to find adaptive and conflict-free target assignment in the first stage that guarantees 1/2 -approximation in the worst case. Since robots can be assigned more than one target, the second stage optimizing for control is shown to take the form of a multi-objective control problem with conflicting objectives, and a strategy is proposed to solve it for real-time applications. A novel information-gain-based tracking objective is developed, which can be used to solve the two stages, suited specifically for the scenario under consideration. Simulation results show that the novel approach, called bundle-based assignment and control (BBAC), optimizing the novel tracking objective, outperforms existing algorithms and achieves performance very close to that of the optimal solution in a shorter time. Physical experiments with a network of ground robots tracking human targets further validate the applicability of these approaches in the real world.
Autonomous target search is crucial for deploying Micro Aerial Vehicles (MAVs) in emergency response and rescue missions. Existing approaches either focus on 2D semantic navigation in structured environments – which is less effective in complex 3D settings, or on robotic exploration in cluttered spaces – which often lacks the semantic reasoning needed for efficient target search. This paper overcomes these limitations by proposing a novel framework that utilizes a semantically-guided viewpoint planner to minimize target search and exploration time in unstructured 3D environments using an MAV. Specifically, we develop a combinatorial planner that generates efficient semantic exploration plans by prioritizing viewpoints that likely lead to the target. To guide the planner towards the target, an active perception pipeline is developed that propagates semantic priorities of observed objects into neighboring frontier voxels for computing semantic information gains of frontier viewpoints. In addition, we demonstrate how LLM-based similarity scores can be leveraged as semantic priority input to our pipeline. Evaluations in two distinct simulation environments show that the proposed method consistently outperforms baselines by quickly finding the target while maintaining reasonable exploration times. Real-world experiments with an MAV further demonstrate the method's ability to handle practical constraints like limited battery life, small sensor range, and semantic uncertainty.
Healthcare staff shortages increase workloads and create a need for assistive robots capable of handling routine hospital tasks. Traditional rule-based systems struggle in dynamic environments and with unforeseen requests. We present an agile hospital assistive robot that interprets natural language instructions using an AI-based task planner and keyword retrieval to generate executable task sequences. The robot adapts in real time to additional user requests, reschedules tasks using a suboptimal genetic-algorithm-based approach, and recovers from execution failures with vision-language reasoning and AI suggestions. Deployed on a Temi robot with a custom Android application, the system demonstrates effective handling of dynamic changes, improved human-robot interaction efficiency, and positive feedback from nursing staff.
Unmanned Aerial Systems (UASs) prioritizing extended flight times and agility often rely on lightweight camera-based depth estimation systems due to payload constraints, but this study reveals critical vulnerabilities in such navigation approaches. We demonstrate that adversarially designed triggers in black-box settings can significantly compromise UAS navigation, altering flight paths and potentially causing crashes. We demonstrate that an attacker that can infer UAS’s monocular depth estimation outputs by exploiting telemetry or external sensors for a given relative pose, can learn to create an adversarial tracking marker pad. This adversarial marker pad can subsequently be used to induce a controlled bias in the controller’s range/altitude estimates so as to manipulate the UAS flight paths without any access to UAS internals. Our method employs a surrogate model-guided Bayesian optimization framework to generate adversarial depth over the original pad’s semantic search space, maintaining tracking while being robust to environmental and sensor variability. This highlights a critical security concern in UAS tracking/navigation reliant on camera-only depth cues, necessitating the development of more resilient tracking/navigation frameworks.
Ergodic exploration has spawned a lot of interest in mobile robotics due to its ability to design time trajectories that match desired spatial coverage statistics. However, current ergodic approaches are for continuous spaces, which require detailed sensory information at each point and can lead to fractal-like trajectories that cannot be tracked easily. This paper presents a new ergodic approach for graph-based discretization of continuous spaces. It also introduces a new time-discounted ergodicity metric, wherein early visitations of information-rich nodes are weighted more than late visitations. A Markov chain synthesized using a convex program is shown to converge more rapidly to just ergodicity than the traditional fastest mixing Markov chain. The resultant ergodic traversal method is used within a hierarchical framework for active inspection of confined spaces with the goal of detecting anomalies robustly using SLAM-driven Bayesian hypothesis testing. Experiments on a ground robot show the advantages of this framework over typical continuous space ergodic planners as well as greedy and random exploration methods for left-behind foreign object debris detection in a ballast tank.
In unstructured environments, grasping tasks face challenges in grasping pose estimation and insufficient real-time performance. Existing methods struggle to suppress background noise and highlight critical grasping regions, leading to false detections or positioning errors in cluttered environments. To address this, we propose the Grasp Detection Network GFESF-Grasp, which integrates Global Feature Enhancement (GFE) and Skip-Connected Feature Fusion (SF). The GFE module is designed at the bottleneck layer, combining parallel dilated convolutions with differential dilation rates to capture both local details and global context. This expands the effective receptive field, enhancing the network’s ability to perceive cross-scale geometric features of objects. It also integrates Mamba-Like Linear Attention (MLLA) to strengthen feature responses in critical regions like object edges, improving the network’s discriminative power under noisy conditions. During decoding, the SF module adaptively fuses cross-level features, effectively suppressing redundant information while preserving key features. The proposed network achieves 98.9 https://github.com/wangxauat/gfesf-grasp.
The Peg-in-Hole assembly is a fundamental robotic task, but in certain scenarios, contact forces can cause the target to shift, creating a floating target. Traditional methods struggle with overshoot and static errors under these conditions. We propose an iterative target position prediction strategy, utilizing force-torque sensor data to dynamically adjust for floating targets. Our approach effectively addresses the challenges of floating targets. We build a physical prototype of a Peg-in-Hole task of a floating target to test our method.
Multi robot teams are well suited to perform tasks like covering and monitoring large spaces and/or distributing and searching for resources in the environment. To perform these tasks, teams require methods to address the long standing Multi Robot Task Allocation (MRTA) problem. Existing MRTA solutions focus on either assigning individual robots to tasks (microscopic) or modeling team-wide population dynamics deriving time-based robot allocation policies (macroscopic). In dynamic environments, microscopic methods require computationally expensive reallocation strategies and macroscopic methods lack model descriptiveness to achieve time-varying populations without replanning. In this work, we take inspiration from population modeling in other disciplines and present a nonlinear macroscopic ensemble allocation model that describes individual robot collaboration and has the potential for time-varying task assignment without the need for replanning populations. Our results demonstrate a range of possible time-varying task assignment behaviors that are potential solutions to handling known periodic environments or task changes. In addition, we explore the breakdown of classic macroscopic modeling assumptions and present model reinterpretations to mitigate their impact. Our simulation and experimental results demonstrate time-varying task assignment, which could be applied to tasks like environmental monitoring, collective construction, and resource distribution.
Dynamic content in traffic scenes can significantly affect LiDAR-Inertial Simultaneous Localization and Mapping (SLAM). Robust or maximum-likelihood formulations can attenuate inconsistent returns, but when motion is not explicitly modeled, challenges remain. Dynamic motion disrupts scan registration and mapping, leading to ghosting artifacts and a degradation in global map quality. To address this, we propose a dual-stage SLAM framework that constructs a visibility-consistent static subset. The first stage performs multi-resolution range-image differencing and applies visibility-aware incidence-angle correction to restore grazing-angle ground points that are often misclassified. The second stage conducts cluster-level reclassification to eliminate residual dynamics and recover static structures. The refined static points are fused within a tightly coupled factor graph using IMU pre-integration, scan-context loop closure, and a static-prior factor. Odometry features are visibility-gated and confidence-weighted to improve correspondence reliability in dynamic traffic conditions. Evaluations on the UrbanLoco and a self collected dynamic traffic dataset demonstrate up to 11.9
The overlapping coalition formation (OCF) game has emerged as a key framework in multi-agent systems, enabling unmanned aerial vehicles (UAVs) to participate in multiple distinct coalitions simultaneously. This paper studies constrained task allocation in heterogeneous UAV systems using OCF, with a focus on resource heterogeneity, communication constraints, and temporal limitations. We introduce an adaptive exploration–exploitation mechanism that systematically refines the solution space while avoiding premature convergence in coalition optimization. Comparative simulations with state-of-the-art OCF methods and variable neighborhood search (VNS) demonstrate that our approach significantly improves performance in resource-limited settings, achieving a 24.57
Recent advancements in 3D Gaussian Splatting (3DGS) have demonstrated promising results in dense visual SLAM. However, existing Gaussian-based SLAM systems struggle to generate Gaussian maps in sparse views, constrained by memory limitations and real-time performance requirements. This typically leads to two key problems: First, the rendering quality and localization accuracy of Gaussian primitives are highly sensitive to unstable initial point clouds, with errors accumulating during the densification process. Second, when observing Gaussians from multiple views, the attributes of the Gaussian primitives can become overly influenced by the final training view, leading to a forgetting problem where earlier views are not adequately retained. To address these issues, we propose a robust RGB-D SLAM framework that incorporates enhanced spatial distribution and view-consistency optimization. Specifically, we introduce: (1) a texture-density-driven sampling and graph-based structure densification method that uses geometry information to improve Gaussian primitives accuracy, and (2) an optimization strategy based on Gaussian attributes fusion of view-consistency, which explores Gaussian attributes in different views, enabling Gaussian primitives to adapt to various scene perspectives. Our evaluation on Replica and TUM RGB-D datasets demonstrates superior performance, offering new insights for robust 3D reconstruction in resource-constrained systems.
This paper considers a robot moving in a 3D environment that is tasked with estimating a quasi-stationary environmental field (e.g., temperature, concentration of a chemical pollutant, or distribution of light radiation density) in the presence of localization uncertainties, as is typical in underwater or other GPS-denied environments. Gaussian process regression has been widely adopted to model environmental fields. However, a drawback of Gaussian process regression is its difficulty in accounting for data with uncertain input. This work proposes a novel multi-fidelity Gaussian process-based regression approach to address the challenge by splitting the data collected by the robot into different datasets corresponding to the amount of input (localization) uncertainty. Furthermore, a sampling-based trajectory planning algorithm is proposed for adaptive robot exploration that optimizes a field-reconstruction objective function while accommodating resource constraints. The proposed approach is experimentally evaluated using a miniature gliding robotic fish that measures light intensity in a large indoor tank. The adaptive exploration algorithm is tested using both a multi-fidelity Gaussian process model and a baseline single-fidelity model. Two objective functions, based on the information gain and an ergodic metric, respectively, are adopted in the evaluation. The experiments show that, for both objective functions, using multi-fidelity Gaussian process reduces the weighted mean squared error between the model prediction and the ground-truth field compared to using the baseline single-fidelity model that ignores localization uncertainty. Accompanying code available at Coleman (Adaptive exploration under localization uncertainty using multi-fidelity Gaussian processes, 2025, https://github.com/colem404/Adaptive-Exploration-Under-Localization-Uncertainty-Using-Multi-fidelity-Gaussian-Processes/tree/main ).
From refrigerators to kitchen drawers, humans interact with articulated objects effortlessly every day while completing household chores. For automating these tasks, service robots must be capable of manipulating a variety of common articulated objects. Recent deep learning methods have been shown to predict valuable priors on the affordance of articulated objects from vision. In contrast, many other works estimate object articulations by observing the articulation motion, but this requires the robot to already be capable of manipulating the object. In this article, we propose a novel approach combining these methods by using a factor graph for online estimation of articulation, which fuses learned visual priors and proprioceptive sensing during interaction into an analytical model of articulation based on Screw Theory. With our method, a robotic system makes an initial prediction of articulation from vision before touching the object, and then quickly updates the estimate from kinematic and force sensing during manipulation. We evaluate our method extensively in both simulations and real-world robotic manipulation experiments. We demonstrate several closed-loop estimation and manipulation experiments in which the robot was capable of opening previously unseen drawers. In real hardware experiments, the robot achieved a 75
The “Last Mile Challenge” has long been considered an important, yet unsolved, challenge for autonomous vehicles, public service robots, and delivery robots. A central issue in this challenge is the ability of robots to navigate constrained and cluttered environments that have high agency (e.g., doorways, hallways, corridor intersections), often while competing for space with other robots and humans. We refer to these environments as “Social Mini-Games” (SMGs). Traditional navigation approaches designed for MRN do not perform well in SMGs, which has led to focused research on dedicated SMG solvers. However, publications on SMG navigation research make different assumptions (on centralized versus decentralized, observability, communication, cooperation, etc.), and have different objective functions (safety versus liveness). These assumptions and objectives are sometimes implicitly assumed or described informally. This makes it difficult to establish appropriate baselines for comparison in research papers, as well as making it difficult for practitioners to find the papers relevant to their concrete application. Such ad-hoc representation of the field also presents a barrier to new researchers wanting to start research in this area. SMG navigation research requires its own taxonomy, definitions, and evaluation protocols to guide effective research moving forward. This survey is the first to catalog SMG solvers using a well-defined and unified taxonomy and to classify existing methods accordingly. It also discusses the essential properties of SMG solvers, defines what SMGs are and how they appear in practice, outlines how to evaluate SMG solvers, and highlights the differences between SMG solvers and general navigation systems. The survey concludes with an overview of future directions and open challenges in the field. Our project is open-sourced at .
Robots should be able to learn complex behaviors from human demonstrations. In practice, these human-provided datasets are inevitably imbalanced: i.e., the human demonstrates some subtasks more frequently than others. State-of-the-art methods default to treating each element of the human’s dataset as equally important. So if—for instance—the majority of the human’s data focuses on reaching a goal, and only a few state-action pairs move to avoid an obstacle, the learning algorithm will place greater emphasis on goal reaching. More generally, misalignment between the relative amounts of data and the importance of that data causes fundamental problems for imitation learning approaches. In this paper we analyze and develop learning methods that automatically account for mixed datasets. We formally prove that imbalanced data leads to imbalanced policies when each state-action pair is weighted equally; these policies emulate the most represented behaviors, and not the human’s complex, multi-task demonstrations. We next explore algorithms that rebalance offline datasets (i.e., reweight the importance of different state-action pairs) without human oversight. Reweighting the dataset can enhance the overall policy performance. However, there is no free lunch: each method for autonomously rebalancing brings its own pros and cons. We formulate these advantages and disadvantages, helping other researchers identify when each type of approach is most appropriate. We conclude by introducing a novel meta-gradient rebalancing algorithm that addresses the primary limitations behind existing approaches. Our experiments show that dataset rebalancing leads to better downstream learning, improving the performance of general imitation learning algorithms without requiring additional data collection. See our project website: https://collab.me.vt.edu/data_curation/ .
