In this work, we investigate how to leverage learning processes to design a perception-driven control framework for robot motion. In this regard, inspired by the fact that potential-based control represents an effective approach for modeling robotic tasks, we study how neural networks can be effectively exploited to approximate unknown perception-based potential functions, for which an analytical closed form may not even be available, thus extending the field of applicability of potential-based control. Numerical results along with an experimental validation are provided to empirically demonstrate the validity of the proposed control architecture.
Autonomous navigation for unmanned ground vehicles (UGVs) in complex environments requires robust perception for reliable traversability estimation and path planning. Traditional geometric methods often fail in unstructured terrains, necessitating robust scene understanding. Current 3D semantic segmentation methods are mostly directed towards structured environments and are not generalisable for off-road domains. To address this, we propose modifications to LiDAR semantic segmentation by incorporating spatial context and adding an auxiliary task of learning point-wise ground-relative height to capture robust features. Experiments on both real-to-real and synthetic-to-real transfer demonstrate that the proposed approach outperforms state-of-the-art methods in segmentation accuracy and adaptability, offering a scalable solution for UGV perception in diverse outdoor environments.
This paper introduces a technical solution for efficient rebar tying by developing a multi-arm rebar tying robot system. We introduced a multi-arm collaboration to achieve parallel task execution, which is 2.6 times more efficient than a single-arm system. A real-time dynamic adjustment algorithm based on market auctions ensures a coordinated and efficient direct operation of robotic arms. An optimized hybrid path planning method is used to improve the positioning accuracy of the rebar intersection, reducing the positioning error to 0.59 mm. Through modeling and simulation analysis, we show that the system can increase efficiency to 62% under different working conditions and minimize labor requirements.
Autonomous terrain leveling and material handling operations play a pivotal role in earthworks, directly affecting project timelines, cost efficiency, and environmental impact. We present the Optimal Orchestration Transport Planner Framework (OOTPF), a unified orchestration architecture for multi robot systems that casts surface leveling as an online Multi Agent Pickup and Delivery (MAPD) problem. OOTPF comprises two principal layers: an Optimal Transport layer that formulates material movement via discrete Earth Mover's problems, and a Task Allocation layer that employs heuristic bidding and consensus based auctions to assign pickup-delivery jobs to agents. To guarantee safe, collision free execution, we embed a token driven, priority based planning discipline atop a shared space time Reservation Table, invoking Safe Interval Path Planning (SIPP) for single robot requests and Conflict Based Search (CBS) when multiple robots contend. This hierarchical scheme dynamically balances throughput with conflict resolution. Numerical experiments on representative leveling scenarios demonstrate that OOTPF achieves high utilization, zero collisions, and flexible to import different path planner algorithms.
Vision-based coordination of Unmanned Aerial Vehicles (UAVs) presents challenges in target tracking, formation control, and real-time adaptability under dynamic conditions. This work proposes a unified Nonlinear Model Predictive Controller (NMPC) framework that fully integrates perception-driven maneuvering, energy-efficient trajectory generation, formation maintenance, and robust information acquisition for vision-based target pursuit in UAV swarms. The framework shapes UAV trajectories to maintain proximity to a moving target while optimizing perception-related objectives, such as centering the target in each UAV’s field of view (FOV). Multi-agent collaboration is exploited to enhance target state observability by dynamically distributing UAVs around the target, while soft constraints ensure operational feasibility in rapidly changing environments. The proposed methodology is validated through extensive simulations across diverse target behaviors and environmental scenarios, demonstrating robustness, formation stability, and computational efficiency. Real-world experiments with a single UAV confirm the framework’s viability for operating in real-time, performing computations on-board. The results highlight the framework’s potential for scalable multi-agent UAV applications in surveillance, search and rescue, and environmental monitoring, combining perception-driven control with robust multi-agent coordination.
We present a hierarchical planning methodology for approximating Pareto-optimal policies in Multi-Objective Markov Decision Process (MOMDP) models. These models describe missions in which a mobile robot must navigate an environment and perform actions at specific locations. Our approach relies on clustering the state space of the full MOMDP into hierarchical subproblem MOMDPs. We then build the set of Pareto-optimal policies for these sub-problem MOMDPs, and treat them as macro-actions in a high-level MOMDP which selects the policy to use for each of the sub-problems, as well as the order in which to address them. Our bottom-up approach synthesises approximations of Pareto-optimal policies for large problems while providing precise performance guarantees. We empirically evaluate our method, showing it achieves substantial scalability gains over a non-hierarchical approach while preserving high-quality solutions.
Assistive robots can greatly enhance the autonomy and quality of life of mobility-impaired individuals, who make up approximately 16% of the global population, by supporting daily tasks in home environments. However, most current systems rely on rigid, non-intuitive interfaces and struggle with natural language understanding, spatial reasoning, and robust navigation. Vision-Language Navigation (VLN) offers a more accessible alternative by enabling robots to interpret and act on human language grounded in visual context. This paper presents a VLN system implemented on a TIAGo robot in a ROS2 Gazebo simulation. The system integrates YOLO11-Seg for real-time object detection, Real-Time Appearance-Based Mapping (RTAB)-Map for Simultaneous Localization and Mapping (SLAM), and GPT-4o-mini for natural language parsing. Detected objects are converted into 3D point clouds and clustered via Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to identify accurate object centroids, enabling semantic-aware navigation through the Nav2 stack. Experimental results show that the system performs reliably across components: YOLO11-Seg delivers high segmentation accuracy (Dice 0.94) at real-time speeds, the LLM consistently interprets natural language commands correctly, and the navigation module maintains goal accuracy within 0.7 m. The integrated system successfully completes complex tasks and recovers from failures, highlighting the potential of off-the-shelf AI for real-time assistive navigation.
Anomaly detection and model interpretation are key components for robots deployed in safety-critical scenarios. In this paper, we propose to use Sentinel —a Transformer-based architecture for multivariate time series forecasting —to improve anomaly detection performance for mobile robots, and we investigate whether the model’s attention mechanisms faithfully reflect the underlying statistical structure of the data. Our results on the ALFA dataset (a widely used aerospace benchmark) demonstrate that Sentinel achieves good anomaly detection performance when compared to state-of-the-art approaches. Moreover, the empirical evaluation shows that sentinel’s attention mechanisms capture relevant dependencies among the features hence offering key insight for early warning indicators. These findings highlight the potential of attention-based interpretability in complex, sensor-rich robotic environments and pave the way towards explainable and resilient anomaly detection frameworks.
Pedestrian trajectory prediction is crucial for human-robot interaction applications. In this paper, we employ maps of spatial motion patterns combined with diffusion models to improve prediction accuracy. Our method projects both input and output trajectories into an embedding space, leveraging spatial motion patterns to bias anchors that represent the general motion flow within corresponding environments. These place-dependent spatial motion patterns are previously learned from prior environmental observations, enabling better adaptation to new surroundings. Diffusion models are then applied to generate multi-modal trajectory predictions based on the anchors. Experimental results demonstrate that our method outperforms existing models in domain adaptation and long-term trajectory prediction, showcasing the effectiveness of maps of spatial motion patterns in adapting to new environments and improving trajectory prediction performance.
This paper studies collaborative exploration of an initially uncharted environment, employing a tandem composed of an unmanned aerial vehicle and an unmanned ground vehicle. The proposed method harnesses the complementary capabilities of both platforms, which exhibit distinct heterogeneous characteristics, to enhance exploration performance. The UAV offers high maneuverability and a broad aerial vantage point but with restricted payload while the UGV, despite its kinematic limits, provides ground-level stability, extended sensing capabilities, and superior payload capacity. The approach takes advantage of their combined strengths to improve exploration and coverage with two distinct strategies based on Next Best View and frontier exploration. A frontier extraction and redefinition method is proposed in order to limit the number of candidate viewpoints to the most reachable ones, which also speeds up evaluation. Simulations were conducted to test and discuss the scenarios and to highlight the practical relevance of the proposed system.
The problem of coordinating the movements of a group of agents that perform pickup and delivery operations in a known environment is called Multi-Agent Pickup and Delivery (MAPD). Warehouses, where mobile robots complete transportation tasks composed of pickup and delivery operations, are the typical domain of application of MAPD. Over the years, the number of robots employed in these settings has gradually increased to the order of hundreds, leading to new challenges in the scalability of algorithms. In this paper, we introduce a new algorithm called hierarchical Token Passing (h-TP) to face this problem through a logical segmentation of the environment that enables the parallel computation of agents’ paths. We test the performance of h-TP in simulated warehouse environments and compare it against a classical algorithm for solving MAPD problems that does not operate in parallel. The results show a reduction of the total execution time of about 50% at the cost of a small worsening of the solution quality.
Due to the demographic shift towards an aging population, there is growing interest in the development of Person Navigation Systems (PNS), which support people with impairments and their caregivers. However, there is a lack of algorithms for navigating these systems that behave socially and simultaneously take the passenger comfort factors, such as side tendency, into account. In this paper, we propose a novel method that incorporates side tendency into the path planning for PNS or social robots using local sensor data. The sensor data of our intelligent wheelchair is utilized to approximate the walls on the left and right. Based on these detections, a predefined comfort function is applied to assign lateral costs in the costmap according to the estimated width of the corridor, representing the side tendency. Utilizing dynamic sensor data, the proposed approach facilitates adaptation to changes or unknown environments, thus demonstrating its superiority over previous methods based on a static global map.
This paper introduces a novel wheeled-grasping hybrid robot with the ability to vertically climb columns and rotate about the column axis via a wheel drive system mounted on a turret. One of the defining features focused on in this work is the robot’s underactuated tendon-driven modular arms, which allow the robot to autonomously grasp columns for climbing and be easily adjusted to different column diameters. Mathematical models are developed to describe the kinematics and statics of the modular underactuated tendon-driven systems in the context of these grasping arms. Additionally, models describing the relationship between the tendon forces and the friction resisting the climbing motion are also presented. Simulations and experimental results validate the kinematics models, shed light on the real-world arm statics, and demonstrate the viability of the robot’s pre-climb grasping deployment mechanism. Ultimately, this work not only emphasizes the potential of these robots in industries that would benefit from autonomous climbing, but it also provides guidance for similar tendon-driven designs and hybrid climbers in the future.
Localization and navigation in unstructured environments are essential functions of autonomous mobile robots, with their application domains continuing to expand across various fields. In this context, this paper focuses on the problem of localization in orchard environments. Because the trees are typically planted at relatively wide intervals, they are well-suited to be used as landmarks. This paper investigates two key aspects of utilizing tree landmarks for robust localization. The first is to improve odometry accuracy by introducing a triangle-based signature that enables reliable landmark correspondence. The second is to evaluate how the quality of a landmark-based map affects localization accuracy, by comparing maps generated through 2D SLAM and 3D SLAM. The proposed localization method is tested in a real-world orchard environment to demonstrate its effectiveness as a foundation for autonomous navigation.
Underground mines present inherently hazardous and complex environments, characterized by poor visibility, confined geometries, and uneven terrain. Autonomous machines offer a promising means to reduce human exposure in such conditions; however, their deployment is hindered by numerous challenges, including the absence of Global Navigation Satellite Systems (GNSS), sensor degradation, and the difficulty of maintaining reliable localization and situational awareness in such environments. This work presents the retrofitting of scaling machine for autonomous operation within an active mining site, detailing the sensor integration, system architecture and automation. We incorporate state-of-the-art mapping and localization algorithms, and leverage existing Wi-Fi access points to provide coarse global positioning within the mine. Experimental validation conducted on a live mining environment demonstrates the system’s ability to maintain accurate localization and consistent mapping under real-world conditions.
Accurate soil sampling is essential for sustainable crop production, and the automation of this process using robotic systems is an important step to make this process faster and more efficient. In this paper, the individual parts of this process are analyzed: soil sampling using drill and preparation of soil samples for analysis using custom made scoop. For soil sampling with a drill, two different drill types (a soil drill and a wood drill) were tested on two soil types, and the heights and weights of the extracted samples were compared. A feedback control loop was created to regulate the drill’s motion to avoid too high torque along the lateral axis. In the soil preparation phase, the custom made scoop was used for preparing a soil sample, where the force required to penetrate and compress the sample was evaluated, to extract an optimal amount depending on the soil type and moisture level. Also soil height before and after compression are analyzed. The results indicate that, due to its specific geometry, the wood drill is more suitable for the application than the soil drill. For scoop control, commercial substrate and higher moisture soil required less force than mixed cultivated soil and low moisture. Depending on the soil type and moisture, the best results were achieved for a force between 30N and 60N, and especially for 50N to 60N.
The paper introduces build123things, a software tool for consistent robot design, manufacturing, and physical modeling. It is motivated by designing robotic systems with multimodal locomotion that require multiple simulation models to evaluate their motion capabilities using multimodal planning techniques. The tool uses a Hierarchical Assembly Graph and an algorithm to select the kinematic root, leveraging the Code-CAD library build123d and the OpenCASCADE geometry kernel. It employs explicit reference geometries to overcome the Topological Naming Problem (TNP) of existing mainstream Computer Aided Design (CAD) systems relying on implicit geometry element naming. Rich material annotations and assembly semantics enable use across design, simulation, and manufacturing, narrowing the sim-to-real gap. Its usage is demonstrated in a robotic arm design and a climbing magnetic robot case study, where build123things supports multimodal planning by implementing kinematic hierarchy transforms.
Intelligent control has been extensively applied to aerial robots, which is important for autonomous flight applications. However, it is susceptible to noise and uncertainties when integrating different sensors and nonlinear aerial robot dynamics. This paper proposes a robust, neural network-based adaptive hybrid control framework for a unified position and orientation tracking in Hexarotors. The proposed framework is considered for receiving uncertain data from altitude and orientation sensors. It further tackles estimating these time-varying noises, uncertainties, and unmodeled dynamics by synergistically integrating a second-order sliding mode control (SMC) with radial basis function neural networks (RBFNNs). The RBFNN-augmented SMC dynamically adapts control commands, with stability proven via the Lyapunov method. These advancements ensure a simultaneous and accurate position and orientation tracking under varying conditions. The framework is validated through extensive software and hardware in the loop (SIL/HIL) simulations, showcasing significant improvements in tracking accuracy, disturbance rejection, and adaptability compared to traditional methods.
Dense annotation of real 3D LiDAR point clouds for mobile robot applications remains challenging. Unsupervised Domain Adaptation (UDA) enables the segmentation of unlabeled real-world point clouds by leveraging labeled synthetic data. However, existing self-training-based UDA methods rely on fixed thresholds for pseudo-label selection, limiting adaptation performance. In this work, we address this limitation. We propose a novel UDA framework for 3D LiDAR semantic segmentation, centered on a confidence-guided pseudo-label sampling strategy (ConSamp). Specifically, ConSamp adopts a probabilistic sampling strategy in which pseudo-labels with higher confidence are more likely to be retained. Meanwhile, the sampling function itself evolves adaptively throughout training to respond to changes in confidence distribution. Experiments show that our model achieves strong performance on synthetic-to-real 3D LiDAR semantic segmentation tasks. In particular, results better than state-of-the-art methods have been achieved on two public 3D point cloud datasets: SemanticKITTI [1] and SemanticPOSS [2].
In this work, we address the problem of persistent environmental monitoring using autonomous robotic teams. In prior work, we cast the task as an online patrolling problem using a fleet of Autonomous Surface Vehicles (ASVs), in which the location importance is dynamically inferred from pollutant concentration levels. However, the limited speed of ASVs can reduce performance when the phenomenon to be measured is dynamic. The availability of faster - but less precise - Unmanned Aerial Vehicles (UAVs) can reduce this issue. In this work, we investigate how using heterogeneous fleets of both ASVs and UAVs improves environmental monitoring, by combining the strengths and smoothing the limitations in terms of speed and accuracy of these two types of robots. To do so, we leverage a Variational Autoencoder based on the UNet architecture (VAE-Unet) to generate global maps of pollutant distribution from sparse and noisy measurements gathered by ASVs and UAVs. This enables informed patrolling decisions in complex, dynamic environments. Simulations inspired by a real-world use case demonstrate that our approach improves mapping accuracy and patrolling effectiveness compared to baseline and where only ASVs are used.
