IEEE International Conference on Automation Science and Engineering (CASE) : [proceedings]

Leveraging the power of a graph neural network (GNN) with message passing, we present a Monte Carlo Tree Search (MCTS) method to solve stochastic orienteering problems with chance constraints. While adhering to an assigned travel budget, the algorithm seeks to maximize collected reward while incurring stochastic travel costs. In this context, the acceptable probability of exceeding the assigned budget is expressed as a chance constraint. Our MCTS solution is an online and anytime algorithm, alternating planning and execution, that determines the next vertex to visit by continuously monitoring the remaining travel budget. The novelty of our work is that the rollout phase in the MCTS framework is implemented using a message-passing GNN, predicting both the utility and failure probability of each available action. This allows to enormously expedite the search process. Our experimental evaluation shows that with the proposed method and architecture, we manage to efficiently solve complex problem instances while incurring moderate losses in terms of collected reward. Moreover, we demonstrate how the approach is capable of generalizing beyond the characteristics of the training dataset. The paper's website, open-source code, and supplementary documentation can be found at ucmercedrobotics.github.io/gnn-sop.

To address the challenges of hyperparameter tuning and reliance on manual experience in dual-channel pulse-coupled neural networks (DCPCNN) for image fusion tasks, the paper proposes a data-driven Bayesian maximum entropy (DBME) multi-objective optimization method to enhance image fusion performance. Firstly, the source images are transformed into the NSST domain with low-frequency bands and high-frequency bands. Secondly, we design a DCPCNN model based on DBME optimization to fuse the high-frequency subbands. For the low-frequency subbands, a method based on weighted local energy and multi-scale morphological gradient fusion rule is proposed. Finally, the fused image is reconstructed by the NSST inverse transform. The results demonstrate that, compared to other fusion methods, this approach outperforms in the fusion of infrared and visible light images, as well as multi-focus images, with significant advantages in metrics such as AG, SD, SCD, and VIFF.

Collaborative loco-manipulation by multiple quadruped manipulators enables handling bulky, heavy objects beyond the capabilities of individual robots. However, coordinating robot teams while navigating complex terrains and avoiding obstacles remains challenging. We propose a hierarchical control framework consisting of a model predictive control (MPC)-based manipulation planner with integrated obstacle avoidance, a geometry-aware mapping converting object trajectories into robot commands, and decentralized loco-manipulation MPC controllers. The framework supports collision-free collaborative manipulation tasks and enhances payload capacities. Validation through simulation and real-world hardware experiments with diverse quadruped robot teams demonstrates the approach’s effectiveness, robustness, and practical applicability.

Robotic unscrewing plays a pivotal role in the automation of remanufacturing processes. Despite advancements in deep learning-based screw detection, which nowadays achieve detection accuracies exceeding 98%, consistent unscrewing success remains challenging due to physical localization errors. Within prior research various vision-based methods have been developed to correct positional deviations that arise from neural network predictions. On the other hand, force-based methods have been developed to compensate for all accumulating errors during physical contact. The vision-based approaches often lack reliability, and force-based methods, although highly effective, have to date, only been validated on larger, self-centering fasteners and will result in additional wear to the unscrewing end-effector. Therefore, this study presents a comparative analysis of two robotic unscrewing pipelines designed to enhance detection accuracy, localization precision, and unscrewing time efficiency. The first baseline approach employs a single-stage, coarse-only sense-plan-act approach. The second, advanced two-stage approach integrates coarse localization with a refinement step using stereo detection, matching and triangulation. Experimental results highlight the advantages of the advanced two-stage approach, which incorporates a region-of-interest recommendation step and an eyes-in-hand stereo module for position refinement. With the pilot system developed in the presented research, vision-based precision and recall rates of 99.2% and 99.4% are obtained, with an unscrewing success rate of 99%. At the system level, these results correspond to unscrewing precision and recall of 98.2% and 98.4% with only 0.75 seconds needed to reclassify and compute the updated screw location.

With Industry 4.0 advancing, manufacturing requires real-time data acquisition, device connectivity, and cloud-based analytics. However, the diversity of equipment and legacy systems within factories often hinders a seamless transition to intelligent production. To address this, this paper proposes a three-layer cloud factory architecture featuring a platform called OPC UA Fusion. Multiple OPC UA servers share data in real-time, enabling machine-to-machine scheduling and linking to a NoSQL database and the remote OPC UA Fusion Cloud. Two case studies validate the feasibility of the platform. With the help of cloud analytics, the results show improved interoperability and visualisation. Future work includes better OPC Classic support and advanced scheduling to further enhance fully automated intelligent manufacturing capabilities.

The transition from Industry 4.0 to Industry 5.0 has highlighted the critical role of Modular Integrated Construction (MiC), particularly in rapidly deployable modular hospitals that address urgent healthcare demands. As the final stage before delivery, fit-out directly impacts both project speed and healthcare quality. However, scheduling in this phase faces challenges from dynamic labor allocation and worker fatigue, which traditional methods struggle to handle in out-of-distribution (OOD) settings. To tackle this, we reformulate the problem as a Flexible Job-shop Scheduling Problem with Workload Constraints (WL-FJSP) and propose a memory-augmented framework that models worker-task dynamics. By incorporating adaptive gating mechanisms, the model captures fatigue variations and jointly optimizes medical task fulfillment and fit-out efficiency. Experiments show improved performance over traditional and state-of-the-art methods, with strong generalization across varying instance scales.

Hydrodynamic real-time hybrid simulation (hydro-RTHS) is a testing approach that couples physical and numerical sub-assemblies through actuators and sensors in real time, offering an enhanced-fidelity alternative to small-scale experiments; e.g., testing of a floating offshore wind turbine (FOWT) in a hydrodynamic laboratory. To address multi-degree-of-freedom (DOF) actuation in a unique, floating environment, a Franka Emika Panda robotic arm was integrated into the actuation approach for hydro-RTHS. The actuation system emulates the 6-DOF small-scale aerodynamic forces computed from a numerical model acting on a small-scale FOWT specimen in a wave basin. A task-prioritized control strategy was designed to apply forces with secondary "pose-keeping" control to limit the robot’s configuration drift. The system was experimentally validated for a 1:50-scale FOWT specimen for operational wind-only cases in still water at the O.H. Hinsdale Wave Research Laboratory. Results demonstrate consistent force tracking across all six DOFs and agreement between measured platform motions and OpenFAST numerical simulations of the complete FOWT. These results illustrate the feasibility of using an off-the-shelf robotic arm for 6-DOF force control in hydro-RTHS, expanding actuation options and enhancing force control capabilities.

We propose a multi-environment motion-teaching approach for a robotic auto-photographing system designed to replicate an inspector’s viewpoint during remote visual inspection of train bogies. The system utilizes a head-mounted device (HMD) to record the inspector’s gaze points and head pose, and then determine the robot’s camera pose based on them Previous works extracted camera poses based on dense clusters of gaze points recorded by the HMD, assuming that the inspector’s gaze points are concentrated. However, this approach fails to capture large components, where the gaze naturally moves along the surface. Additionally, these methods require the environments for HMD data collection and robot operation to be identical, limiting inspections in locations inaccessible to human inspectors. To address these limitations, we propose a method that leverages gaze speed information in addition to a 3D clustering method to identify both small and large components. The method also adjusts camera poses, enabling the robot to capture images in diverse environments. Experiments conducted in a mock-up environment replicating a real train bogie for HMD data collection, as well as simulations using robot and operational environment models, confirm that the proposed method allows the robot to successfully capture images of all the small and large components, even in environments different from the one where the HMD data was collected.

The teaching process in robotic coating operations remains heavily reliant on operator expertise, often requiring repetitive cycles to achieve the desired quality. This increases lead times and material waste. To address this, we propose a predictive simulation system for electrostatic coating, enabling smart manufacturing by automating coating path generation. Our system integrates history-dependent kinematic modeling of the coating material before the nozzle and hydrodynamic behavior after application, based on rheological properties. Unlike prior studies, which were limited to non-electrostatic applications, our method incorporates particle dynamics under electrostatic force using the Navier-Stokes equation. It also considers variables such as voltage, discharge rate, spray angle, and gun speed. We validated this digital twin approach through physical experiments, achieving a coating film thickness prediction accuracy within +/- 15 mu m. This enables parameter-based process optimization without requiring repeated physical trials. Based on this system, we developed a scale-selectable coating strategy tailored to H-beams, commonly used in industrial settings. The strategy supports full automation across various web and flange dimensions (Web: 168-420 mm, Flange: 0-70 mm), replacing manual processes. The effectiveness of the strategy was confirmed through both simulation and experimental validation, demonstrating the potential to reduce worker exposure to spatter and to enhance safety and efficiency in automated manufacturing.

We propose a low-rank multi-modal shearing and splicing fusion (LMSSF) method for accurate emotion recognition by effectively integrating the information from three modalities: text, image, and voice. Recognizing users’ emotions accurately using social media information is challenging due to the diversity of user-generated data and the difficulty in accurately identifying and extracting features from multi-modal information. Our method fills the gap in the multi-modal field by leveraging feature extraction and fusion techniques to combine voice, image, and text modalities for emotion recognition. To address the interaction of multi-modal information, we introduce standard feature extraction and private feature retention methods to ensure the integrity of the multimodal information. Further-more, we have developed a step-by-step discrimination approach that significantly reduces calculation and discrimination time by distinguishing standard features of three modalities, common features of two modalities, and private features. Our method effectively solves the problem of accurately recognizing emotions in social media users with diverse information modalities, achieving 97.3% accuracy with only 180K and an accuracy improvement of close to 10% than others. Code is available at https://github.com/wjm666666/LMSSF.

The limited productivity growth within the construction industry in the last decades has increasingly driven the development of innovative manufacturing processes. Especially the expanding research field of robotic additive manufacturing in construction (AMC) is said to enhance the flexibility and efficiency. In particular, the usage of mobile manipulators as 3D printers enables the creation of manufacturing environments that are not constrained by the reach of the robotic arm. While initial approaches have implemented mobile manipulators that print from stationary positions before relocating, more sophisticated approaches focus on print-while-drive. Print-while-drive eliminates the risk of inducing weakening cold joints into the component during repositioning and further enhances the flexibility of the printing process. Existing approaches to print-while-drive rely exclusively on mobile manipulators with holonomic drives, such as Mecanum wheels. However, due to their design, Mecanum wheels are not suitable for use on uneven, contaminated, or loosely deposited surfaces. Such conditions, however, are common on construction sites. The application of alternative drive concepts, such as differential drive systems — commonly employed in track driven platforms — necessitates the development of novel trajectory-planning concepts for mobile manipulators. To this end, this publication proposes a trajectory planning algorithm to derive a suitable mobile platform trajectory based on a given tool center point (TCP) / printing trajectory. The functionality of the developed algorithm is demonstrated and evaluated by simulating the trajectories for large-scale components.

This paper proposes a method for topological mapping and navigation using a monocular camera. Based on AnyLoc, keyframes are converted into descriptors to construct topological relationships, enabling loop detection and map building. Unlike metric maps, topological maps simplify path planning and navigation by representing environments with key nodes instead of precise coordinates. Actions for visual navigation are determined by comparing segmented images with the image associated with target nodes. The system relies solely on a monocular camera, ensuring fast map building and navigation using key nodes. Experiments show effective loop detection and navigation in real and simulation environments without pre-training. Compared to a ResNet-based method, this approach improves success rates by 63.8% on average while reducing time and space costs, offering a lightweight solution for robot and human navigation in various scenarios.

In discrete manufacturing systems, the concurrent production of customized orders increases the complexity of multi-order remaining completion time (MORCT) prediction. Most existing methods rely on empirical estimates of production capacity, ignoring production planning constraints and failing to incorporate real-time manufacturing performance, resulting in limited prediction accuracy. To address this, a DT-based MORCT prediction framework is proposed. It constructs spatiotemporal datasets of production tasks and performance from DT and introduces a spatiotemporal multi-graph cascade algorithm (STMG) for prediction. The algorithm first embeds data into two types of graph models, which contain manufacturing system information and order-task scheduling information, extracts real-time performance features across multiple manufacturing units (MUs), and applies heterogeneous graph attention to model the embedded features of order nodes in the order-task-MU graph for final MORCT prediction. The experimental results show that the STMG model can accurately predict MORCT and is suitable for parallel production scenarios with multiple orders. It outperforms existing methods in terms of loss and adaptability.

In today’s data-driven supply chain management landscape, establishing connections among stakeholders from diverse resources is essential for effective analysis and decision-making. Knowledge graphs (KGs) provide a transformative solution by organizing fragmented inventory data into semantically rich, interconnected networks, enabling contextualized insights and robust reasoning. However, automating KG construction for supply chain datasets is challenging due to issues such as heterogeneous data integration (e.g., text documents, spreadsheets), domain-specific contextualization, and the need to model implicit operational dependencies. This paper introduces a novel framework that leverages large-language models (LLMs) with multi-step prompting workflow to address these challenges. Our AutoKG4SC approach automates the extraction of entities from various sources and constructs KGs to capture complex interdependencies among these entities. We utilize zero-shot prompting for ontology construction, Named Entity Recognition (NER), and Relation Extraction (RE) tasks, thereby eliminating the need for extensive domain-specific training and human prior knowledge. We validate the framework through a case study that demonstrates AutoKG4SC’s ability to construct high-quality KGs from supply chain datasets. This research presents an effective framework and prompt strategy for KG construction, which can be easily adapted to datasets with richer information and other application scenarios.

This paper presents a new cognitive architecture to enable human-robot collaborative assembly in complex, unstructured environments. While existing human-robot collaboration technologies have demonstrated success in simple setups, they struggle with fluent interaction in complex scenarios characterized by unpredictable human intentions and flexible workspace configurations. Our approach addresses these limitations by developing a cognitive architecture built upon the SOAR architecture, emphasizing internal cognitive processes, including real-time learning, adaptive decision-making, and knowledge evolution. The proposed system integrates perception, learning, memory, and execution components into a unified architecture that enables robots to continuously acquire skills through human interaction. Through HRC experiments for assembling the Generic Assembly Box (GAB), we demonstrated a 6.74% improvement in task success rates, a 10.9% reduction in execution time, and a 15.6% decrease in human instruction needs over traditional methods. These results validate the system’s potential to bridge the gap between traditional cognitive architectures and practical robotic applications, contributing to more adaptive and intelligent human-robot collaboration.

Diffusion-based policy learning has shown strong performance across diverse robotic tasks, often achieving high success rates. However, real-world deployment requires more than task success—it demands efficient execution and the ability to handle complex environments. In many assembly and disassembly settings, a single scene contains multiple potential task goals. This can confuse learned policies, leading to ambiguous behavior. Enabling explicit task selection via natural language is thus crucial for robust and flexible operation. In this paper, we address two key challenges: (1) improving task execution efficiency by structuring tasks into distinct sub-task modes using language, and (2) resolving goal ambiguity by allowing human operators to specify desired tasks through natural language commands. We further introduce an adaptive parameter selection mechanism that adjusts reliance on different sensory modalities depending on the active sub-task. We evaluate our approach on the NIST Task Board, a representative benchmark with multiple co-located task goals. Our method improves execution speed by 57% and increases task success rate by 19% compared to baseline approaches. Demonstration videos are available at: https://rros-lab.github.io/task-aware-diffusion/.

This paper presents the SP-Diff parallel gripper system, addressing the limited adaptability of conventional end-effectors in intelligent industrial automation. The proposed design employs an innovative differential linkage mechanism with a modular symmetric dual-finger configuration to achieve linear-parallel grasping. By integrating a planetary gear transmission, the system enables synchronized linear motion and independent finger pose adjustment while maintaining structural rigidity, reducing Z-axis recalibration requirements by 30% compared to arc-trajectory grippers. The compact palm architecture incorporates a kinematically optimized parallelogram linkage and Differential mechanism, demonstrating adaptive grasping capabilities for diverse industrial workpieces and deformable objects such as citrus fruits. Future-ready interfaces are embedded for potential force/vision sensor integration to facilitate multimodal data acquisition (e.g., trajectory planning and object deformation) in digital twin frameworks. Designed as a flexible manufacturing solution, SP-Diff advances robotic end-effector intelligence through its adaptive architecture, showing promising applications in collaborative robotics, logistics automation, and specialized operational scenarios.

This paper presents and evaluates methods of fusing semantic image segmentation predictions, and highlights a novel hybrid approach that combines spatial frequency and edge features. Tool-labeled endoscopy from sinus surgery served as the image dataset, while two methods of surgical tool segmentation via morphological polar transform provided distinct predictions. The morphological transform acted as an input pre-processing step prior to segmentation via the U-Net architecture. Two separate predictions were available for each image based on the transformation center: one at the surgical tool-tip (TT) and one at the surgical tool vanishing point (VP). The goal in this work was to systematically generate a superior segmentation by fusing information from the two aforementioned predictions. Improved segmentation performance in this domain is envisioned to enable vision-based force estimation in robot-assisted minimally invasive surgery (RMIS), where lack of reliable force and tactile feedback has continued to be an ongoing challenge. While methods for deep learning based segmentation fusion exist, such methods require extensive datasets and potentially obfuscate explainability. Thus, three approaches relying solely on low-level features to fuse gray-scale segmentation predictions were proposed in this work: (1) gradient estimation, (2) Laplacian pyramid and (3) a modified spatial frequency method. The latter two demonstrated enhanced segmentation compared to original predictions. This work also explores explainability towards identifying candidate prediction pairs for fusion via unsupervised clustering as well as a ResNet-18 model. Cursory investigations into properties of the fused predictions provide insight into the potential use of the proposed methods in domains other than surgical tool segmentation.

Traditional omniwheel-based ballbots typically use three omniwheels for locomotion, offering high maneuverability but limited stability and load capacity. Introducing a fourth wheel can address these limitations; however, it also makes torque distribution underdetermined, requiring an additional constraint to distribute motor torques effectively. This paper presents a normal force based torque distribution strategy for a four-wheeled omniwheel ballbot. The proposed method leverages real-time normal force measurements to dynamically adjust torque limits, reducing wheel slippage and improving stability. The effectiveness of the proposed strategy is evaluated through recovery angle and braking experiments. Compared to the baseline symmetric torque distribution strategy, incorporating normal force feedback reduces wheel slippage by 43-74% during recovery and by 76% during braking. These improvements demonstrate the efficacy of the normal force based torque distribution strategy in enhancing stability.

The Skill Orchestration Agent (SkillOA) introduces a modular, distributed approach to manufacturing control, enhancing flexibility beyond traditional programmable logic controllers (PLCs). It is capable of determining and executing ad-hoc orchestrations of skills—representing manufacturing functions—by combining two sub-areas of AI: semantic knowledge graphs and multi-agent systems.By decomposing production orders into executable skills, the SkillOA concept enables reconfiguration and efficient resource utilization during operative processes. A core component is its semantic knowledge graph, which dynamically determines optimal skill sequences, reducing engineering complexity, and system downtime. The queue-based execution model prioritizes service request, ensuring adaptability in high-mix, low-volume production. The integration of parallel and asynchronous execution strategies enhances process efficiency but also introduces system complexity, requiring robust synchronization mechanisms. Challenges include managing execution dependencies, ensuring interoperability across automation architectures, and refining error-handling mechanisms. The presented concept represents a significant step toward autonomous, reconfigurable manufacturing, aligning with Industry 4.0 principles.