3D Gaussian Splatting has recently emerged as a powerful representation for photorealistic rendering and reconstruction of complex scenes. However, its practical applications in augmented/virtual reality, digital‐twin, and robotics demand accurate and structurally consistent meaningful 3D segmentation, which remains a significant challenge. Existing 3D segmentation approaches, predominantly based on multiview 2D images, frequently rely on appearance‐driven criteria, resulting in semantic misclassification—either incorrectly merging distinct object parts or excessively fragmenting coherent regions. Moreover, these methods significantly struggle with objects with multiple components and occluded scenes. To address these limitations, we propose an interactive human‐in‐the‐loop segmentation framework that combines a fast optimization‐based 3D segmentation algorithm with intuitive finger‐based user interactions within a virtual reality environment. Our optimization‐based segmentation module runs within a few seconds (tens of times faster than existing learning‐based methods) providing users with real‐time visual updates on current segmentation results, enabling them to refine outputs interactively by adjusting prompts and viewpoints in a human‐in‐the‐loop manner. Our finger‐based interface system allows precise 3D spatial prompting, enabling accurate and multiview consistent prompts, thereby overcoming the limitations of traditional 2D multiview prompts and segmentation. This combination significantly improves segmentation accuracy, semantic consistency, and robustness to occlusion and multipart structures, as demonstrated by experimental results showing fine‐grained subpart segmentation in cluttered scenes.

Replicating real-world environments into simulation by realistic visual representation like NeRF and 3D Gaussian Splatting (3DGS) has emerged as an effective strategy to reduce the sim-to-real gap in robot learning. However, implementing object articulation during the real-to-sim process is still a challenging task. Existing motion tracking or learning based articulation methods shows low success rates on complex kinematic structures having multiple joints. Furthermore, those methods require scan of dynamic motion of objects, which makes reconstruction process much complicated. In this work, we propose the first end-to-end pipeline that reconstructs simulation-ready assets with accurate articulation from a single static object video input through suggestion based human-in-the-loop process. Our approach exports a hybrid representation combining 3DGS for photorealistic rendering and mesh-based geometry for physical interaction. In the reconstruction process, our pipeline performs convex decomposition followed by user grouping for intuitive part segmentation, subsequently binding 3D Gaussians to the corresponding mesh parts. An Automatic Joint Suggestion Algorithm then calculates candidate joint axes from local boundary geometries and presents them to users for efficient articulated asset reconstruction. We have shown that our method achieves precise articulation results on partnet-mobility-v0 dataset and real objects. Additionally we presented a potential usage of our framework on robot learning, deploying the reconstructed assets in Unreal Engine and NVIDIA Isaac Sim, demonstrating real-time dexterous hand manipulation tasks.

Narrow passage path planning is a prevalent problem from industrial to household sites, often facing difficulties in finding feasible paths or requiring excessive computational resources. In this article, we propose a homotopy optimization method tailored for the narrow passage problem, utilizing a novel collision constraint interpolation method using signed distance functions. The framework begins by decomposing the environment into convex objects and representing it as a simplicial complex based on their connectivity. This representation enables topological analysis to induce an easy-to-hard sequence of collision constraint interpolation that preserves homotopy equivalence. Using this collision constraint interpolation, the optimization proceeds through a series of subproblems, gradually guiding the path to the final solution. Several examples are presented to demonstrate how the proposed framework addresses narrow passage path planning problems.

Safe, precise teleoperation demands a third-person 3D view that reveals collision clearances and task-critical geometry in full detail. Yet most systems still rely on live camera streams that offer tunnel-vision perspectives and weak depth cues, hiding hazards and denying operators the spatial context for precise manipulation. 3D Gaussian Splatting (GS) renders photorealistic views in real time, yet safe, efficient multi-view acquisition in cluttered teleoperation remains a bottleneck. We propose Human-in-the-Loop Gaussian Splatting (HIL-GS) that delivers safe, robust, and efficient 3D scene reconstruction for challenging teleoperation environments. HIL-GS combines three modules in a tightly-coupled loop: (1) motion-aware GS reconstruction that fuses RGB-D and proprioceptive sensors for drift-free and robust mapping under aggressive motions; (2) VR-based informative display that renders the GS map with contextual overlays/ feedback in real time to ensure situational awareness and reconstruction completeness; and (3) finger-based control interface to guide the robot toward informative viewpoints through safe, non-redundant motions. Through simulation and real-world experiments, we demonstrate that HIL-GS outperforms traditional approaches in reconstruction quality, usability, and efficiency.

The multi-contact nonlinear complementarity problem (NCP) is a naturally arising challenge in robotic simulations. Achieving high performance in terms of both accuracy and efficiency remains a significant challenge, particularly in scenarios involving intensive contacts and stiff interactions. In this article, we introduce a new class of multi-contact NCP solvers based on the theory of the Augmented Lagrangian (AL). We detail how the standard derivation of AL in convex optimization can be adapted to handle multi-contact NCP through the iteration of surrogate problem solutions and the subsequent update of primal-dual variables. Specifically, we present two tailored variations of AL for robotic simulations: the Cascaded Newton-based Augmented Lagrangian (CANAL) and the Subsystem-based Alternating Direction Method of Multipliers (SubADMM). We demonstrate how CANAL can manage multi-contact NCP in an accurate and robust manner, while SubADMM offers superior computational speed, scalability, and parallelizability for high degrees-of-freedom multibody systems with numerous contacts. Our results showcase the effectiveness of the proposed solver framework, illustrating its advantages in various robotic manipulation scenarios.

Collision detection is a fundamental problem in robotics, but handling collisions between non-convex objects remains challenging. A common approach for representing non-convex geometry is a signed distance function (SDF). Voxel-based SDF (VoxelSDF) enables fast distance queries but suffers from discretization artifacts and high memory costs. Neural implicit SDF (NeuralSDF) provides a continuous and memory-efficient representation with generalization, yet their slow query speed has limited their use in collision detection. To overcome these limitations, this paper proposes a novel amortized NeuralSDF–mesh collision detection framework. NeuralSDF–mesh collisions are formulated as a constrained optimization problem at the triangle level, and the Karush–Kuhn–Tucker conditions are derived to enable the amortization. A learning-based amortized optimization directly predicts collisions in a single forward pass, eliminating iterative optimization procedures. The amortized model adopts an auto-decoder architecture, extending the advantages of NeuralSDF in memory efficiency and category-level generalization to collision detection. Experiments demonstrate substantial speedups over baseline methods while maintaining comparable contact quality and reduced memory usage. The proposed approach also exhibits category-level generalization to unseen objects and can be applied to various robotic simulation scenarios.

Human-swarm interaction (HSI) in 3D environments faces critical challenges, including the high degrees of freedom (DOFs) of large swarms and limited operator spatial awareness. To address these issues, we introduce a novel finger-based HSI interface capable of managing 100 or more agents. The interface integrates three core interaction methods—Attraction, Repulsion, and Relaxed-mapping—along with auxiliary utilities and viewpoint controls, leveraging finger dexterity for expressive and responsive swarm manipulation. We conducted rigorous human-subject studies across three scenarios: pattern formation, collective exploration, and coordinated navigation. Results demonstrate that our interface significantly outperforms the baseline in performance and workload, primarily due to efficient, implicit viewpoint control. We also found strong evidence for scenario-dependent optimality, where the effectiveness of interaction methods varied by task demands. Scalability analysis revealed that performance in macro-management tasks remained constant regardless of swarm size, whereas micro-management tasks scaled linearly. Furthermore, while objective performance and perceived workload generally correlated, user preference sometimes diverged when performance gains were marginal, highlighting the importance of intuitiveness. This study provides empirical insights for designing adaptive, context-aware HSI systems for large-scale human-swarm collaboration.

Hand tracking plays a key role in capturing and transferring dexterous human manipulation skills to robots. However, achieving reliable tracking across diverse conditions and during complex interactions (e.g., object manipulation) remains challenging. A promising solution is to combine wearable sensors such as IMUs with vision, where previous studies have handled the vision input by attaching markers to wearables or by relying on depth data to avoid the domain gap in color images. In this work, we present a hand tracking framework that fuses inertial measurements with state-of-the-art vision methods, eliminating the need for markers while fully exploiting visual cues. For this, we introduce a dataset generation scheme that produces synthetic and real data for the target glove using a compact setup, without manual annotation. Using the dataset, we train the keypoint detection network that predicts the likelihood of an image for keypoints, designed based on a lightweight vision transformer (ViT) for real-time usage. Based on the network prediction, the IMU-propagated pose is used as a prior in probabilistic inference to estimate the keypoint positions and uncertainties. Tracking primarily relies on high-rate IMU updates for fast motion estimation, while the pose is corrected through factor graph optimization. The framework is validated in challenging scenarios, demonstrating its robustness and accuracy, and can be used for high-quality demonstration data acquisition and teleoperation for dexterous manipulation.

We propose a novel inverse kinematics (IK) solver based on an optimization-simulation duality, in which the conventional optimization-based IK problem is reformulated as a forward dynamics simulation of a virtual mechanical system. The IK solution emerges as the equilibrium of this system, corresponding to the Karush-Kuhn-Tucker (KKT) conditions of the original problem. To ensure the stability of the discrete-time forward simulation, we adopt the passive midpoint integrator (PMI) [Kim et al., 2017], which enforces a discrete-time passivity. This allows the proposed method to converge to an equilibrium without the need for line search. The resulting solver is lightweight, scalable, and suitable for real-time inverse kinematics computation in high-degree-of-freedom (DOF) systems, while satisfying joint limits, obstacle avoidance, and other task-specific constraints. The efficacy of the algorithm is validated and demonstrated through extensive simulations on various robotic platforms.

Narrow passage path planning is a prevalent problem from industrial to household sites, often facing difficulties in finding feasible paths or requiring excessive computational resources. Given that deep penetration into the environment can cause optimization failure, we propose a framework to ensure feasibility throughout the process using a series of subproblems tailored for narrow passage problem. We begin by decomposing the environment into convex objects and initializing collision constraints with a subset of these objects. By continuously interpolating the collision constraints through the process of sequentially introducing remaining objects, our proposed framework generates subproblems that guide the optimization toward solving the narrow passage problem. Several examples are presented to demonstrate how the proposed framework addresses narrow passage path planning problems.

Precise hand tracking is critical for realistic finger-based interaction in VR applications, yet tracking errors are inevitable. Since the real hand remains unseen in many applications like VR and teleoperation, imprecise proprioception allows users to tolerate bounded tracking errors. This paper quantifies this threshold for wearable hardware and shows how cutaneous feedback expands it. First, we establish a baseline detection threshold of 5.04 cm for synthetic random offsets. Second, using an IMU-based testbed reproducing realistic errors, we find users are more sensitive to these realistic dynamics, with the threshold dropping to 3.87 cm. We demonstrate that synchronized plausible touch sensations can mask this error, consistently enlarging tolerance by approximately 1 cm for synthetic (5.04 -> 6.0cm) and realistic (3.87 -> 4.93cm) errors. These results provide concrete accuracy guidelines for hand tracking systems and prove that haptics can relax stringent accuracy requirements.

In this paper, we introduce time-correlated model predictive path integral (TC-MPPI), a novel approach to mitigate action noise in sampling-based control methods. Unlike conventional smoothing techniques that rely on post-processing or additional state variables, TC-MPPI directly incorporates temporal correlation of actions into stochastic optimal control, effectively enforcing quadratic costs on action derivatives. This reformulation enables us to generate smooth action sequences without extra modifications, using a time-correlated and conditional Gaussian sampling distribution. We demonstrate the effectiveness of our approach through simulations on various robotic platforms, including a pendulum, cart-pole, 2D bicopter, 3D quadcopter, and autonomous vehicle. Simulation videos are available at https://youtu.be/nWfJ2MAV2JI.

Shape abstraction, simplifying shape representation into a set of primitives, is a fundamental topic in computer vision. The choice of primitives shapes the structure of world understanding, yet achieving both high abstraction accuracy and versatility remains challenging. In this paper, we introduce a novel framework for shape abstraction utilizing a differentiable support function (DSF), which offers unique advantages in representing a wide range of convex shapes with fewer parameters, providing smooth surface approximation and enabling differentiable contact features (gap, point, normal) essential for downstream applications involving contact-related problems. To tackle the associated optimization and combinatorial challenges, we introduce two techniques: differentiable shape parameterization and hyperplane-based marching to enhance accuracy and reduce DSF requirements. We validate our method through experiments demonstrating superior accuracy and efficiency, and showcase its applicability in tasks requiring differentiable contact information.

We propose a novel high-force/high-precision interaction control framework of a dual-arm robot system on a flexible base, with one arm holding, or making contact with, a supporting surface, while the other arm can exert any arbitrary wrench in a certain polytope through a desired pose against environments or objects. Our proposed framework can achieve high-force/precision tasks by utilizing the supporting surface just as we humans do while taking into account various important constraints (e.g., system stability, joint angle/torque limits, friction-cone constraint, etc.) and the passive compliance of the flexible base. We first design the control as a combination of: 1) nominal control; 2) active stiffness control; and 3) feedback wrench control. We then sequentially perform optimizations of the nominal configuration (and its related wrenches) and the active stiffness control gain. We also design the proportional-integral type feedback wrench control to improve the robustness and precision of the control. The key theoretical enabler for our framework is a novel stiffness analysis of the dual-arm system with flexibility, which, when combined with certain constraints, provides some peculiar relations, that can effectively be used to significantly simplify the optimization problem-solving and to facilitate the feedback wrench control design by manifesting the compliance relation at the interaction port. The efficacy of the theory is then validated and demonstrated through simulations and experiments.

We propose a novel real-time excavation trajectory modulation framework on a slope for an autonomous excavator with a low-level digital kinematic control as common for hydraulic industrial excavators. Excavation on a slope is challenging because of a higher risk of slips and rollovers. To deal with this, we propose a real-time excavation trajectory modulation framework based on slope tangential/normal force ratio mu and zero moment point xi. The slip and rollover prevention conditions are incorporated in a single linear inequality using the same fractional structure in mu and xi with the common denominator. However, due to the adoption of the low-level digital kinematic control, this prevention requires the prediction of the excavation force at the next timestamp, and, for this, we develop a data-driven excavation force difference prediction model utilizing a deep learning architecture, Transformer. The remaining error of this prediction is then addressed by using the technique of robust optimization with box uncertainty of the developed excavation force difference model. Our proposed framework is validated experimentally with our customized scaled-down excavator.

We propose a novel cross-source point cloud registration (CSPR) method for USV-AAV cooperation in lentic environments. In the wild outdoors, which is the typical working domain of the USV-AAV team, CSPR faces significant challenges due to platform-domain problems (complex unstructured surroundings and viewing angle difference) in addition to sensor-domain problems (varying density, noise pattern, and scale). These characteristics make large discrepancies in local geometry, causing existing CSPR methods that rely on point-to-point correspondence based on local geometry around key points (e.g. surface normal, shape function, angle) to struggle. To address this challenge, we propose the novel concept of a directional correspondence-based iterative cross-source point cloud registration algorithm. Instead of using point-to-point correspondence under large discrepancies in local geometry, we build correspondence about directions to enable robust registration in the wild outdoors. Also, since the proposed directional correspondence uses bearing angle and normalized coordinate, we can separate scale estimation with transformation, effectively resolving the problem of different scales between two point clouds. Our algorithm outperforms the state-of-the-art methods, achieving an average error of 1.60 degrees for rotation and 1.83% for translation. Additionally, we demonstrated a USV-AAV team operation with enhanced visual information achieved with the proposed method.

This paper presents a framework designed to tackle a range of planning problems arise in manipulation, which typically involve complex geometric-physical reasoning related to contact and dynamic constraints. We introduce the Contact Factor Graph (CFG) to graphically model these diverse factors, enabling us to perform inference on the graphs to approximate the distribution and sample appropriate solutions. We propose a novel approach that can incorporate various phenomena of contact manipulation as differentiable factors, and develop an efficient inference algorithm for CFG that leverages this differentiability along with the conditional probabilities arising from the structured nature of contact. Our results demonstrate the capability of our framework in generating viable samples and approximating posterior distributions for various manipulation scenarios.

Recent advances in Reinforcement Learning (RL) have demonstrated promising results in autonomous car racing. However, two fundamental challenges remain: sparse rewards, which hinder efficient learning process, and the quality of demonstrations, which directly affects the effectiveness of RL from Demonstration (RLfD) approaches. To address these issues, we propose SAC(lambda), a novel RLfD algorithm tailored for sparse-reward racing tasks with imperfect demonstrations. SAC(lambda) introduces two key components: (1) a discriminator-augmented Q-function, which integrates prior knowledge from demonstrations into value estimation while maintaining off-policy learning benefits, and (2) a Positive-Unlabeled (PU) learning framework with adaptive prior adjustment, which enables the agent to progressively refine its understanding of positive behaviors, while mitigating the overfitting problem. Through extensive experiments in the Assetto Corsa simulator, we demonstrate that SAC(lambda) significantly accelerates training, surpasses the provided demonstrations, and achieves superior lap times over existing RL and RLfD approaches. Code and videos are available at https://heesungsung.github.io/AC-RLRacer/.

In this paper, we implement the GPU-accelerated subsystem-based Alternating Direction Method of Multipliers (SubADMM) for interactive simulation. The challenging objective for interactive simulations is to deliver realistic results under tight performance, even for large-scale scenarios. We aim to achieve this by exploiting the parallelizable nature of SubADMM to the fullest extent. We introduce a new subsystem division strategy to make SubADMM ‘GPU friendly' along with custom kernel designs and optimization regarding efficient memory access patterns. We successfully implement the GPUaccelerated SubADMM and show the accuracy and speed of the framework for large-scale scenarios, highlighted with an interactive ‘Hand demo’ scenario. We also show improved robustness and accuracy compared to other state-of-the-art interactive simulators with several challenging scenarios that introduce large-scale ill-conditioned dynamics problems.

This paper presents a study on optimal design to determine the installation position and link lengths of a robot within a designated workspace for welding, aiming to minimize singularities during the robot’s motion. Bi-objective functions are formulated to minimize singularities while maximizing the volumes of linear velocity manipulability ellipsoid and angular velocity manipulability ellipsoid, respectively, ensuring isotropy. We have constructed a simulation environment incorporating PID control to account for robot tracking errors. This environment was utilized as a simulator to derive a Bi-objective function set within a genetic algorithm. Through this, we optimized four robot link length variables and two installation position variables, selecting the optimal design variables on the Pareto Front. In the standard work object, the volume average of the linear velocity manipulability ellipsoid was confirmed to have improved by 72% compared to the initial level, and the isotropy of the angular velocity manipulability ellipsoid was confirmed to have improved by 23% compared to the initial level. Furthermore, correlation analysis between design parameters identified those with a high correlation with the objective functions, and the analysis results are discussed.