Imitation learning for manipulation has a well-known data scarcity problem. Unlike natural language and 2D computer vision, there is no Internet-scale corpus of data for dexterous manipulation. One appealing option is egocentric human video, a passively scalable data source. However, existing large-scale datasets such as Ego4D do not have native hand pose annotations and do not focus on object manipulation. To this end, we use Apple Vision Pro to collect EgoDex: the largest and most diverse dataset of dexterous human manipulation to date. EgoDex has 829 hours of egocentric video with paired 3D hand and finger tracking data collected at the time of recording, where multiple calibrated cameras and on-device SLAM can be used to precisely track the pose of every joint of each hand. The dataset covers a wide range of diverse manipulation behaviors with everyday household objects in 194 different tabletop tasks ranging from tying shoelaces to folding laundry. Furthermore, we train and systematically evaluate imitation learning policies for hand trajectory prediction on the dataset, introducing metrics and benchmarks for measuring progress in this increasingly important area. By releasing this large-scale dataset, we hope to push the frontier of robotics, computer vision, and foundation models. EgoDex is publicly available for download.

This paper introduces a framework, called EMOTION, for generating expressive motion sequences in humanoid robots, enhancing their ability to engage in humanlike non-verbal communication. Non-verbal cues such as facial expressions, gestures, and body movements play a crucial role in effective interpersonal interactions. Despite the advancements in robotic behaviors, existing methods often fall short in mimicking the diversity and subtlety of human non-verbal communication. To address this gap, our approach leverages the in-context learning capability of large language models (LLMs) to dynamically generate socially appropriate gesture motion sequences for human-robot interaction. We use this framework to generate 10 different expressive gestures and conduct online user studies comparing the naturalness and understandability of the motions generated by EMOTION and its human-feedback version, EMOTION++, against those by human operators. The results demonstrate that our approach either matches or surpasses human performance in generating understandable and natural robot motions under certain scenarios. We also provide design implications for future research to consider a set of variables when generating expressive robotic gestures.

Robotic arms have significant gaps in their sensing and perception, making it hard to perform motion planning in dense environments. To address this, we introduce ARMOR, a novel egocentric perception system that integrates both hardware and software, specifically incorporating wearable-like depth sensors for bimanual robotic platforms with arms. Our distributed perception approach enhances the robot's spatial awareness, and facilitates more agile motion planning. We also train a transformer-based imitation learning (IL) policy in simulation to perform dynamic collision avoidance, by leveraging around 86 hours worth of human realistic motions from the AMASS dataset. We show that our ARMOR perception is superior against a setup with multiple dense head-mounted, and externally mounted depth cameras, with a 63.7% reduction in collisions, and 78.7% improvement on success rate. We also compare our IL policy against a sampling-based motion planning expert cuRobo, showing 31.6% less collisions, 16.9% higher success rate, and 26x reduction in computational latency. Lastly, we deploy our ARMOR perception on our real-world GR1 humanoid robot from Fourier Intelligence. The simulation environment, HW description, and 3D CAD files are available at https://daehwakim.com/armor.

Effectively representing 3D scenes for Multimodal Large Language Models (MLLMs) is crucial yet challenging. Existing approaches commonly only rely on 2D image features and use varied tokenization approaches. This work presents a rigorous study of 3D token structures, systematically comparing video-based and point-based representations while maintaining consistent model backbones and parameters. We propose a novel approach that enriches visual tokens by incorporating 3D point cloud features from a Sonata pretrained Point Transformer V3 encoder. Our experiments demonstrate that merging explicit 3D features significantly boosts performance. Furthermore, we show that point-based token structures can rival video-based ones when the points are cleverly sampled and ordered. Our best models from both structures achieve state-of-the-art results on multiple 3D understanding benchmarks. We emphasize our analysis of token structures as a key contribution, alongside transparent reporting of results averaged over multiple seeds, a practice we believe is vital for robust progress in the field.

Training manipulation policies for humanoid robots with diverse data enhances their robustness and generalization across tasks and platforms. However, learning solely from robot demonstrations is labor-intensive, requiring expensive tele-operated data collection, which is difficult to scale. This paper investigates a more scalable data source, egocentric human demonstrations, to serve as cross-embodiment training data for robot learning. We mitigate the embodiment gap between humanoids and humans from both the data and modeling perspectives. We collect an egocentric task-oriented dataset (PH2D) that is directly aligned with humanoid manipulation demonstrations. We then train a human-humanoid behavior policy, which we term Human Action Transformer (HAT). The state-action space of HAT is unified for both humans and humanoid robots and can be differentiably retargeted to robot actions. Co-trained with smaller-scale robot data, HAT directly models humanoid robots and humans as different embodiments without additional supervision. We show that human data improve both generalization and robustness of HAT with significantly better data collection efficiency.

How can a robot safely navigate around people with complex motion patterns? Deep Reinforcement Learning (DRL) in simulation holds some promise, but much prior work relies on simulators that fail to capture the nuances of real human motion. Thus, we propose Deep Residual Model Predictive Control (DR-MPC) to enable robots to quickly and safely perform DRL from real-world crowd navigation data. By blending MPC with model-free DRL, DR-MPC overcomes the DRL challenges of large data requirements and unsafe initial behavior. DR-MPC is initialized with MPC-based path tracking, and gradually learns to interact more effectively with humans. To further accelerate learning, a safety component estimates out-of-distribution states to guide the robot away from likely collisions. In simulation, we show that DR-MPC substantially outperforms prior work, including traditional DRL and residual DRL models. Hardware experiments show our approach successfully enables a robot to navigate a variety of crowded situations with few errors using less than 4 hours of training data.

This paper presents the Embedding Pose Graph (EPG), an innovative method that combines the strengths of foundation models with a simple 3D representation suitable for robotics applications. Addressing the need for efficient spatial understanding in robotics, EPG provides a compact yet powerful approach by attaching foundation model features to the nodes of a pose graph. Unlike traditional methods that rely on bulky data formats like voxel grids or point clouds, EPG is lightweight and scalable. It facilitates a range of robotic tasks, including open-vocabulary querying, disambiguation, image-based querying, language-directed navigation, and re-localization in 3D environments. We showcase the effectiveness of EPG in handling these tasks, demonstrating its capacity to improve how robots interact with and navigate through complex spaces. Through both qualitative and quantitative assessments, we illustrate EPG's strong performance and its ability to outperform existing methods in re-localization. Our work introduces a crucial step forward in enabling robots to efficiently understand and operate within large-scale 3D spaces.

In the field of deep point cloud understanding, KP-Conv is a unique architecture that uses kernel points to locate convolutional weights in space, instead of relying on Multi-Layer Perceptron (MLP) encodings. While it initially achieved success, it has since been surpassed by recent MLP networks that employ updated designs and training strategies. Building upon the kernel point principle, we present two novel designs: KPConvD (depthwise KP-Conv), a lighter design that enables the use of deeper architectures, and KPConvX, an innovative design that scales the depthwise convolutional weights of KPConvD with kernel attention values. Using KPConvX with a modern architecture and training strategy, we are able to outperform current state-of-the-art approaches on the ScanObjectNN, Scannetv2, and S3DIS datasets. We validate our design choices through ablation studies and release our code and models.

In this paper, we introduce a LiDAR-based robot navigation system, based on novel object-aware affordance-based costmaps. Utilizing a 3D object detection network, our system identifies objects of interest in LiDAR keyframes, refines their 3D poses with the Iterative Closest Point (ICP) algorithm, and tracks them via Kalman filters and the Hungarian algorithm for data association. It then updates existing object poses with new associated detections and creates new object maps for unmatched detections. Using the maintained object-level mapping system, our system creates affordance-driven object costmaps for proactive collision avoidance in path planning. Additionally, we address the scarcity of indoor semantic LiDAR data by introducing an automated labeling technique. This method utilizes a CAD model database for accurate ground-truth annotations, encompassing bounding boxes, positions, orientations, and point-wise semantics of each object in LiDAR sequences. Our extensive evaluations, conducted in both simulated and real-world robot platforms, highlights the effectiveness of proactive object avoidance by using object affordance costmaps, enhancing robotic navigation safety and efficiency. The system can operate in real-time onboard and we intend to release our code and data for public use.

Step-by-step decision planning with large language models (LLMs) is gaining attention in AI agent development. This paper focuses on decision planning with uncertainty estimation to address the hallucination problem in language models. Existing approaches are either white-box or computationally demanding, limiting use of black-box proprietary LLMs within budgets. The paper's first contribution is a non-parametric uncertainty quantification method for LLMs, efficiently estimating point-wise dependencies between input-decision on the fly with a single inference, without access to token logits. This estimator informs the statistical interpretation of decision trustworthiness. The second contribution outlines a systematic design for a decision-making agent, generating actions like “turn on the bathroom light” based on user prompts such as “take a bath”. Users will be asked to provide preferences when more than one action has high estimated point-wise dependencies. In conclusion, our uncertainty estimation and decision-making agent design offer a cost-efficient approach for AI agent development.

AbstractLaminectomy is one of the most common posterior spinal operations. Since the lamina is adjacent to important tissues such as nerves, once damaged, it can cause serious complications and even lead to paralysis. In order to prevent the above injuries and complications, ultrasonic bone scalpel and surgical robots have been introduced into spinal laminectomy, and many scholars have studied the recognition method of the bone tissue status. Currently, almost all methods to achieve recognition of bone tissue are based on sensor signals collected by high‐precision sensors installed at the end of surgical robots. However, the previous methods could not accurately identify the state of spinal bone tissue. Innovatively, the identification of bone tissue status was regarded as a time series classification task, and the classification algorithm LSTM‐FCN was used to process fusion signals composed of force and cutting depth signals, thus achieving an accurate classification of the lamina bone tissue status. In addition, it was verified that the accuracy of the proposed method could reach 98.85% in identifying the state of porcine spinal laminectomy. And the maximum penetration distance can be controlled within 0.6 mm, which is safe and can be used in practice.

We present a method for generating, predicting, and using spatiotemporal occupancy grid maps (SOGM), which embed future semantic information of real dynamic scenes. We present an autolabeling process that creates SOGMs from noisy real navigation data. We use a 3-D–2-D feedforward architecture, trained to predict the future time steps of SOGMs, given 3-D Lidar frames as input. Our pipeline is entirely self-supervised, thus enabling lifelong learning for real robots. The network is composed of a 3-D back-end that extracts rich features and enables the semantic segmentation of the lidar frames, and a 2-D front-end that predicts the future information embedded in the SOGM representation, potentially capturing the complexities and uncertainties of real-world multiagent interactions. We also design a navigation system that uses these predicted SOGMs within planning, after they have been transformed into spatiotemporal risk maps. We verify our navigation system's abilities in simulation, validate it on a real robot, study SOGM predictions on real data in various circumstances, and provide a novel indoor 3-D lidar dataset, collected during our experiments, which includes our automated annotations.

Percutaneous transforaminal endoscopic discectomy (PTED) is a decompression surgery on patients with lumbar disc herniation and spinal canal stenosis in a minimally invasive environment, which greatly shorten the rehabilitation cycle of patients. However, the puncture of traditional PTED is performed under non-direct vision, which relies on the surgeon’s clinical experience heavily and can easily cause collateral damage. In this paper, we propose a robotic positioning method in PTED based on X-ray image and DLT algorithm. The end-effector with three rings (EETR) for PTED positioning has been specially designed. During the operation, the 3D-2D transformation matrix is calculated by direct linear transform (DLT) algorithm. Finally, the motion parameters are calculated to control the robot. Based on our method, only one X-ray image is needed to complete the positioning, which also works well in a large range of deflection angle between EETR and target puncture channel. Thus, it greatly simplifies the surgical process and reduces the radiation exposure time. By conducting a series of comparative experiments on the positioning of model bones, the translation error is less than l.lmm and rotation error is less than 0.8°, when the deflection angle is less than 30°. And the translation error is 1.61mm and rotation error is 1.98°, when the angle reached 60°. It meets the requirements for PTED positioning, and has improved the positioning accuracy.

Recent efforts to enable visual navigation using large language models have mainly focused on developing complex prompt systems. These systems incorporate instructions, observations, and history into massive text prompts, which are then combined with pre-trained large language models to facilitate visual navigation. In contrast, our approach aims to fine-tune large language models for visual navigation without extensive prompt engineering. Our design involves a simple text prompt, current observations, and a history collector model that gathers information from previous observations as input. For output, our design provides a probability distribution of possible actions that the agent can take during navigation. We train our model using human demonstrations and collision signals from the Habitat-Matterport 3D Dataset (HM3D). Experimental results demonstrate that our method outperforms state-of-the-art behavior cloning methods and effectively reduces collision rates.

A household robot should be able to navigate to target objects without requiring users to first annotate everything in their home. Most current approaches to object navigation do not test on real robots and rely solely on reconstructed scans of houses and their expensively labeled semantic 3D meshes. In this work, our goal is to build an agent that builds self-supervised models of the world via exploration, the same as a child might - thus we (1) eschew the expense of labeled 3D mesh and (2) enable self-supervised in-situ finetuning in the real world. We identify a strong source of self-supervision (Location Consistency - LocCon) that can train all components of an ObjectNav agent, using unannotated simulated houses. Our key insight is that embodied agents can leverage location consistency as a self-supervision signal - collecting images from different views/angles and applying contrastive learning. We show that our agent can perform competitively in the real world and simulation. Our results also indicate that supervised training with 3D mesh annotations causes models to learn simulation artifacts, which are not transferrable to the real world. In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.

High-precision image segmentation of the spine in computed tomography (CT) images is important for the diagnosis of spinal diseases and surgical path planning. Manual segmentation is often tedious and time consuming. Thus, an automatic segmentation algorithm is expected to solve this problem. However, because different areas are scanned, the number of spines in the original CT image and the coverage area are often different, making it extremely difficult to directly conduct a fully autonomous spine segmentation. In this study, we propose a two-stage automatic spine segmentation method based on 3D Swin Transformer. In the first stage, the 3D Swin-YoloX algorithm is used to achieve an accurate positioning of each spine segment in the CT images. In the second stage, 3D Swin-UNet is used to achieve a high-precision segmentation of the spine. Using an open dataset, the average Dice of our approach can reach 0.942 and the average Hausdorff distance can reach 6.24, indicating a higher accuracy in comparison with other published methods. Our proposed method can effectively eliminate any adverse effects of the different scanning areas on a spinal image segmentation and has a high application value.

Human following is a crucial feature of human-robot interaction, yet it poses numerous challenges to mobile agents in real-world scenarios. Some major hurdles are that the target person may be in a crowd, obstructed by others, or facing away from the agent. To tackle these challenges, we present a novel person re-identification module composed of three parts: a 360-degree visual registration, a neural-based person re-identification using human faces and torsos, and a motion tracker that records and predicts the target person's future position. Our human-following system also addresses other challenges, including identifying fast-moving targets with low latency, searching for targets that move out of the camera's sight, collision avoidance, and adaptively choosing different following mechanisms based on the distance between the target person and the mobile agent. Extensive experiments show that our proposed person re-identification module significantly enhances the human-following feature compared to other baseline variants.

Collision avoidance is key for mobile robots and agents to operate safely in the real world. In this work we present SAFER, an efficient and effective collision avoidance system that is able to improve safety by correcting the control commands sent by an operator. It combines real-world reinforcement learning (RL), search-based online trajectory planning, and automatic emergency intervention, e.g. automatic emergency braking (AEB). The goal of the RL is to learn an effective corrective control action that is used in a focused search for collision-free trajectories, and to reduce the frequency of triggering automatic emergency braking. This novel setup enables the RL policy to learn safely and directly on mobile robots in a real-world indoor environment, minimizing actual crashes even during training. Our real-world experiments show that, when compared with several baselines, our approach enjoys a higher average speed, lower crash rate, less emergency intervention, smaller computation overhead, and smoother overall control.

AbstractTo eliminate unnecessary background information, such as soft tissues in original CT images and the adverse impact of the similarity of adjacent spines on lumbar image segmentation and surgical path planning, a two‐stage approach for localising lumbar segments is proposed. First, based on the multi‐scale feature fusion technology, a non‐linear regression method is used to achieve accurate localisation of the overall spatial region of the lumbar spine, effectively eliminating useless background information, such as soft tissues. In the second stage, we directly realised the precise positioning of each segment in the lumbar spine space region based on the non‐linear regression method, thus effectively eliminating the interference caused by the adjacent spine. The 3D Intersection over Union (3D_IOU) is used as the main evaluation indicator for the positioning accuracy. On an open dataset, 3D_IOU values of 0.8339 ± 0.0990 and 0.8559 ± 0.0332 in the first and second stages, respectively is achieved. In addition, the average time required for the proposed method in the two stages is 0.3274 and 0.2105 s respectively. Therefore, the proposed method performs very well in terms of both precision and speed and can effectively improve the accuracy of lumbar image segmentation and the effect of surgical path planning.

Spinal surgery robots have a great application value in laminar decompression surgery. For a safe surgery, the robot needs to accurately identify the cutting state of the lamina. Therefore, it is very important to deal with various sensing signals in the form of a time series. However, various state recognition algorithms proposed so far cannot completely avoid cutting through the lamina, leaving hidden dangers for nerve thermal damage caused by high-temperature liquid splashing. We propose a long time series prediction algorithm called STP-Net, which combined with the existing algorithms for recognizing the lamina cutting state can stop the lamina cutting in advance, thus preserving a thin inner layer of cortical bone and blocking high-temperature liquid splashing. STP-Net has the following advantages: (1) Dimension reduction is first performed on Swin (shifted window) Transformer to obtain 1D Swin Transformer, which was used to develop STP-Net for long time series prediction, which not only has high accuracy but also shows a linear relationship between computational complexity and input sequence length L , namely, O ( L ). (2) The token merging layer is proposed and applied to the encoder of STP-Net, which reduces the computation cost and improves the global information extraction capability of the algorithm. (3) STP-Net uses a generative decoder to output the prediction sequence directly through a single operation, which not only has high efficiency but also avoids the accumulation of errors. Taking force signals as an example, when 400 numbers are used to predict 100 numbers, the average mean square error (MSE) and mean absolute error (MAE) of standard STP-Net are 1.22 × 10 −3 and 2.42 × 10 −2 , respectively. STP-Net, combined with the existing method for recognizing the lamina cutting status, can stop the robot from cutting in advance in 87.14% of the cases. In addition, the results of practical lamina cutting experiments confirmed the effectiveness of STP-Net.