We introduce Habilis-β, a fast-motion and long-lasting on-device vision-language-action (VLA) model designed for real-world deployment. Current VLA evaluation remains largely confined to single-trial success rates under curated resets, which fails to capture the fast-motion and long-lasting capabilities essential for practical operation. To address this, we introduce the Productivity-Reliability Plane (PRP), which evaluates performance through Tasks per Hour (TPH) and Mean Time Between Intervention (MTBI) under a continuous-run protocol that demands both high-speed execution and sustained robustness. Habilis-β achieves high performance by integrating language-free pre-training on large-scale play data for robust interaction priors with post-training on cyclic task demonstrations that capture state drift across consecutive task iterations. The system further employs ESPADA for phase-adaptive motion shaping to accelerate free-space transit, utilizes rectified-flow distillation to enable high-frequency control on edge devices, and incorporates classifier-free guidance (CFG) as a deployment-time knob to dynamically balance instruction adherence and learned interaction priors. In 1-hour continuous-run evaluations, Habilis-β achieves strong performance under the PRP metrics, compared to π_0.5 in both simulation and real-world environments. In simulation, Habilis-β achieves 572.6 TPH and 39.2 s MTBI (vs. 120.5 TPH and 30.5 s for π_0.5), while in a real-world humanoid logistics workflow it achieves 124 TPH and 137.4 s MTBI (vs. 19 TPH and 46.1 s for π_0.5). Finally, Habilis-β achieves the highest reported performance on the standard RoboTwin 2.0 leaderboard across representative tasks, validating its effectiveness in complex manipulation scenarios.

Applying multi-agent reinforcement learning (MARL) to real-world scenarios is challenging because agents often need to adapt quickly to unexpected situations, including those rarely or never encountered in training. Recent methods for out-of-distribution generalization are unsuitable for applications on out-of-distribution tasks with limited communication, because they are typically restricted to centralized training or some specialized instances of distribution shifts. To address this limitation, we introduce the Unexpectedness Encoding Scheme, a new decentralized MARL algorithm in which agents communicate ‘‘unexpectedness,’’ the surprising aspects of the environment. In addition to sending their usual reward-driven messages, each agent predicts the next observation based on past experience and then compares this prediction with the actual outcome. The discrepancy between the two is encoded as a message, enabling agents to adapt more effectively to sudden or extreme changes. Experimental results on multi-agent cooperative tasks demonstrate that our method adapts robustly to both dynamically changing training environments and previously unseen out-of-distribution scenarios.

In this work, we propose Efficient Collision-Aware Hierarchical Diffusion Navigation (ECAHD), a hierarchical diffusion-based framework designed for both safety and computational efficiency. ECAHD generates a sparse trajectory for global path planning and a dense trajectory for local path refinement. The robot follows a rapidly sampled sparse global trajectory, and when a potential collision is detected, a collision-aware guidance diffusion mechanism—which accounts for the robot’s shape—adjusts the local trajectory accordingly. Conventional full-sequence diffusion planners suffer from slow sampling speeds and performance degradation when collision-aware guidance is applied across the entire trajectory. ECAHD addresses these issues by significantly reducing the number of waypoints predicted by the global diffusion planner, while delegating robot shape aware collision guidance to the local diffusion planner. This separation not only accelerates planning but also preserves global trajectory quality, as goal-conditioned sampling is no longer disrupted by collision-related constraints. Furthermore, ECAHD allows for increasing the number of global trajectory samples to enhance performance, without incurring substantial computational overhead. In maze2d-large planning tests, ECAHD improved success rates by approximately 1.3% while reducing collision rates by more than 50%, all while cutting inference time by nearly half.

Articulation modeling enables robots to learn joint parameters of articulated objects for effective manipulation which can then be used downstream for skill learning or planning. Existing approaches often rely on prior knowledge about the objects, such as the number or type of joints. Some of these approaches also fail to recover occluded joints that are only revealed during interaction. Others require large numbers of multi-view images for every object, which is impractical in real-world settings. Furthermore, prior works neglect the order of manipulations, which is essential for many multi-DoF objects where one joint must be operated before another, such as a dishwasher. We introduce PokeNet, an end-to-end framework that estimates articulation models from a single human demonstration without prior object knowledge. Given a sequence of point cloud observations of a human manipulating an unknown object, PokeNet predicts joint parameters, infers manipulation order, and tracks joint states over time. PokeNet outperforms existing state-of-the art methods, improving joint axis and state estimation accuracy by an average of over 27% across diverse objects, including novel and unseen categories. We demonstrate these gains in both simulation and real-world environments.

Behavior-cloning based visuomotor policies enable precise manipulation but often inherit the slow, cautious tempo of human demonstrations, limiting practical deployment. However, prior studies on acceleration methods mainly rely on statistical or heuristic cues that ignore task semantics and can fail across diverse manipulation settings. We present ESPADA, a semantic and spatially aware framework that segments demonstrations using a VLM-LLM pipeline with 3D gripper-object relations, enabling aggressive downsampling only in non-critical segments while preserving precision-critical phases, without requiring extra data or architectural modifications, or any form of retraining. To scale from a single annotated episode to the full dataset, ESPADA propagates segment labels via Dynamic Time Warping (DTW) on dynamics-only features. Across both simulation and real-world experiments with ACT and DP baselines, ESPADA achieves approximately a 2x speed-up while maintaining success rates, narrowing the gap between human demonstrations and efficient robot control.

This paper explores how non-experts can teach robots desired skills in their environments. We argue that natural language is an intuitive and accessible interface for robot learning. To this end, we investigate two key aspects: (1) how non-experts collect robotic data using natural language supervision and (2) how pre-trained vision-language models learn end-to-end policies directly from this supervision. We propose a data collection framework that collects robot demonstrations based on natural language supervision (e.g., "move forward") and further augments these demonstrations. Next, we introduce a model that learns language-conditioned policies from natural language supervision called CLIP-RT. Our model employs pre-trained CLIP models and learns to predict actions represented in language via contrastive imitation learning. We first train CLIP-RT on large-scale robotic data and then enable it to learn desired skills using data collected from our framework. CLIP-RT shows strong capabilities in acquiring novel manipulation skills, outperforming the state-of-the-art model, OpenVLA (7B parameters), by 17 parameters (1B).

As robots become more generalized and deployed in diverse environments, they must interact with complex objects, many with multiple independent joints or degrees of freedom (DoF) requiring precise control. A common strategy is object modeling, where compact state-space models are learned from real-world observations and paired with classical planning. However, existing methods often rely on prior knowledge or focus on single-DoF objects, limiting their applicability. They also fail to handle occluded joints and ignore the manipulation sequences needed to access them. We address this by learning object models from human demonstrations. We introduce Object Kinematic Sequence Machines (OKSMs), a novel representation capturing both kinematic constraints and manipulation order for multi-DoF objects. To estimate these models from point cloud data, we present Pokenet, a deep neural network trained on human demonstrations. We validate our approach on 8,000 simulated and 1,600 real-world annotated samples. Pokenet improves joint axis and state estimation by over 20 percent on real-world data compared to prior methods. Finally, we demonstrate OKSMs on a Sawyer robot using inverse kinematics-based planning to manipulate multi-DoF objects.

In this paper, we propose a novel method for plane clustering specialized in cluttered scenes using an RGB-D camera and validate its effectiveness through robot grasping experiments. Unlike existing methods, which focus on large-scale indoor structures, our approach -- Multi-Object RANSAC emphasizes cluttered environments that contain a wide range of objects with different scales. It enhances plane segmentation by generating subplanes in Deep Plane Clustering (DPC) module, which are then merged with the final planes by post-processing. DPC rearranges the point cloud by voting layers to make subplane clusters, trained in a self-supervised manner using pseudo-labels generated from RANSAC. Multi-Object RANSAC demonstrates superior plane instance segmentation performances over other recent RANSAC applications. We conducted an experiment on robot suction-based grasping, comparing our method with vision-based grasping network and RANSAC applications. The results from this real-world scenario showed its remarkable performance surpassing the baseline methods, highlighting its potential for advanced scene understanding and manipulation.

Current bin picking methods that rely heavily on end-to-end learning often falter when confronted with unfamiliar or complex objects in unstructured environments. To overcome these limitations, we introduce Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios. Seg2Grasp is built on a three-step process: Segmentation, Grasping, and Classification. The Segmentation module employs a Transformer-based model to generate class-agnostic object masks from RGB-D images, ensuring accurate detection across various conditions. The Grasping module uses surface normals and mask proposals to determine the optimal suction points, enhancing grasp success. Finally, the Classification module leverages fine-tuned open-vocabulary Mask-CLIP for precise object identification, enabling versatile handling of diverse objects. Real-world robotic experiments demonstrate that Seg2Grasp outperforms existing methods in success rates and adaptability, establishing it as a powerful tool for automated bin picking in industrial settings.

Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation learning can be supported.

This paper addresses the problem of training a robot to carry out temporal tasks of arbitrary complexity via evaluative human feedback that can be inaccurate. A key idea explored in our work is a kind of curriculum learning---training the robot to master simple tasks and then building up to more complex tasks. We show how a training procedure, using knowledge of the formal task representation, can decompose and train any task efficiently in the size of its representation. We further provide a set of experiments that support the claim that non-expert human trainers can decompose tasks in a way that is consistent with our theoretical results, with more than half of participants successfully training all of our experimental missions. We compared our algorithm with existing approaches and our experimental results suggest that our method outperforms alternatives, especially when feedback contains mistakes.

To widen their accessibility and increase their utility, intelligent agents must be able to learn complex behaviors as specified by (non-expert) human users. Moreover, they will need to learn these behaviors within a reasonable amount of time while efficiently leveraging the sparse feedback a human trainer is capable of providing. Recent work has shown that human feedback can be characterized as a critique of an agent's current behavior rather than as an alternative reward signal to be maximized, culminating in the COnvergent Actor-Critic by Humans (COACH) algorithm for making direct policy updates based on human feedback. Our work builds on COACH, moving to a setting where the agent's policy is represented by a deep neural network. We employ a series of modifications on top of the original COACH algorithm that are critical for successfully learning behaviors from high-dimensional observations, while also satisfying the constraint of obtaining reduced sample complexity. We demonstrate the effectiveness of our Deep COACH algorithm in the rich 3D world of Minecraft with an agent that learns to complete tasks by mapping from raw pixels to actions using only real-time human feedback in 10-15 minutes of interaction.

Mutually beneficial behavior in repeated games can be enforced via the threat of punishment, as enshrined in game theory's well-known "folk theorem." There is a cost, however, to a player for generating these disincentives. In this work, we seek to minimize this cost by computing a "Stackelberg punishment," in which the player selects a behavior that sufficiently punishes the other player while maximizing its own score under the assumption that the other player will adopt a best response. This idea generalizes the concept of a Stackelberg equilibrium. Known efficient algorithms for computing a Stackelberg equilibrium can be adapted to efficiently produce a Stackelberg punishment. We demonstrate an application of this idea in an experiment involving a virtual autonomous vehicle and human participants. We find that a self-driving car with a Stackelberg punishment policy discourages human drivers from bullying in a driving scenario requiring social negotiation.

An agent with an inaccurate model of its environment faces a difficult choice: it can ignore the errors in its model and act in the real world in whatever way it determines is optimal with respect to its model. Alternatively, it can take a more conservative stance and eschew its model in favor of optimizing its behavior solely via real-world interaction. This latter approach can be exceedingly slow to learn from experience, while the former can lead to "planner overfitting" - aspects of the agent's behavior are optimized to exploit errors in its model. This paper explores an intermediate position in which the planner seeks to avoid overfitting through a kind of regularization of the plans it considers. We present three different approaches that demonstrably mitigate planner overfitting in reinforcement-learning environments.

Observations of trained deep reinforcement-learning agents give the impression that agents have constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions and techniques based on an agent’s performance in on-policy, off-policy, and unreachable states. We demonstrate these techniques on a common benchmark task for deep RL, and show that more experimentation and analysis is necessary before claims of generalization can be supported.

As robot algorithms for manipulation and navigation advance and robot hardware is becoming more robust and readily available, industry demands robots to perform more sophisticated tasks in our homes and factories. For many years, direct teleoperation was the most common and traditional form of control for robots. However, due to the complexity of robot motion, human operators must focus most of their attention on solving low-level motion control which leads to their heightened cognitive load. In this abstract, we propose a goal-directed approach to programming robots by providing a tool to model the world and provide goal states for a given task. Operators will be able to set the initial positions of objects and their affordances along with their goal positions by imposing three dimensional (3D) templates on point clouds. Robots will solve the given task using the combination of task and motion planning algorithms.

This study explores how people's social response toward a humanoid robot can change when we vary the number of the active degrees of freedom in the robot's head and face area. We investigate this problem by conducting two wizard-of-oz user studies that situate an elder person in a self-disclosure dialogue with a remotely operated robot. In our first study, we investigated the effect of expressive head gestures with a four-degree-of-freedom neck. In the second study we focused on the face where we investigated the effect of expressive eyebrow movement versus active gaze and eyelid movement. In the first study, we found that participants are willing to disclose more to the robot when the robot moved its neck in an expressive manner. In the second study, our data suggests a trend where gaze and expressive eyelid movement results in more disclosure over eyebrow movement

Robots as an embodied, multi-modal technology have great potential to be used as a new type of communication device. In this paper we outline our development of the Huggable robot as a semi-autonomous robot avatar for two specific types of remote interaction — family communication and education. We also describe three different operator control interfaces (Web Interface, Wearable Interface and Sympathetic Interface) being developed to explore how these systems will impact the experience. Furthermore, through our discussion we highlight how we have applied five important elements in our system to allow the robot to function as a richly embodied communication channel. These five elements include sharing and directing attention, situational awareness through real-time sensor feedback, alleviating the cognitive load of a user, conveying the personality and character of the robot and global accessibility. Lastly, we provide results from a pilot study of the Web Interface.

Robotic companions offer a unique combination of embodiment and computation which open many new interesting opportunities in the field of pediatric care. As these new technologies are developed, we must consider the central research questions of how such systems should be designed and what the appropriate applications for such systems are. In this paper we present the Huggable, a robotic companion in the form factor of a teddy bear and outline a series of studies we are planning to run using the Huggable in a pediatric care unit.