Robot learning from humans has been proposed and researched for several decades as a means to enable robots to learn new skills or adapt existing ones to new situations. Recent advances in AI, including learning approaches like reinforcement learning and architectures like transformers and foundation models, combined with access to massive datasets, have created attractive opportunities to apply those data-hungry techniques to this problem. We argue that the focus on massive amounts of pre-collected data, and the resulting learning paradigm, where humans demonstrate and robots learn in isolation, is overshadowing a specialized area of work we term Human-Interactive Robot Learning (HIRL). This paradigm, wherein robots and humans interact during the learning process, is at the intersection of multiple fields (AI, robotics, human-computer interaction, design and others) and holds unique promise. Using HIRL, robots can achieve greater sample efficiency (as humans can provide task knowledge through interaction), align with human preferences (as humans can guide the robot behavior toward their expectations), and explore more meaningfully and safely (as humans can utilize domain knowledge to guide learning and prevent catastrophic failures). This can result in robotic systems that can more quickly and easily adapt to new tasks in human environments. The objective of this article is to provide a broad and consistent overview of HIRL research and to guide researchers toward understanding the scope of HIRL, and current open or underexplored challenges related to four themes-namely, human, robot learning, interaction, and broader context. The article includes concrete use cases to illustrate the interaction between these challenges and inspire further research according to broad recommendations and a call for action for the growing HIRL community.

This full-day workshop addresses the problems of accessibility in HRI and the interplay of ethical considerations for disability-centered design and research, accessibility concerns for disabled researchers, and the design of assistive HRI technologies. We invite authors to submit extended abstracts (up to 2 pages, excluding references) and short papers (up to 4 pages, excluding references) on a range of topics relevant to ethics, accessibility, and assistive applications in HRI, including critical reflections on methodologies, design papers on human-centered or anti-ableist assistive technology, and papers from those outside the HRI community who may have insight to share on these concerns. The workshop will use a hybrid format to allow participants who due to disability, geographic, financial, or other constraints, are unable to travel, and will feature keynote speakers, panel discussions, and breakout sessions.

Despite the growth of physically assistive robotics (PAR) research over the last decade, nearly half of PAR user studies do not involve participants with the target disabilities. There are several reasons for this -- recruitment challenges, small sample sizes, and transportation logistics -- all influenced by systemic barriers that people with disabilities face. However, it is well-established that working with end-users results in technology that better addresses their needs and integrates with their lived circumstances. In this paper, we reflect on multiple approaches we have taken to working with people with motor impairments across the design, development, and evaluation of three PAR projects: (a) assistive feeding with a robot arm; (b) assistive teleoperation with a mobile manipulator; and (c) shared control with a robot arm. We discuss these approaches to working with users along three dimensions -- individual- vs. community-level insight, logistic burden on end-users vs. researchers, and benefit to researchers vs. community -- and share recommendations for how other PAR researchers can incorporate users into their work.

We demonstrate a robot-assisted feeding system that enables people with mobility impairments to feed themselves. Our system design embodies Safety, Portability, and User Control, with comprehensive full-stack safety checks, the ability to be mounted on and powered by any powered wheelchair, and a custom web-app allowing care-recipients to leverage their own assistive devices for robot control. For bite acquisition, we leverage multi-modal online learning to tractably adapt to unseen food types. For bite transfer, we leverage real-time mouth perception and interaction-aware control. Co-designed with community researchers, our system has been validated through multiple end-user studies.

Food acquisition with common general-purpose utensils is a necessary component of robot applications like in-home assistive feeding. Learning acquisition policies in this space is difficult in part because any model will need to contend with extensive state and actions spaces. Food is extremely diverse and generally difficult to simulate, and acquisition actions like skewers, scoops, wiggles, and twirls can be parameterized in myriad ways. However, food's visual diversity can belie a degree of physical homogeneity, and many foods allow flexibility in how they are acquired. Due to these facts, our key insight is that a small subset of actions is sufficient to acquire a wide variety of food items. In this work, we present a methodology for identifying such a subset from limited human trajectory data. We first develop an over-parameterized action space of robot acquisition trajectories that capture the variety of human food acquisition technique. By mapping human trajectories into this space and clustering, we construct a discrete set of 11 actions. We demonstrate that this set is capable of acquiring a variety of food items with $\geq80\%$ success rate, a rate that users have said is sufficient for in-home robot-assisted feeding. Furthermore, since this set is so small, we also show that we can use online learning to determine a sufficiently optimal action for a previously-unseen food item over the course of a single meal.

Robots can incorporate data from human teachers when learning new tasks. However, this data can often be noisy, which can cause robots to learn slowly or not at all. One method for learning from human teachers is Human-in-the-loop Reinforcement Learning (HRL), which can combine information from both an environmental reward and external feedback from human teachers. However, many HRL methods assume near-perfect information from teachers or must know the skill level of each teacher before starting the learning process. Our algorithm, Classification for Learning Erroneous Assessments using Rewards (CLEAR), is a feedback filter for Reinforcement Learning (RL) algorithms, enabling learning agents to learn from imperfect teachers without prior modeling. CLEAR is able to determine whether human feedback is correct based on observations of the RL learning curve. Our results suggest that CLEAR improves the quality of human feedback - from 57.5% to 65% correct in a human study - and performs more reliably than baselines by matching or outperforming RL without human teachers in all tested cases.

With robots poised to enter our daily environments, they will not only need to work for people, but also learn from them. An active area of investigation in the robotics, machine learning, and human-robot interaction communities is the design of teachable robots that can learn interactively from human input. To refer to these research efforts, we use the umbrella term Human-Interactive Robot Learning (HIRL). While algorithmic solutions for robots learning from people have been investigated in a variety of ways, HIRL, as a fairly new research area, is still lacking: 1) a formal set of definitions to classify related but distinct research problems or solutions, 2) benchmark tasks, interactions, and metrics to evaluate the performance of HIRL algorithms and interactions, and 3) clear long-term research challenges to be addressed by different communities. Last year we began consolidating the needed definitions and vocabulary to enable fruitful discussions between researchers from these interdisciplinary fields, and identified a preliminary list of long, medium, and short-term research problems for the community to tackle, and existing tools and frameworks that can be leveraged to this end. This workshop will build upon these discussions, focusing on promoting the specification and design of HIRL benchmarks.

With robots poised to enter our daily environments, we conjecture that they will not only need to work for people, but also learn from them. An active area of investigation in the robotics, machine learning, and human-robot interaction communities is the design of teachable robotic agents that can learn interactively from human input. To refer to these research efforts, we use the umbrella term Human-Interactive Robot Learning (HIRL). While algorithmic solutions for robots learning from people have been investigated in a variety of ways, HIRL, as a fairly new research area, is still lacking: 1) a formal set of definitions to classify related but distinct research problems or solutions, 2) benchmark tasks, interactions, and metrics to evaluate the performance of HIRL algorithms and interactions, and 3) clear long-term research challenges to be addressed by different communities. The main goal of this workshop will be to consolidate relevant recent work falling under the HIRL umbrella into a coherent set of long, medium, and short-term research problems, and identify the most pressing future research goals in this area. As HIRL is a developing research area, this workshop is an opportunity to break the existing boundaries between relevant research communities by developing and sharing a diverse set of benchmark tasks and metrics for HIRL, inspired by other fields including neuroscience, biology, and ethics research.

Policy Shaping is a Human-in-the-loop Reinforcement Learning (HRL) algorithm. We extend this work to continuous states with our algorithm, Deep Policy Shaping (DPS). DPS uses a feedback neural network that learns the optimality of actions from noisy feedback combined with an RL algorithm. In simulation, we find that DPS outperforms or matches baselines averaged over multiple hyperparameter settings and varying feedback correctness.

Robots can use information from people to improve learning speed or quality. However, people can have short attention spans and misunderstand tasks. Our work addresses these issues with algorithms for learning from inattentive teachers that take advantage of feedback when people are present, and an algorithm for learning from inaccurate teachers that estimates which state-action pairs receive incorrect feedback. These advances will enhance robots' ability to take advantage of imperfect feedback from human teachers.

For robots to be perceived as full-fledged team members, they must display intelligent behavior along multiple dimensions. One challenge is that even when the robot and human are on the same team, the interaction may not feel like teamwork to the human. We present a novel algorithm, Teammate Algorithm for Shared Cooperation (TASC). TASC is motivated by the concept of shared cooperative activity (SCA) for human-human teamwork, developed in prior work by Bratman. We focus on enabling the robot to prioritize certain SCA facets in its action selection depending on the task. We evaluated TASC in three experiments using different tasks with human users on Amazon Mechanical Turk. Our results show that TASC enabled participants to predict the robot’s goal earlier by one robot move and with greater confidence. The robot also helped reduce participants’ energy usage in a simulated block-moving task. Altogether, these results show that considering the SCA facets in the robot’s action selection improves teamwork.

Interactive Reinforcement Learning (RL) enables agents to learn from two sources: rewards taken from observations of the environment, and feedback or advice from a secondary critic source, such as human teachers or sensor feedback. The addition of information from a critic during the learning process allows the agents to learn more quickly than non-interactive RL. There are many methods that allow policy feedback or advice to be combined with RL. However, critics can often give imperfect information. In this work, we introduce a framework for characterizing Interactive RL methods with imperfect teachers and propose an algorithm, Revision Estimation from Partially Incorrect Resources (REPaIR), which can estimate corrections to imperfect feedback over time. We run experiments both in simulations and demonstrate performance on a physical robot, and find that when baseline algorithms do not have prior information on the exact quality of a feedback source, using REPaIR matches or improves the expected performance of these algorithms.

We present the Active Attention-Modified Policy Shaping (Active AMPS) algorithm, which allows learning robots to request feedback from multi-tasking human teachers. Active AMPS uses Reinforcement Learning supplemented with feedback from teachers, while avoiding frequently interrupting the teacher. This algorithm does so by selectively asking for attention from teachers in low-information areas of the state space when there is uncertainty about the teacher's feedback. Active AMPS allows people to take breaks from teaching the robot to complete other tasks, and is forgiving to lapses in human attention if learning occurs over long periods of time. We test Active AMPS both in simulation and on a physical robot in a human study. In simulation, we find that Active AMPS outperforms Attention-Modified Policy Shaping (AMPS), achieving an 11.0% increase in area under its learning curve while receiving 89.9% less feedback. In the human study, we find statistically significant results showing that Active AMPS allows people to complete 77.5% more work than AMPS while the robot receives 48.5% less feedback, without decreasing performance.

Robots deployed for long periods of time need to be able to explore and learn from their environment. One approach to this problem has been reinforcement learning (RL), in which robots receive rewards from the environment that allow them to choose optimal actions. To speed learning when human supervision is available, interactive reinforcement learning solicits feedback from a human teacher. However, this approach typically assumes that learning takes place under continuous supervision, which is unlikely to hold in long-term scenarios. We propose an extension to a method of interactive reinforcement learning, policy shaping, that takes into account human attention. Our approach enables better performance while unattended by favoring information-gathering actions when attended and actions that have received positive feedback when unattended. We test our approach in both simulation and on a robot, finding that our method learns faster than policy shaping and performs more safely than policy shaping while no one is paying attention to the robot.

Autonomous robots deployed around humans must be able to ask for help when problems arise. However, people may have incorrect mental models of the robots» capabilities or task, making them unable to help. We propose a data-driven method to estimate humans» beliefs after hearing task-related utterances and build sets of utterances that influence people towards useful help in expectation. We present an example to show our method selects effective utterances when the desired help is much different than a person expects.

In exploring representative databases, a primary issue has been finding accurate models of user preferences. Given this, our work generalizes the method of regret minimization as proposed by Nanongkai et al. to include nonlinear utility functions. Regret minimization is an approach for selecting k representative points from a database such that every user's ideal point in the entire database is similar to one of the k points. This approach combines benefits of the methods top- k and skyline; it controls the size of the output but does not require knowledge of users' preferences. Prior work with k -regret queries assumes users' preferences to be modeled by linear utility functions. In this paper, we derive upper and lower bounds for nonlinear utility functions, as these functions can better fit occurrences such as diminishing marginal returns, propensity for risk, and substitutability of preferences. To model these phenomena, we analyze a broad subset of convex, concave, and constant elasticity of substitution functions. We also run simulations on real and synthetic data to prove the efficacy of our bounds in practice.

Future autonomous robots deployed in homes and workplaces must be able to competently ask for help when situations arise that fall outside of their knowledge or capabilities. When asking for help, these robots must provide enough information to solve the problem without giving overwhelming amounts of information. We present a novel algorithm that learns to predict how natural language utterances affect a classifier model of human task knowledge. We show how these predictions can be used to generate language that is both minimal and sufficiently communicative.