To investigate the use of different pointing forms in service scenarios, we collected the ShopPoint dataset, a skeleton-based dataset of pointing gestures from customer–shopkeeper interactions in a camera shop scenario. Thirteen participants took part in the data collection, including 3 shopkeepers with real-world customer service experience and 10 customers. We recorded 61 one-to-one role-played interactions. Coders annotated pointing gestures from videos of these interactions, emphasizing pointing arm forms (straight-arm, bent-arm, and hand-only pointing) and hand forms (index-finger and open-hand pointing). This annotation process resulted in 2,959 pointing gestures. We conducted statistical analysis on the annotated data. The analysis revealed that bent-arm pointing was used more frequently than other arm forms. Straight-arm pointing was used more for far targets than for close targets, and hand-only was used more for close targets. Shopkeepers used bent-arm pointing more frequently than customers when referring to far targets. To evaluate the recognition of these pointing gestures, we tested several existing Skeleton-based Action Recognition (SAR) methods on the dataset. The highest accuracy was achieved at 72.51% by using transfer learning (i.e., pretraining and fine-tuning). This evaluation indicates that though transfer learning aids performance, recognizing pointing with diverse forms remains challenging.

Data-driven social imitation learning is a minimally supervised approach to generating robot behaviors for human–robot interaction (HRI). However, like any learning-based approach, there are inherent challenges and potential errors. Existing strategies for error prevention in HRI predominantly rely on handcrafted methods and predefined design guidelines, with limited focus on learning-based approaches. In this article, we present an autonomous, online continual learning strategy for data-driven HRI that uses error detections to facilitate error prevention. We leverage error detections as negative training signals, which dynamically update the data-driven robot action model parameters, guiding it away from repeating errors in future interactions. This approach modifies the standard (offline) data-driven HRI training objective to continually refine the robot’s performance in an online manner. We use simulated HRI data to show that our error prevention method can effectively reduce the average rate of robot errors in HRI by up to 6.68%. Furthermore, even with a limited-precision error detector, our method can achieve a 4.68% error rate reduction, paving the way towards a completely autonomous error management framework for data-driven HRI.

We proposed a system for generating relational gestures that convey semantic relations such as similarity and difference between two objects. To understand how humans naturally express such relations, we conducted an observational study with experienced shopkeepers as they frequently compare objects using both speech and gestures. Through analysis of their interactions, we identified four common types of object relations and extracted representative gesture patterns for each. For example, similarity was often conveyed through synchronized hand movements bringing both hands closer together, accompanied by alternating gaze between two objects. Based on these findings, we developed a gesture generation system in which one large language model (LLM) infers the intended object relation from utterance text, and another LLM adapts gestures from a co-speech gesture system that aligns them with speech, integrating relational cues without disrupting this alignment. These modified gestures were automatically mapped onto a dual-arm robot. We evaluated the system through two user studies. In the first study, 20 participants were asked to identify object relations from 24 relational gestures performed by the robot without accompanying speech. They correctly identified the intended relations with an average accuracy of 89.8% across all relation types. In the second study, another 20 participants compared two robot conditions in a within-subjects design: one with relational gestures and one without. Results showed that the robot using relational gestures was perceived as more competent, sociable, and animate compared to the robot without them.

When verbal communication is limited, robots passing objects to humans without providing additional information (e.g., temperature, weight) can result in potential poor handovers and disappointing user experiences. To address this issue, we introduced a method for conveying object properties through robot manipulation. We began by proposing four criteria for selecting properties from two widely recognized sets: one focusing on the semantic features of objects and the other on tactile sensations. These properties were clustered into eight physical categories: hot, cold, heavy, light, slippery, sticky, fragile, and smelly. Professional actors were then recruited to demonstrate these properties through object manipulation, from which we extracted a set of fundamental yet expressive manipulation behaviors, i.e., key elements, that help people recognize these properties. These elements were implemented on a dual-arm robot, followed by an evaluation of their utility through participant feedback. To generate time-constrained sequences of elements, we developed a property-based motion planner that balances time and utility in conveying object properties. Results from a within-subjects study involving 20 participants showed that individuals could accurately interpret the properties conveyed by robot object manipulation, validating the effectiveness of the proposed approach.

Data-driven social imitation learning is a minimally-supervised approach to generating robot behaviors for human-robot interaction (HRI). However, this type of learning-based approach is error-prone. Existing error detection methods for HRI rely on data labeling, rendering them inappropriate for the data-driven paradigm. We present a zero-shot error detection strategy that requires no labeled data. We use human interaction data to learn models of normal human behavior, then use these models to extract features that help discriminate abnormal human reactions to robot errors. In this feature space, we frame error detection as a novelty detection task, utilizing human interaction data to learn a model of non-erroneous interactions in an unsupervised fashion. Then, we apply the fitted novelty detector to HRI data to identify erroneous robot behavior. We show that our method obtains an average precision of 0.497 on errors, outperforming unsupervised baselines and supervised approaches with limited training data.

In retail settings, a robot's one-handed manipulation of objects can come across as thoughtless and impolite, thus creating a negative customer experience. To solve this problem, we first observed how human shopkeepers interact with customers, specifically focusing on their hand movements during object manipulation. From the observation and analysis of shopkeepers' hand movements, we identified an essential element of their idle hand movements: "support" provided by the idle hand as the primary hand manipulates an object. Based on this observation, we proposed a model that coordinates the movements of a robot's idle hand with its primary task-engaged hand, emphasizing its supportive behaviors. In a within-subjects study, 20 participants interacted with robot shopkeepers under different conditions to assess the impact of incorporating support behavior with the idle hand. The results show that the proposed model significantly outperforms a baseline in terms of politeness and competence, suggesting enhanced object-based interactions between the robot shopkeepers and customers.

Data-driven imitation learning enables service robots to learn social interaction behaviors, but these systems cannot adapt after training to changes in the environment, such as changing products in a store. To solve this, a novel learning system that uses neural attention and approximate string matching to copy information from a product information database to its output is proposed. A camera shop interaction dataset was simulated for training/testing. The proposed system was found to outperform a baseline and a previous state of the art in an offline, human-judged evaluation.

We envision a future where service robots autonomously learn how to interact with humans directly from human-human interaction data, without any manual intervention. In this paper, we present a data-driven pipeline that: (1) takes in low-level data of a human shopkeeper interacting with multiple customers (28 hours of collected data); (2) autonomously extracts high-level actions from that data; and (3) learns -- without manual intervention -- how a robotic shopkeeper should respond to customers' actions online. Our proposed system for learning the interaction logic uses neural networks to first learn which customer actions are important to respond to and then learn how the shopkeeper should respond to those important customer actions. We present a novel technique for learning which customer actions are important by first learning the hidden causal relationship between customer and shopkeeper actions. In an offline evaluation, we show that our proposed technique significantly outperforms state-of-the-art baselines, in both which customer actions are important and how to respond to them.

This study presents a learning-by-imitation technique that learns social robot interaction behaviors from natural human- human interaction data and requires minimum input from a designer. To solve the problem of responding to ambiguous human actions, a novel topic clustering algorithm based on action cooccurrence frequencies is introduced. The system learns human-readable rules that dictate which action the robot should take, based on the most recent human action and the current estimated topic of conversation. The technique is demonstrated in a scenario where the robot learns to play the role of a travel agent. The proposed technique outperformed several baseline techniques in qualitative and quantitative evaluations. It responded more accurately to ambiguous questions and participants found it was easier to understand, provided more information, and required less effort to interact with.

Learning from human interaction data is a promising approach for developing robot interaction logic, but behaviors learned only from offline data simply represent the most frequent interaction patterns in the training data, without any adaptation for individual differences. We developed a robot that incorporates both data-driven and interactive learning. Our robot first learns high-level dialog and spatial behavior patterns from offline examples of human-human interaction. Then, during live interactions, it chooses among appropriate actions according to its curiosity about the customer's expected behavior, continually updating its predictive model to learn and adapt to each individual. In a user study, we found that participants thought the curious robot was significantly more humanlike with respect to repetitiveness and diversity of behavior, more interesting, and better overall in comparison to a non-curious robot.

Many recent studies have shown that behaviors and interaction logic for social robots can be learned automatically from natural examples of human-human interaction by machine learning algorithms, with minimal input from human designers [1--4]. In this work, we exceed the capabilities of the previous approaches by giving the robot memory . In earlier work, the robot's actions were decided based only on a narrow temporal window of the current interaction context. However, human behaviors often depend on more temporally distant events in the interaction history. Thus, we raise the question of whether (and how) an automated behavior learning system can learn a memory representation of interaction history within a simulated camera shop scenario. An analysis of the types of memory-setting and memory-dependent actions that occur in the camera shop scenario is presented. Then, to create more examples of such actions for evaluating a shopkeeper robot behavior learning system, an interaction dataset is simulated. A Gated Recurrent Unit (GRU) neural network architecture is applied in the behavior learning system, which learns a memory representation for performing memory-dependent actions. In an offline evaluation, the GRU system significantly outperformed a without-memory baseline system at generating appropriate memory-dependent actions. Finally, an analysis of the GRU architecture's memory representation is presented.

To overcome these challenges, we have proposed the approach of learning interaction logic directly from data captured from natural human-human interactions. In comparison with teleoperation or web-based crowdsourcing, our approach has the benefit of capturing the naturalness and immersion of real interactions, but it faces the added challenges of dealing with sensor noise and an unconstrained action space.

Linguistics studies have shown that action verbs often denote some Change of State (CoS) as the result of an action.However, the causality of action verbs and its potential connection with the physical world has not been systematically explored.To address this limitation, this paper presents a study on physical causality of action verbs and their implied changes in the physical world.We first conducted a crowdsourcing experiment and identified eighteen categories of physical causality for action verbs.For a subset of these categories, we then defined a set of detectors that detect the corresponding change from visual perception of the physical environment.We further incorporated physical causality modeling and state detection in grounded language understanding.Our empirical studies have demonstrated the effectiveness of causality modeling in grounding language to perception.

To facilitate referential communication between humans and robots and mediate their differences in representing the shared environment, we are exploring embodied collaborative models for referring expression generation (REG). Instead of a single minimum description to describe a target object, episodes of expressions are generated based on human feedback during human-robot interaction. We particularly investigate the role of embodiment such as robot gesture behaviors (i.e., pointing to an object) and human's gaze feedback (i.e., looking at a particular object) in the collaborative process. This paper examines different strategies of incorporating embodiment and collaboration in REG and discusses their possibilities and challenges in enabling human-robot referential communication.

In situated dialogue with artificial agents (e.g., robots), although a human and an agent are co-present, the agent's representation and the human's representation of the shared environment are significantly mismatched. Because of this misalignment, our previous work has shown that when the agent applies traditional approaches to generate referring expressions for describing target objects with minimum descriptions, the intended objects often cannot be correctly identified by the human. To address this problem, motivated by collaborative behaviors in human referential communication, we have developed two collaborative models - an episodic model and an installment model - for referring expression generation. Both models, instead of generating a single referring expression to describe a target object as in the previous work, generate multiple small expressions that lead to the target object with the goal of minimizing the collaborative effort. In particular, our installment model incorporates human feedback in a reinforcement learning framework to learn the optimal generation strategies. Our empirical results have shown that the episodic model and the installment model outperform previous non-collaborative models with an absolute gain of 6% and 21% respectively.