Human communication relies on multiple modal-ities-language, gestures, facial expressions, and more-each conveying different types of information over varying time scales. Over time, humans gradually learn to integrate these modalities to understand others and interpret their commands. While recent advances in machine learning have enabled robots to fuse information from images, videos, and language commands, effectively integrating human gestures and linguistic instructions in real-world scenarios remains a challenge. In this paper, we present a simulated dataset of varying complexity, including ambiguous language commands and ambiguous pointing gestures, which might be used for evaluating different learning approaches. On this dataset, we quantitatively compare 3 different approaches for combining ambiguous information from language and pointing gestures - both learning-based and deterministic ones. Furthermore, we demonstrate how the mapping between these two modalities can be gradually learned with an increasing number of observations and samples.
Transcribing naturalistic caregiver-child interactions is a labor-intensive task in developmental research. While automatic speech recognition (ASR) models offer potential solutions, current ASR systems, including OpenAI’s Whisper, were primarily trained on adult-directed speech (ADS) and have not been systematically evaluated for infant-directed speech (IDS). IDS differs from ADS in prosody, phonetics, utterance structure, and lexical patterns, posing unique challenges for ASR transcription. In this study, we benchmarked Whisper’s transcription accuracy on a dataset of naturalistic caregiverinfant recordings, comparing ASR-generated transcripts to human-annotated gold-standard transcriptions. We evaluated Whisper’s performance using word error rate (WER) and accuracy, and conducted a failure analysis to identify systematic errors made by Whisper, including utterance segmentation mismatches, high error rates in short and repetitive speech and inconsistent filler word handling. Results indicated that Whisper struggles with short, contextually ambiguous utterances and fails to reliably segment IDS utterances. Based on these results and early user experience, we propose a preliminary pipeline for integrating ASR technology with human transcription workflows to enhance the efficiency of processing large-scale naturalistic speech data in developmental research.
This study investigated how action histories – unfolding sequences of actions with objects – provide a context for both attentional allocation and linguistic repair strategies. Building on theories of enactive cognition [34] and sensorimotor contingency theory [40], we experimentally manipulated action sequences (action history) to create either simple or rich “situational models,” and investigated how these models interact with attention and reflect in linguistic processes during human–robot interaction. Participants (N = 30) engaged in a controlled object placement task with a humanoid robot, where the action (manner) information was either provided or omitted. The omission elicited repair behaviors in participants that were in focus of our investigation. For rich models (competing action possibilities) participants demonstrated: a) increased attentional reorientation, reflecting active engagement with the situational model b) preference for restricted repairs, targeting the specific source of trouble in action selection. Conversely, a simple situational model led to more generalized attention patterns and open repair strategies, suggesting weaker constraints on internal processing. These findings highlight how situational structures emerge externally to scaffold internal cognitive processes, with action histories serving as a crucial context for the interface between perception, action, and language. We discuss how to implement such a tight loop in the assistance of a system.
This paper presents a novel approach to address personalized motion retargeting (PMR) from humans to robots. In contrast to most existing retargeting methods that overlook personalized human motion features, we propose a continuous learning framework that enables robots to learn and refine retargeting through human-robot interaction. Within the framework, we design a bidirectional imitation loop to encode the user’s preferences: The robot imitates the human first, and then the human replicates the robot’s movements. By being imitated, the robot gains insight into the effect of its own action, allowing the robot to adjust its retargeting accordingly. Furthermore, to enhance the interaction efficiency, our method significantly reduces training time by optimizing the robot’s local behavior rather than learning from scratch, leading to rapid and responsive personalization in seconds.
This study investigates how school-aged children ($\mathbf{N} \boldsymbol{=} \mathbf{6 0}$, ages 8–12) from a non-WEIRD population in Egypt balance breadth and depth during self-directed exploration. Using a novel “TRIVIA” paradigm, children freely explored topics across three days, with their choices analyzed through linear and cumulative link mixed models. Results revealed that socioeconomic status (SES) significantly influenced exploration patterns: lower-SES children explored more questions on the second day compared to higher-SES peers ($\beta=-0.17, 95\% CI [-0.28, -0.05]$) and were more likely to visit previously explored questions ($\beta=-0.005, 95 \% CI [-0.009,-0.002]$). While age did not predict exploration quantity, children progressively shifted from in-breadth to in-depth exploration as the task progressed ($\beta=0.295,95 \% CI [0.092,0.498]$). This effect was moderated by SES, with higher-SES children’s exploration becoming more in-depth over time ($\beta=-0.008, 95\% CI [-0.012,-0.004])$. Motivation and active learning scores showed no significant effects. These findings highlight how SES and task structure shape children’s exploration patterns, offering insights into curiosity-driven learning in understudied cultural contexts.
