Effective robotic teammates should be able to interact with humans in natural language about all task aspects, keep track of task and team states to coordinate their actions, and handle unexpected events autonomously. In this paper, we introduce a multi-robot architectural framework for effective robot teammates that allows robots to learn new tasks on the fly and monitor task execution to be able to detect unexpected faults and events. It enables robots to generate recovery plans, assess their effectiveness, and engage with human teammates in problem solving dialogues. We demonstrate the capabilities and operation of the framework in a complex mixed-initiative human-robot medical assembly and delivery task.

"Open world" environments are those in which novel objects, agents, events, and more can appear and contradict previous understandings of the environment. This runs counter to the "closed world" assumption used in most AI research, where the environment is assumed to be fully understood and unchanging. The types of environments AI agents can be deployed in are limited by the inability to handle the novelties that occur in open world environments. This paper presents a novel cognitive architecture framework to handle open -world novelties. This framework combines symbolic planning, counterfactual reasoning, reinforcement learning, and deep computer vision to detect and accommodate novelties. We introduce general algorithms for exploring open worlds using inference and machine learning methodologies to facilitate novelty accommodation. The ability to detect and accommodate novelties allows agents built on this framework to successfully complete tasks despite a variety of novel changes to the world. Both the framework components and the entire system are evaluated in Minecraft-like simulated environments. Our results indicate that agents are able to efficiently complete tasks while accommodating "concealed novelties" not shared with the architecture development team.

We present a novel integration between a computational framework for modeling attention-driven perception and cognition (ARCADIA) with a cognitive robotic architecture (DIARC), demonstrating how this integration can be used to drive the gaze behavior of a robotic platform. Although some previous approaches to controlling gaze behavior in robots during human-robot interactions have relied either on models of human visual attention or human cognition, ARCADIA provides a novel framework with an attentional mechanism that bridges both lower-level visual and higher-level cognitive processes. We demonstrate how this approach can produce more natural and human-like robot gaze behavior. In particular, we focus on how our approach can control gaze during an interactive object learning task. We present results from a pilot crowdsourced evaluation that investigates whether the gaze behavior produced during this task increases confidence that the robot has correctly learned each object.

Regular irradiation of indoor environments with ultraviolet C (UVC) light has become a regular task for many in-door settings as a result of COVID-19, but current robotic systems attempting to automate it suffer from high costs and inefficient irradiation. In this paper, we propose a purpose-made inexpensive robotic platform with off-the-shelf components and standard navigation software that, with a novel algorithm for finding optimal irradiation locations, addresses both shortcomings to offer affordable and efficient solutions for UVC irradiation. We demonstrate in simulations the efficacy of the algorithm and show a prototypical run of the autonomous integrated robotic system in an indoor environment. In our sample instances, our proposed algorithm reduces the time needed by roughly 30% while it increases the coverage by a factor of 35% (when compared to the best possible placement of a static light).

Autonomous robots with sophisticated capabilities can make it difficult for human instructors to assess its capabilities and proficiencies. Therefore, it is important future robots have the ability to: introspect on their capabilities and assess their task performance. Introspection allows the robot to determine what it can accomplish and self-assessment allows the robot estimate the likelihood it will accomplish at given task. We introduce a general framework for introspection and self-assessment that enables robots to have task and performance-based dialogues before, during, and after a mission. We then realize aspects of the framework in the cognitive robotic DIARC architecture, and finally show a proof-of-concept demonstration on a Nao robot showing its self-assessment capabilities before, during, and after an instructed task.

Past work on acquisition of word-object associations in robots has focused on either fast instruction-based methods which accept highly constrained input or gradual cross-situational learning methods, but not a mixture of both. In this paper, we present an integrated robotic system which allows for a combination of these methods to contribute to the task of learning the labels of objects in AI agents. We demonstrate the expanded word learning capabilities in the outcome system and how learning from both human-human and human-robot dialogues can be achieved in one integrated system.

DIARC has been under development for over 15 years. Different from other cognitive architectures like SOAR or ACT-R, DIARC is an intrinsically component-based distributed architecture scheme that can be instantiated in many different ways. Moreover, DIARC has several distinguishing features, such as affect processing and deep natural language integration, is open-world and multi-agent enabled, and allows for "one-shot instruction-based learning" of new percepts, actions, concepts, rules, and norms. In this chapter, we will present an overview of the DIARC architecture and compare it to classical cognitive architectures. After laying out the theoretical foundations, we specifically focus on the action, vision, and natural language subsystems. We then give two examples of DIARC configurations for "one-shot learning" and "component-sharing". We also briefly mention different use cases of DIARC, in particular, for autonomous robots in human-robot interaction experiments and for building cognitive models.

Learning new knowledge from single instructions and being able to apply it immediately is highly desirable for artificial agents. We provide the first demonstration of spoken instruction-based one-shot object and action learning in a cognitive robotic architecture and briefly discuss the architectural modifications required to enable such fast learning, demonstrating the new capabilities on a fully autonomous robot.

Human-robot communication in situated environments involves a complex interplay between knowledge representations across a wide variety of modalities. Crucially, linguistic information must be associated with representations of objects, locations, people, and goals, which may be represented in very different ways. In previous work, we developed a Consultant Framework that facilitates modality-agnostic access to information distributed across a set of heterogeneously represented knowledge sources. In this work, we draw inspiration from cognitive science to augment these distributed knowledge sources with Short Term Memory Buffers to create an STM-augmented algorithm for referring expression generation. We then discuss the potential performance benefits of this approach and insights from cognitive science that may inform future refinements in the design of our approach.

Learning the meaning of words in noisy contexts with multiple unknown words in an utterance and multiple unknown objects in a scene is a typical part of language acquisition for infants. However, incremental word learning in ambiguous contexts is a challenging problem in artificial intelligence. While past models of cross-situational word learning benefit from full access to all learning situations and their statistical regularities to arrive at the right hypothesis, it is cognitively implausible for children to remember all word learning situations they encounter. Hence, we present an incremental Bayesian model of cross-situational word learning with limited access to past situations and demonstrate its superior performance compared to other baseline incremental models, especially under conditions of sensory noise in the speech and visual modalities. Then we embed our model in a cognitive robotic architecture and demonstrate the first robotic model capable of incremental cross-situational word learning.

Learning new knowledge from single instructions and being able to apply it immediately is a highly desirable capability for artificial agents. We provide the first demonstration of spoken instruction-based one-shot object and action learning in a cognitive robotic architecture and discuss the modifications to several architectural components required to enable such fast learning, demonstrating the new capabilities on two different fully autonomous robots.

Humans are often able to generalize knowledge learned from a single exemplar. In this paper, we present a novel integration of mental simulation and analogical generalization algorithms into a cognitive robotic architecture that enables a similarly rudimentary generalization capability in robots. Specifically, we show how a robot can generate variations of a given scenario and then use the results of those new scenarios run in a physics simulator to generate generalized action scripts using analogical mappings. The generalized action scripts then allow the robot to perform the originally learned activity in a wider range of scenarios with different types of objects without the need for additional exploration or practice. In a proof-of-concept demonstration we show how the robot can generalize from a previously learned pick-and-place action performed with a single arm on an object with a handle to a pick-and-place action of a cylindrical object with no handle with two arms.

An Embodied Real-Time Model of Language-Guided Incremental Visual Search Matthias Scheutz and Evan Krause and Sepideh Sadeghi {matthias.scheutz,evan.krause,sepideh.sadeghi}@tufts.edu Human-Robot Interaction Laboratory, Department of Computer Science, Tufts University Medford, MA 02155, USA Abstract A recent body of work has demonstrated that the incremen- tal presentations of linguistic search cues can speed up visual processing in conjunctive visual search. In this paper, we in- vestigate different processing configurations using a real-time embodied computational model and demonstrate that, differ- ent from previous hypotheses, the same incremental process- ing configuration can explain all experimental conditions. Keywords: Incremental interactive processing, embodied real-time model, natural language and vision interaction Introduction A large body of work in cognitive science has demonstrated that much of human information processing is interactive and incremental: “interactive” means that information is ex- changed and shared among multiple processes; “incremen- tal” means that the received information is integrated as it be- comes available. Hence, interactive incremental processing modules can incorporate information from other modules as constraints in their own processing and thus potentially finish their processing sooner. A well-studied case of such interactive incremental pro- cessing is the interaction between visual and natural language processes. Converging evidence from studies using, in par- ticular, the “visual words paradigm” demonstrate that vision and natural language processing in humans are highly interac- tive and incremental, being able to utilize constraints from the other modality to reduce processing effort and improve pro- cessing performance (Eberhard, Spivey-Knowlton, Sedivy, & Tanenhaus, 1995). For example, a visual search process at- tempting to find a target object in a visual scene such as a particular pen on a cluttered desk can be modulated through natural language instructions that provide additional informa- tion about the object (e.g., “small black”), leading to a more targeted, faster search (Spivey, Tyler, Eberhard, & Tanenhaus, 2001; Krause, Cantrell, Potapova, Zillich, & Scheutz, 2013). Conversely, visual processing of a scene can influence natu- ral language processing by helping to disambiguate otherwise ambiguous referential phrases such as the syntactic ambigu- ity due to different possible prepositional attachments in “put the black pen on the book on the table” where the black pen could be put either on the table or on the book that is on the ta- ble (Eberhard et al., 1995; Scheutz, Eberhard, & Andronache, 2004; Brick & Scheutz, 2007). While various theoretically motivated hypotheses have been proposed about an underlying processing architecture that could enable such incremental natural language and vi- sion interaction and information integration, only a few com- putational models actually demonstrate possible computa- tional mechanisms (Scheutz et al., 2004; Hamker, 2004; Brick & Scheutz, 2007; Chiu & Spivey, 2012; Krause et al., 2013). However, computational models are often necessary to show that conclusions drawn about the processing archi- tecture based on experimental evidence or theory alone might not be warranted. In this paper, we present an embodied real-time model of interactive incremental vision and natural language process- ing that can explain previous experimental findings in a novel way by showing that divergent results found in different ex- perimental conditions by Spivey et al. (2001) might not be due to differences in processing configurations (such as serial vs. parallel), but rather the specific effects of these experi- mental manipulations on the same processing configuration. We start by reviewing some of the empirical evidence for the hypothesis that natural language can incrementally con- strain vision processing and describe, in particular, the exper- iments in (Spivey et al., 2001) which we use for our model simulations. Next, we introduce the model architecture and provide a more detailed description of its vision system which is critical for the replication of the human data. We then spec- ify the simulation setup, which used the same human stimuli as Spivey et al. (2001), and report the results from extensive simulation experiments with different configurations of the processing system. The analysis of the simulation data con- firms many of the expected properties, but also shows that the same configuration can explain different experimental con- ditions that have been assumed to be the result of different processing configurations. This point is further elaborated in the subsequent discussion section and summarized in the con- clusion which also points to future work. Background and Motivation It has long been hypothesized that early stages of bottom- up visual processing are highly parallel as single-feature vi- sual search is not affected by the number of co-present dis- tractors, while later stages must include a “serial bottleneck” since conjunctive visual search (assumed to tap into later pro- cessing stages) takes longer as the number of distractors in- creases (Wolfe, 2007). While various stimuli properties can affect search speed in conjunctive search, Spivey and col- leagues demonstrated in a series of experiments that the in- cremental presentation of linguistic search cues can reduce the effect of distractors in the visual search process (Spivey et al., 2001; Reali, Spivey, Tyler, & Terranova, 2006; Chiu & Spivey, 2011, 2012). They hypothesized that the incremen- tal presentation of search cues (which is natural in spoken language) enforced a serialization of the search process, al- lowing search results based on the first cue to be utilized in

Being able to quickly and naturally teach robots new knowledge is critical for many future open-world human-robot interaction scenarios. In this paper we present a novel approach to using natural language context for one-shot learning of visual objects, where the robot is immediately able to recognize the described object. We describe the architectural components and demonstrate the proposed approach on a robotic platform in a proof-of-concept evaluation.

Humans expect interlocutors both human and robot to resolve spoken references to visually-perceivable objects incrementally as the referents are verbally described. For this reason, tight integration of visual search with natural language processing, and real-time operation of both are requirements for natural interactions between humans and robots. In this paper, we present an integrated robotic architecture with novel incremental vision and natural language processing. We demonstrate that incrementally refining attentional focus using linguistic constraints achieves significantly better performance of the vision system compared to non-incremental visual processing.

Natural human-like human-robot interactions require many functional capabilities from a robot that have to be reflected in architectural components in the robotic control architecture. In particular, various mechanisms for producing social behaviors, goal-oriented cognition, and robust intelligence are required. In this paper, we present an overview of the most recent version of our DIARC architecture and show how several novel algorithms attempt to address these three areas, leading to more natural interactions with humans, while also extending the overall capability of the integrated system.

Introspection mechanisms are employed in agent architectures toimprove agent performance. However, there is currently no approach tointrospection that makes automatic adjustments at multiple levels inthe implemented agent system. We introduce our novel multi-levelintrospection framework that can be used to automatically adjustarchitectural configurations based on the introspection results at theagent, infrastructure and component level. We demonstrate the utilityof such adjustments in a concrete implementation on a robot where thehigh-level goal of the robot is used to automatically configure thevision system in a way that minimizes resource consumption whileimproving overall task performance.

Human-robot interaction poses tight timing requirements on visual as well as natural language processing in order to allow for natural human-robot interaction. In particular, humans expect robots to incrementally resolve spoken references to visually perceivable objects as the referents are verbally described. In this paper, we present an integrated robotic architecture with novel incremental vision and natural language processing and demonstrate that incrementally rening attentional focus using linguistic constraints achieves significantly better performance of the vision system compared to non-incremental visual processing.

Robots participating in natural dialogue may need to discuss, reason about, or initiate actions concerning dialoguereferenced entities. To do so, the robot must first identify or create new representations for those entities, a capability known as reference resolution. We previously presented GH-POWER: an algorithm that used a Givenness Hierarchy theoretic approach to resolving definite, indefinite, anaphoric, and deictic noun phrases in uncertain and open worlds. In this work, we introduce GROWLER: a new reference resolution algorithm which enables more robust reference resolution by extending GH-POWER with a model of relevance, and discuss how this extension is able to handle some cases not handled by our original algorithm.