Deployed, autonomous AI systems must often evaluate multiple plausible courses of action (extended sequences of behavior) in novel or under-specified contexts. Despite extensive training, these systems will inevitably encounter scenarios where no available course of action fully satisfies all operational constraints (e.g., operating procedures, rules, laws, norms, and goals). To achieve goals in accordance with human expectations and values, agents must go beyond their trained policies and instead construct, evaluate, and justify candidate courses of action. These processes require contextual "knowledge" that may lie outside prior (policy) training. This paper characterizes requirements for agent decision making in these contexts. It also identifies the types of knowledge agents require to make decisions robust to agent goals and aligned with human expectations. Drawing on both analysis and empirical case studies, we examine how agents need to integrate normative, pragmatic, and situational understanding to select and then to pursue more aligned courses of action in complex, real-world environments.

Interactive Task Learning (ITL) enables cognitive agents to learn novel tasks (in one shot) from natural-language instruction and allows humans to customize agents to align with individual preferences. ITL relies on reasoning over and learning from multiple sources of knowledge, a strength of cognitive architectures. However, ITL requires frequent human input, which can be tedious and time-consuming. We evaluate large language models (LLMs) as an additional source of knowledge for ITL. We summarize initial experiments exploring the potential use of LLMs in ITL and then describe a novel method (STARS) that markedly improves the reliability of task learning from LLMs for embodied ITL agents. We demonstrate that LLMs speed agent learning and greatly reduce the human effort required to achieve robust, reliable, and repeatable task performance.

Large language models (LLMs) offer unprecedented natural-language understanding and generation capabilities. However, evaluations of their ability to demonstrate other cognitive functions, especially various categories of reasoning, have been, at best, mixed. The limited scope of reliable and robust LLM capabilities has resulted in a new class of AI systems, LLM-Modulo AI, in which LLMs are used to contribute to the overall capabilities of an intelligent system. In this paper, we explore the applicability of LLMs for one specific capability acutely missing in most cognitive systems: problem formulation. Cognitive systems generally require a human to translate a problem definition into some specification that the cognitive system can use to attempt to solve the problem or perform the task. We explore how large language models (LLMs) can be utilized to map a problem class, defined in natural language, into a semi-formal specification that can then be utilized by an existing reasoning and learning system to solve instances from the problem class. The result is a Modulo-LLM cognitive system in which the LLM roughly acts as a cognitive task analyst, generating a problem specification that can be used by a typical cognitive system to solve specific problems. The agent uses prompts derived from the definition of problem spaces in the AI literature and general problem-solving strategies (Polya’s How to Solve It). We offer preliminary evidence illustrating the potential for LLM-based problem specification. Such automatic problem specification offers the potential to speed cognitive systems research via disintermediation of problem formulation while also retaining core capabilities of cognitive systems, such as robust inference and online learning.

One goal of AI (and AGI) is to identify and understand specific mechanisms and representations sufficient for general intelligence. Often, this work manifests in research focused on architectures and many cognitive architectures have been explored in AI/AGI. However, different research groups and even different research traditions have somewhat independently identified similar/common patterns of processes and representations or cognitive design patterns that are manifest in existing architectures. Today, AI systems exploiting large language models (LLMs) offer a relatively new combination of mechanisms and representations available for exploring the possibilities of general intelligence. This paper outlines a few recurring cognitive design patterns that have appeared in various pre-transformer AI architectures. We then explore how these patterns are evident in systems using LLMs, especially for reasoning and interactive (“agentic”) use cases. By examining and applying these recurring patterns, enables predictions of gaps or deficiencies in today’s Agentic LLM Systems and identification of subjects of future research towards general intelligence using generative foundation models.

Regardless of past learning, an agent in an open world will face unfamiliar events outside of prior experience, existing models, or policies. Further, the agent will sometimes lack relevant knowledge and/or sufficient time to assess the situation and evaluate response options. How can an agent respond reasonably to situations that are outside of its original design scope? How can it recognize such situations sufficiently quickly and reliably to determine reasonable, adaptive courses of action? We identify key characteristics needed for solutions, review the state-of-the-art, and outline a proposed, novel approach that combines domain-general meta-knowledge (inspired by human cognition) and metareasoning. This approach offers potential for fast, adaptive responses to unfamiliar situations, more fully meeting the performance characteristics required for open-world, general agents.

One part of complying with norms, rules, and preferences is incorporating constraints (such as knowledge of ethics) into one's goal formulation and planning processing. We explore in a simple domain how the encoding of knowledge in different ethical frameworks influences an agent's goal formulation and planning processing and demonstrate ability of an agent to satisfy and satisfice when its collection of relevant constraints includes a mix of "hard" and "soft" constraints of various types. How the agent attempts to comply with ethical constraints depends on the ethical framing and we investigate tradeoffs between deontological framing and utilitarian framing for complying with an ethical norm. Representative scenarios highlight how performing the same task with different framings of the same norm leads to different behaviors. Our explorations suggest an important role for metacognitive judgments in resolving ethical conflicts during goal formulation and planning.

Cognitive systems generally require a human to translate a problem definition into some specification that the cognitive system can use to attempt to solve the problem or perform the task. In this paper, we illustrate that large language models (LLMs) can be utilized to map a problem class, defined in natural language, into a semi-formal specification that can then be utilized by an existing reasoning and learning system to solve instances from the problem class. We present the design of LLM-enabled cognitive task analyst agent(s). Implemented with LLM agents, this system produces a definition of problem spaces for tasks specified in natural language. LLM prompts are derived from the definition of problem spaces in the AI literature and general problem-solving strategies (Polya's How to Solve It). A cognitive system can then use the problem-space specification, applying domain-general problem solving strategies ("weak methods" such as search), to solve multiple instances of problems from the problem class. This result, while preliminary, suggests the potential for speeding cognitive systems research via disintermediation of problem formulation while also retaining core capabilities of cognitive systems, such as robust inference and online learning.

Large language models (LLMs) offer significant promise as a knowledge source for task learning. Prompt engineering has been shown to be effective for eliciting knowledge from an LLM, but alone it is insufficient for acquiring relevant, situationally grounded knowledge for an embodied agent learning novel tasks. We describe a cognitive-agent approach, STARS, that extends and complements prompt engineering, mitigating its limitations and thus enabling an agent to acquire new task knowledge matched to its native language capabilities, embodiment, environment, and user preferences. The STARS approach is to increase the response space of LLMs and deploy general strategies, embedded within the autonomous agent, to evaluate, repair, and select among candidate responses produced by the LLM. We describe the approach and experiments that show how an agent, by retrieving and evaluating a breadth of responses from the LLM, can achieve 77-94% task completion in one-shot learning without user oversight. The approach achieves 100% task completion when human oversight (such as an indication of preference) is provided. Further, the type of oversight largely shifts from explicit, natural language instruction to simple confirmation/discomfirmation of high-quality responses that have been vetted by the agent before presentation to a user.

We explore the potential integration of Transformers, trained online in real-time using an agent's ongoing experiences, as a learning and memory component of a cognitive architecture such as Soar. We identify key challenges and potential capabilities enabled by such an integration.

Human behavior is conditioned by codes and norms that constrain action. Rules, “manners,” laws, and moral imperatives are examples of classes of constraints that govern human behavior. These systems of constraints are “messy:” individual constraints are often poorly defined, what constraints are relevant in a particular situation may be unknown or ambiguous, constraints interact and conflict with one another, and determining how to act within the bounds of the relevant constraints may be a significant challenge, especially when rapid decisions are needed. General, artificially-intelligent agents must be able to navigate the messiness of systems of real-world constraints in order to behave predictability and reliably. In this paper, we characterize sources of complexity in constraint processing for general agents and describe a computational-level analysis for such constraint compliance. We identify key algorithmic requirements based on the computational-level analysis and outline a limited, exploratory implementation of a general approach to constraint compliance.

Large language models (LLMs) provide capabilities far beyond sentence completion, including question answering, summarization, and natural-language inference. While many of these capabilities have potential application to cognitive systems, our research is exploiting language models as a source of task knowledge for cognitive agents, that is, agents realized via a cognitive architecture. We identify challenges and opportunities for using language models as an external knowledge source for cognitive systems and possible ways to improve the effectiveness of knowledge extraction by integrating extraction with cognitive architecture capabilities, highlighting with examples from our recent work in this area.

Autonomous AI systems that act and interact in human environments must generally comply with systems of rules, laws, norms, and moral principles that humans are expected to abide by. One of the core challenges of maintaining compliance with such systems of constraints is that the current situation (as perceived and experienced by an agent) may be insufficient for immediately determining whether the agent is in compliance with any one of its (often many) individual constraints. We term this problem partial grounding of constraint specifications: in many situations, an agent will only be able to partially ground its evaluation of constraints. We define this problem, describe some requirements for mitigating it, and present a specific computational approach. The approach enables an agent to anticipate potential grounding issues when constraints are introduced. The anticipatory processing then allows an agent to respond more quickly to novel instances of partially-grounded constraint specifications when/as they arise in execution. The approach is assessed in the context of an analytic tool proposed to help researchers compare and evaluate approaches to grounding in intelligent systems.

“Common ground” is the knowledge, facts, beliefs, etc. that are shared between participants in some joint activity. Much of human conversation concerns “grounding,” or ensuring that some assertion is actually shared between participants. Even for highly trained tasks, such teammates executing a military mission, each participant devotes attention to contributing new assertions, making adjustments based on the statements of others, offering potential repairs to resolve potential discrepancies in the common ground and so forth.In conversational interactions between humans and machines (or “agents”), this activity to build and to maintain a common ground is typically one-sided and fixed. It is one-sided because the human must do almost all the work of creating substantive common ground in the interaction. It is fixed because the agent does not adapt its understanding to what the human knows, prefers, and expects. Instead, the human must adapt to the agent. These limitations create burdensome cognitive demand, result in frustration and distrust in automation, and make the notion of an agent “teammate” seem an ambition far from reachable in today’s state-of-art. We are seeking to enable agents to more fully partner in building and maintaining common ground as well as to enable them to adapt their understanding of a joint activity. While “common ground” is often called out as a gap in human-machine teaming, there is not an extant, detailed analysis of the components of common ground and a mapping of these components to specific classes of functions (what specific agent capabilities is required to achieve common ground?) and deficits (what kinds of errors may arise when the functions are insufficient for a particular component of the common ground?). In this paper, we provide such an analysis, focusing on the requirements for human-machine teaming in a military context where interactions are task-oriented and generally well-trained.Drawing on the literature of human communication, we identify the components of information included in common ground. We identify three main axes: the temporal dimension of common ground and personal and communal common ground. The analysis further subdivides these distinctions, differentiating between aspects of the common ground such as personal history between participants, norms and expectations based on those norms, and the extent to which actions taken by participants in a human-machine interaction context are “public” events or not. Within each dimension, we also provide examples of specific issues that may arise due to problems due to lack of common ground related to a specific dimension. The analysis thus defines, at a more granular level than existing analyses, how specific categories of deficits in shared knowledge or processing differences manifests in misalignment in shared understanding. The paper both identifies specific challenges and prioritizes them according to acuteness of need. In other words, not all of the gaps require immediate attention to improve human-machine interaction. Further, the solution to specific issues may sometimes depend on solutions to other issues. As a consequence, this analysis facilitates greater understanding of how to attack issues in misalignment in both the nearer- and longer-terms.

Language models (LLMs) offer potential as a source of knowledge for agents that need to acquire new task competencies within a performance environment. We describe efforts toward a novel agent capability that can construct cues (or "prompts") that result in useful LLM responses for an agent learning a new task. Importantly, responses must not only be "reasonable" (a measure used commonly in research on knowledge extraction from LLMs) but also specific to the agent's task context and in a form that the agent can interpret given its native language capacities. We summarize a series of empirical investigations of prompting strategies and evaluate responses against the goals of targeted and actionable responses for task learning. Our results demonstrate that actionable task knowledge can be obtained from LLMs in support of online agent task learning.

Autonomous agents are able to draw on a wide variety of potential sources of task knowledge; however current approaches invariably focus on only one or two. Here we investigate the challenges and impact of exploiting diverse knowledge sources to learn online, in one-shot, new tasks for a simulated office mobile robot. The resulting agent, developed in the Soar cognitive architecture, uses the following sources of domain and task knowledge: interaction with the environment, task execution and search knowledge, human natural language instruction, and responses retrieved from a large language model (GPT-3). We explore the distinct contributions of these knowledge sources and evaluate the performance of different combinations in terms of learning correct task knowledge and human workload. Results show that an agent's online integration of diverse knowledge sources improves one-shot task learning overall, reducing human feedback needed for rapid and reliable task learning.

Learning systems that adapt to learner capabilities, needs, and preferences have been shown to improve learning outcomes. However, creating systems that can interpret learner state within the context of a dynamic learning environment is costly and often tied to the specific requirements of the learning environment. We overview a new approach for monitoring and assessing system context and learner state that is not specific to a particular domain. The process is designed to transform diverse, continuous, and multichannel streams of heterogeneous system data into a consistent, discrete, and learner-centric interpretation of the situation. Key steps in the process include discretizing the data stream into “events” and then marking some events as “keyframes” that identify important steps or changes in the learning state. This keyframing process provides a compact representation for use by learner-adaptive processes (including assessment and tailoring) and simplifies the challenges of using machine learning as a mechanism for adaptation.

Today, various actors are exploiting and misusing online social media to spread disinformation and to create false narratives. This paper summarizes an education and training approach targeted to help people think more critically about potential disinformation. The approach we outline emphasizes the development and maturation of general critical-thinking skills, in contrast to technical skills (e.g., social network analysis). However, it also offers opportunity to apply these skills in a scaffolded, adaptive environment that supports the learner in putting concepts into use. The approach draws on the situated-learning paradigm to support skill development and reflects empirically-based best practices for pedagogy for critical thinking. This analysis and review provides context to inform the design of a learning environment to enable targeted practice of critical thinking skills. The paper outlines the high-level design, describes several specific “experiential lessons” and overviews a few technical challenges that remain to be overcome to make the training feasible for wide-scale use.

Language models (LMs) are sentence-completion engines trained on massive corpora. LMs have emerged as a significant breakthrough in natural-language processing, providing capabilities that go far beyond sentence completion including question answering, summarization, and natural-language inference. While many of these capabilities have potential application to cognitive systems, exploiting language models as a source of task knowledge, especially for task learning, offers significant, near-term benefits. We introduce language models and the various tasks to which they have been applied and then review methods of knowledge extraction from language models. The resulting analysis outlines both the challenges and opportunities for using language models as a new knowledge source for cognitive systems. It also identifies possible ways to improve knowledge extraction from language models using the capabilities provided by cognitive systems. Central to success will be the ability of a cognitive agent to itself learn an abstract model of the knowledge implicit in the LM as well as methods to extract high-quality knowledge effectively and efficiently. To illustrate, we introduce a hypothetical robot agent and describe how language models could extend its task knowledge and improve its performance and the kinds of knowledge and methods the agent can use to exploit the knowledge within a language model.

This paper describes the characteristics, design and architecture of a learner model, the GTRI Learner Assessment Engine (GLAsE), that is designed to operate within a practice environment for teaching the analysis of social media feeds. The purpose of GLAsE in the practice environment is to provide a state of the learner’s competency or proficiency for the information and use by the learner and instructor to guide the learning activities, e.g., the experience of the learner in the practice environment. The Learner Model is represented as a curriculum overlay model with additional annotations and background information. The contents of the Learner Model consist of items represented in the Curriculum Model as learning objectives: Concepts, Skills, and Problem-solving approaches. This Learner Model is called an overlay model because the Learner Model has the same representation as the expert domain knowledge, i.e., the Curriculum Model—it is overlaid on that representation. The Learner Model will provide mastery or proficiency scores for each of the learning objectives that represent the model’s best estimate of the state of the learner proficiency for that objective. The learner assessment process will identify and prioritize any concepts in which the learner has deficiency and will, whenever it is appropriate in a learning or practice system, provide information to other system components that could be used to direct the learner to particular activities to help increase proficiency in those concepts. Results of the learner assessment will be used to update the Learner Model.

The past several years have seen a strong push toward using Deep Learning systems–Neural Networks with multiple hidden layers. Deep Learning is now used in many machine learning applications and provides leading performance on numerous benchmark tasks. However, this increase in performance requires very large datasets for training. From a practitioner prospective, the model that performs best in benchmark tasks may be too data intensive to be adapted to practical application. We describe a behavior recognition problem that was solved using a sequence-based Deep Learning system and then reimplemented using a more knowledge-driven sequence matching approach due to data constraints. We contrast the two approaches and the data required to achieve sufficient performance and flexibility.