We study how conditioning context shapes personalization behavior in a teacher-facing educational recommender system. We compare contextual conditioning based on the current student question with memory-based conditioning using persistent learner information. Using deviation correlation and paired statistical tests, we find that contextual recommendations exhibit stronger question-level responsiveness, while memory-based recommendations exhibit history-dependent behaviors, including learner-specific differentiation under identical input. Teacher-facing evaluation signals suggest these recommendations are interpretable and actionable. These results indicate that embedding-based similarity metrics capture responsiveness to the current question but do not characterize personalization grounded in learner history, motivating behavior-level diagnostics for studying conditioning effects.

Intelligent tutoring systems excel at generating explanations but rarely provide principled diagnosis of where and why a learner is wrong. We introduce a misstep-aware coaching capability for Ivy, a neurosymbolic AI coach, built on a two-model architecture that augments a Task-Method-Knowledge (TMK) model with a new Pedagogical Model (PM) in the context of an online graduate AI course at Georgia Tech. The PM makes instructor diagnostic knowledge explicit and machine-readable by encoding, for each quiz question and incorrect response, the learner's underlying belief(a brief statement of the incorrect idea or missing knowledge), a TMK locus(the source of the misunderstanding), a misconception type and targeted scaffolding derived from the instructor's Q&A key. Using quiz questions from the course, we demonstrate a proof-of-concept pipeline that detects and classifies learner errors and generates diagnosis-grounded scaffolding, moving Ivy beyond knowledge retrieval toward diagnostic misstep awareness, and enabling more precise, actionable feedback that supports conceptual change and advances adaptive learning systems in AI in education and the learning sciences.

The application of AI in education has evolved from monolithic intelligent tutoring systems to a diverse ecosystem of pedagogical agents, including conversational assistants, virtual coaches, and adaptive tutors. This shift requires a unified and scalable data architecture to manage the complex information feedback loops between human instructors, learners, and the varied AI agents. The design, development, and deployment of the data architecture in turn raises a critical issue of validation. This paper addresses this critical need by describing a practical validation strategy for a high-volume data pipeline developed as part of a data architecture for AI-augmented adult learning at the National AI Institute for Adult Learning and Online Education. Our approach involves a two-stage testing methodology to ensure both functional diversity and real-world scalability. First, the QA environment uses a blend of synthetic and real-world data to validate functional correctness across various event types produced from learner and agent interactions. Following this, the production environment successfully processed a total of over 2.7 million production requests across 21 successful runs carrying authentic event data from a large-scale online program. This validation process surfaced crucial insights into data privacy, a key challenge when handling varied data from multiple AI agent data sources. By outlining a replicable testing strategy for a unified data backbone, this research offers a clear framework for institutions and developers aiming to build and support their own heterogeneous suites of AI-powered learning tools. Keywords: Pedagogical Agents, Learning Ecosystems, Data Architecture, Validation, Scalability, Learning Analytics.

The ability to correct mistakes and adapt to users' changing needs is critical for AI agents to remain robust and trustworthy. LLM-based agents are inherently prone to errors like hallucinations and misinterpretations. We observed this challenge in SAMI, an AI social agent deployed in Georgia Tech's OMSCS program for ten semesters (11,000+ users). Users frequently requested the agent to revise its knowledge base, both to correct LLM-induced errors and to update their information. To support such revisions, we introduce a two-level metacognitive self-adaptation architecture that integrates knowledge-based AI (KBAI) with LLMs. The architecture comprises a cognitive layer that performs the agent's core tasks, and a metacognitive layer that introspects on the cognitive layer's process using a Task–Method–Knowledge (TMK) model of the agent. The metacognitive layer identifies the task that needs revision, updates the knowledge base, and communicates the revision process to the user.

Learner representations play a central role in educational AI systems, yet it is often unclear whether they preserve meaningful differences between students when instructional outcomes are unavailable or highly context-dependent. This work examines how to evaluate learner representations based on whether they retain separation between learners under a shared comparison rule. We introduce distinctiveness, a representation-level measure that evaluates how each learner differs from others in the cohort using pairwise distances, without requiring clustering, labels, or task-specific evaluation. Using student-authored questions collected through a conversational AI agent in an online learning environment, we compare representations based on individual questions with representations that aggregate patterns across a student's interactions over time. Results show that learner-level representations yield higher separation, stronger clustering structure, and more reliable pairwise discrimination than interaction-level representations. These findings demonstrate that learner representations can be evaluated independently of instructional outcomes and provide a practical pre-deployment criterion using distinctiveness as a diagnostic metric for assessing whether a representation supports differentiated modeling or personalization.

Recent advances in AI are integrating AI into the fabric of human social life, creating transformative, co-shaping relationships between humans and AI. This trend makes it urgent to investigate how these systems, in turn, shape their users. We conducted a three-phase design study with 24 participants to explore this dynamic. Our findings reveal critical tensions: (1) social AI often exacerbates the very interpersonal problems it is designed to mitigate; (2) it introduces nuanced privacy harms for secondary users inadvertently involved in AI-mediated social interactions; and (3) it can threaten the primary user's personal agency and identity. We argue these tensions expose a problematic tendency in the user-centered paradigm, which often prioritizes immediate user experience at the expense of core human values like interpersonal ethics and self-efficacy. We call for a paradigm shift toward a more provocative and relational design perspective that foregrounds long-term social and personal consequences.

Scalable AI tutoring for procedural skill learning requires structured knowledge representations, yet constructing these representations remains a labor-intensive bottleneck. This paper presents a human-in-the-loop text-to-model pipeline that uses large language models to transform instructional materials into schema-complete Task-Method-Knowledge models of procedural skills through ontology-constrained prompting and template-based generation. The approach automates structural scaffolding while preserving expert oversight for validating causal transitions and failure conditions. We apply the pipeline to instructional materials from a graduate-level online AI course, constructing 23 procedural skill models. AI-assisted authoring reduced expert modeling time by 50-70

Multimodal Large Language Models (MLLMs) offer an opportunity to support multimedia learning through conversational systems grounded in educational content. However, while conversational AI is known to boost engagement, its impact on learning in visually-rich STEM domains remains under-explored. Moreover, there is limited understanding of how multimodality and conversationality jointly influence learning in generative AI systems. This work reports findings from a randomized controlled online study (N = 124) comparing three approaches to learning biology from textbook content: (1) a document-grounded conversational AI with interleaved text-and-image responses (MuDoC), (2) a document-grounded conversational AI with text-only responses (TexDoC), and (3) a textbook interface with semantic search and highlighting (DocSearch). Learners using MuDoC achieved the highest post-test scores and reported the most positive learning experience. Notably, while TexDoC was rated as significantly more engaging and easier to use than DocSearch, it led to the lowest post-test scores, revealing a disconnect between student perceptions and learning outcomes. Interpreted through the lens of the Cognitive Load Theory, these findings suggest that conversationality reduces extraneous load, while visual-verbal integration induced by multimodality increases germane load, leading to better learning outcomes. When conversationality is not complemented by multimodality, reduced cognitive effort may instead inflate perceived understanding without improving learning outcomes.

Evaluating procedural reasoning in AI-supported learning systems requires question-answer datasets that are both learner-like and grounded in the instructional knowledge the system is expected to use. We study how TMK-based question generation strategies affect dataset quality for procedural and multi-hop reasoning. We compare three strategies: strict generation from Task-Method-Knowledge (TMK) models, transcript-first generation with post-hoc TMK filtering, and TMK-aware generation that combines transcripts with structured guidance. To evaluate generated items, we introduce a grounding validation framework based on closed-set evidence units extracted from TMK models. The framework measures whether answers are supported by the underlying representation, whether questions are self-contained, and whether they target multi-hop procedural reasoning. Across 23 instructional topics and 690 generated question-answer pairs, strict TMK generation achieves the strongest overall quality, with 96.5

AI-augmented classrooms generate rich teacher and student feedback before graded outcomes become available, yet these signals can be difficult to translate into timely instructional decisions. We propose an interpretable decision layer: a transparent mechanism that ranks course topics requiring attention without using grades or post-hoc outcome labels. The approach combines three signals: student learning difficulty prevalence, disagreement between learner self-reports and observed difficulties, and unresolved teacher concerns. The output is a ranked set of topic priorities with per-topic decision records explaining each ranking. In one graduate CS course offering (n=5 instructor interviews; n=279 survey responses), prioritized topics aligned with instructor concerns (top-5 overlap 3/5; Spearman ρ=0.80) and student-reported topic difficulty (ρ=0.46, p=.048). Multi-signal integration also surfaced learners not identified through individual signal sources alone (AUC =0.96 vs. 0.91 for gap prevalence alone). Reflective thinking, help-seeking, and self-efficacy provided additional evidence that student behavioral signals align with learning-related constructs. While preliminary, these findings suggest that transparent coordination mechanisms may help support human-AI co-agency when feedback is incomplete.

Centering humans in explainable artificial intelligence (XAI) research has primarily focused on AI model development and high-stake scenarios. However, as AI becomes increasingly integrated into everyday applications in often opaque ways, the need for explainability tailored to end-users has grown more urgent. To address this gap, we explore end-users' perspectives on embedding XAI into daily AI application scenarios. Our findings reveal that XAI is not naturally accepted by end-users in their daily lives. When users seek explanations, they envision XAI design that promotes contextualized understanding, empowers adoption and adaption to AI systems, and considers multistakeholders' values. We further discuss supporting users' agency in XAI non-use and alternatives to XAI for managing ambiguity in AI interactions. Additionally, we provide design implications for XAI design at personal and societal levels. These include understanding users through a computational rationality lens, adaptive design that coevolves with users, and advancing the "society-in-the-loop" vision with everyday

AI increasingly mediates human social life—from daily communication and romantic partnerships to even "resurrecting" deceased loved ones. As AI’s influence on shaping human relationships and generative AI’s (GenAI) social capabilities grows, opportunities and challenges arise. In this work, we outline three roles AI could play in human-AI social collaboration and three levels of AI capabilities needed to perform various social roles effectively. We also discuss the measurement challenges and ethical risks involved. We aim to spark dialogue on designing sociotechnical systems that prioritize human social flourishing by examining AI’s role in mediating, simulating, and reshaping human social interaction.

Modeling and simulations offer significant benefits for self-directed learning as learners can independently design experiments and investigate their own hypotheses at their discretion. However, many studies on the use of such tools focus on pedagogical contexts in K-12 education with well-defined problems, learning goals, assessments, and outcomes. This study investigates how online learners engage in self-directed modeling by analyzing the behaviors of 315 learners across 822 models within VERA, an ecological modeling tool. Through learning analytics techniques, including activity sequence analysis, hierarchical clustering, and Markov chain models, we identify three distinct behavioral patterns-Observation, Construction, and Exploration-that reflect varying levels of engagement and interaction. Our findings reveal that learners transition from construction-focused behaviors to more active, hypothesis-driven exploration, with observation consistently present across all learning phases. This research contributes to the fields of AI in education and self-directed learning by providing a framework for understanding how learners interact with modeling tools outside traditional classroom settings. Future work may build on these patterns to offer adaptive and personalized learning.

Supporting learners' understanding of taught skills in online settings is a longstanding challenge. While exercises and chat-based agents can evaluate understanding in limited contexts, this challenge is magnified when learners seek explanations that delve into procedural knowledge (how things are done) and reasoning (why things happen). We hypothesize that an intelligent agent's ability to understand and explain learners' questions about skills can be significantly enhanced using the TMK (Task-Method-Knowledge) model, a Knowledge-based AI framework. We introduce Ivy, an intelligent agent that leverages an LLM and iterative refinement techniques to generate explanations that embody teleological, causal, and compositional principles. Our initial evaluation demonstrates that this approach goes beyond the typical shallow responses produced by an agent with access to unstructured text, thereby substantially improving the depth and relevance of feedback. This can potentially ensure learners develop a comprehensive understanding of skills crucial for effective problem-solving in online environments.

The integration of artificial intelligence (AI) in education holds significant promise for transforming personalized learning. By analyzing student learning data, AI systems can adapt instruction to meet individual needs through tailored content, adaptive learning paths, real-time feedback, and continuous improvement loops. However, effective personalization at scale demands not only access to large volumes of learner data but also robust data architectures to collect, organize, standardize, and analyze that data in a secure and meaningful way. However, note that the ability of AI to personalize learning requires data about the learner and prior learning. Personalization at scale requires data at scale. The Architecture for AI-Augmented Learning (A4L) frame-work addresses these needs by establishing a comprehensive data pipeline that supports AI-driven personalization. This pipeline introduced capabilities for direct data ingestion, anonymization, and standardization, as well as integrated analytics and visualization pipelines to deliver actionable insights to educators and learners alike.

The rapid proliferation of Explainable Artificial Intelligence (XAI) has led to the development of numerous evaluation frameworks aimed at guiding and optimizing its design. However, these frameworks often emphasize the technical properties of XAI artifacts, overlooking the nuanced perceptions and values of end-users. Recognizing that XAI impacts society and individuals in non-neutral ways, this study adopts a human-centered approach to systematically examine the effects of recommended XAI properties on the general public in everyday scenarios through a formative study involving 87 end-users. The findings reveal that comprehensibility is the most valued XAI property, while frequently advocated properties like contrastivity may have overall negative effects. These results highlight the necessity of a goal-driven reverse engineering approach that integrates human values into XAI design to ensure positive user outcomes. This paper bridges the gap between XAI design and its impact on end-users, offering practical guidance for user-centered XAI development in everyday contexts.

Multimodal AI is an important step towards building effective tools to leverage multiple modalities in human-AI communication. Building a multimodal document-grounded AI system to interact with long documents remains a challenge. Our work aims to fill the research gap of directly leveraging grounded visuals from documents alongside textual content in documents for response generation. We present an interactive conversational AI agent 'MuDoC' based on GPT-4o to generate document-grounded responses with interleaved text and figures. MuDoC's intelligent textbook interface promotes trustworthiness and enables verification of system responses by allowing instant navigation to source text and figures in the documents. We also discuss qualitative observations based on MuDoC responses highlighting its strengths and limitations.