As human-agent teaming (HAT) research continues to grow, computational methods for modeling HAT behaviors and measuring HAT effectiveness also continue to develop. One rising method involves the use of human digital twins (HDT) to approximate human behaviors and socio-emotional-cognitive reactions to AI-driven agent team members. To help HDT research effectively model human trust in HATs, we offer two lines of insight. First, through a review of the HAT trust literature, we identify key characteristics and attributes of trust that must be considered in order to properly conceptualize, model, and measure trust. Through this review, we outline the theoretical foundations of trust needed for effective HDTs capable of emulating human trust and offer guidance on where and how extant HAT research should translate into HDT modeling and future research. Second, through causal analyses of archival team communication data from a HAT experiment, we supplement theoretical foundations for modeling trust with data-driven insights to guide the trust-related language HDTs may need to effectively emulate human trust. Finally, we discuss implications of these combined theoretical and empirical insights for future HDT research, highlighting the necessity of ongoing validation against human behaviors and the refinement of computational methods. This paper ultimately aims to advance both the fidelity and applicability of HDTs in modeling nuanced human-agent trust dynamics, fostering more effective and realistic human-agent collaborations.

The study of human-artificial intelligence (AI) teaming (HAT) and Human Digital Twin (HDT) modeling currently faces significant challenges in accurately simulating and assessing the effectiveness of interactions between humans and AI systems. Current methods typically rely on limited real-world data or simplified simulated representations that do not capture the complexity and variability of human digital twin and AI agent behavior. VirTLab-Eval is a novel agentic framework of human digital twins (HDTs) and AI agents to model and evaluate HAT behaviors and interventions across operational scenarios. A comprehensive set of measures, including novel socio-cognitive-emotional and behavioral metrics automatically extracted from team communications and interactions, for evaluating HAT performance is integrated into visual analytics enabling the assessment of interventions that target the attributes of HDTs and agents. VirTLab-Eval measurements capture nuanced aspects of trust development, emotional alignment, and team cohesion that conventional performance metrics might miss, providing richer insights into the HAT dynamics. Example results from search and rescue missions indicate that AI teammate reliability has a significant effect on communication dynamics and assistance behaviors, while Human Digital Twin (HDT) personality traits shape trust development and team coordination. These insights directly inform the design of Human-AI Teaming (HAT) training programs aimed at optimizing the use of AI systems, mitigating both over-reliance and under-utilization, and ultimately enhancing mission effectiveness while reducing risk to personnel.

Eating disorders are complex mental health conditions that affect millions of people around the world. Effective interventions on social media platforms are crucial, yet testing strategies in situ can be risky. We present a novel LLM-driven experimental testbed for simulating and assessing intervention strategies in ED-related discussions. Our framework generates synthetic conversations across multiple platforms, models, and ED-related topics, allowing for controlled experimentation with diverse intervention approaches. We analyze the impact of various intervention strategies on conversation dynamics across four dimensions: intervention type, generative model, social media platform, and ED-related community/topic. We employ cognitive domain analysis metrics, including sentiment, emotions, etc., to evaluate the effectiveness of interventions. Our findings reveal that civility-focused interventions consistently improve positive sentiment and emotional tone across all dimensions, while insight-resetting approaches tend to increase negative emotions. We also uncover significant biases in LLM-generated conversations, with cognitive metrics varying notably between models (Claude-3 Haiku > Mistral > GPT-3.5-turbo > LLaMA3) and even between versions of the same model. These variations highlight the importance of model selection in simulating realistic discussions related to ED. Our work provides valuable information on the complex dynamics of ED-related discussions and the effectiveness of various intervention strategies.

Large language models (LLMs) have garnered significant attention for their remarkable performance in a continuously expanding set of natural language processing tasks. However, these models have been shown to harbor inherent societal biases, or stereotypes, which can adversely affect their performance in their many downstream applications. In this paper, we introduce a novel, purely prompt-based approach to uncover hidden stereotypes within any arbitrary LLM. Our approach dynamically generates a knowledge representation of internal stereotypes, enabling the identification of biases encoded within the LLM's internal knowledge. By illuminating the biases present in LLMs and offering a systematic methodology for their analysis, our work contributes to advancing transparency and promoting fairness in natural language processing systems.

Warning: This paper discusses and contains content that is offensive or upsetting. Large Language Models (LLMs) have seen widespread deployment in various real-world applications. Understanding these biases is crucial to comprehend the potential downstream consequences when using LLMs to make decisions, particularly for historically disadvantaged groups. In this work, we propose a simple method for analyzing and comparing demographic bias in LLMs, through the lens of job recommendations. We demonstrate the effectiveness of our method by measuring intersectional biases within ChatGPT and LLaMA, two cutting-edge LLMs. Our experiments primarily focus on uncovering gender identity and nationality bias; however, our method can be extended to examine biases associated with any intersection of demographic identities. We identify distinct biases in both models toward various demographic identities, such as both models consistently suggesting low-paying jobs for Mexican workers or preferring to recommend secretarial roles to women. Our study highlights the importance of measuring the bias of LLMs in downstream applications to understand the potential for harm and inequitable outcomes. Our code is available at https://github.com/Abel2Code/Unequal-Opportunities-of-LLMs.

Large Language Models (LLMs) have seen widespread deployment in various real-world applications. Understanding these biases is crucial to comprehend the potential downstream consequences when using LLMs to make decisions, particularly for historically disadvantaged groups. In this work, we propose a simple method for analyzing and comparing demographic bias in LLMs, through the lens of job recommendations. We demonstrate the effectiveness of our method by measuring intersectional biases within ChatGPT and LLaMA, two cutting-edge LLMs. Our experiments primarily focus on uncovering gender identity and nationality bias; however, our method can be extended to examine biases associated with any intersection of demographic identities. We identify distinct biases in both models toward various demographic identities, such as both models consistently suggesting low-paying jobs for Mexican workers or preferring to recommend secretarial roles to women. Our study highlights the importance of measuring the bias of LLMs in downstream applications to understand the potential for harm and inequitable outcomes.

Despite the potential for antisocial and counterproductive social media behavior–particularly in the context of humanitarian assistance/disaster response (HA/DR)–there is a paucity of automated methods to address it. Current methods focus primarily on detecting hate speech and banning problematic content. We propose an alternative strategy of using automated counter speech to focus not just on moderating uncivil behavior, but also the promotion of civil discourse. In this paper, we propose a novel framework to employ pre-trained language models to alleviate the bottlenecks in adoption of such counter speech, namely a lack of understanding on the dynamics of counter speech and a scarcity of well curated datasets, which are compounded in HA/DR settings. We utilize GPT LMs to create a conversational testbed to simulate online conversations where various approaches for counter speech and other content moderation methods can be evaluated. Additionally, we leverage BERT-based models to detect hate speech and other network and syntactic features to suggest the optimal strategy to employ. We also present empirical results on the experiments we have conducted which provide a proof of concept for the framework.

Future hybrid human-machine teams will need to optimize their performance in uncertain environments by adapting their team structure. To address this need, we have developed a framework based on minimization of variational free energy, an information theoretic measure that has been shown to account for a variety of biological self-organizing phenomena. This paper proposes a novel approach to balance team structure by adapting roles and relationships based upon this framework. We apply this approach to evaluate possible structures for an infantry squad of human soldiers and autonomous systems. Using our STATES team simulation environment, we simulate mission performance for these teams and demonstrate that this approach enables a 12-person team to achieve performance results on par with a 15-person traditional team in terms of mission execution time. We argue that these results indicate that the free energy approach will lead to better hybrid team adaptations and improved performance.

Developing models to extract highest potential from human-machine teaming is the defense technology of the future. In this paper, we present a continuation of our work on adaptive self-organizing teams using the active inference, a model based on iterative minimization of free variational energy encoding task performance and team structure complexity. We describe how roles of agents are (re)defined by decomposing primitive decisions into groups over time using variational clustering on the factor graph, which specifies the contribution of local decisions and their dependencies on the team's objective function. We illustrate how uncertainty of knowledge the agents have about each other affect the distributed decisions, and propose the methods for updating this knowledge and incorporating its uncertainty to improve team performance.

The Internet of things (IoT), from heart monitoring implants to home-heating control systems, is becoming integral to our daily lives. We expect the technologies that comprise IoT to become smarter; to autonomously reason, act, and communicate with other entities in the environment; and to achieve shared goals. To realize the full potential of these systems, we must understand the mechanisms that allow multiple intelligent entities to effectively operate, collaborate, and learn in changing and uncertain environments. The future IoT devices must not only maintain enough intelligence to perceive and act locally, but also possess team-level collaboration and adaptation processes. We posit that such processes embody energy-minimizing mechanisms found in all biological and physical systems, and operate over the objectives and constraints that can be defined and analyzed locally by individual devices without the need for global centralized control. In this chapter, we represent multiple IoT devices as a team of intelligent agents, and postulate that multiagent systems achieve adaptive behaviors by minimizing a team’s free energy, which decomposes into distributed iterative perception (inference) and control (action) processes. First, we discuss instantiation of this mechanism for a joint distributed decision-making problem. Next, we present experimental evidence that energy-based teams outperform utility-based teams. Finally, we discuss different learning processes that support team-level adaptation.

Military planners and decision-makers face a number of challenges with the shift towards operating within diverse, multi-dimensional, and unconventional environments. Leaders require a deeper understanding of the broader social and civil context in which operations occur, including the underlying factors that contribute to instability and the drivers of conflict. This understanding is often derived through the analysis of textual data. Both traditional and non-traditional sources – such as news articles, blog entries, and tweets – represent a vast amount of data that can be brought to bear on problems ranging from measuring the progress of missions to forecasting important changes in the environment. The volume and velocity of this data requires processing tools that can help users understand the concepts and events being discussed. Additionally, these data are inherently ambiguous, and the automated processing techniques necessary for aggregating and analyzing data may introduce further uncertainty. The validity and veracity of data, sources, assumptions, and conclusions must be carefully considered prior to action. Planners and decision-makers require new, data-driven tools that aid in selecting the best course of action through interactive exploration and assessment processes. In this paper, we describe an ongoing research and development effort to create a context-driven, web-based tool that aids planning and decision-making by providing a more comprehensive understanding of the civil component of operational environments. This tool allows users to rapidly find, organize, and assess complex data across multiple phases of civil information management. This includes: (1) researching and assessing civil vulnerabilities; (2) developing plans to address identified vulnerabilities; and (3) tracking ongoing trends and progress towards goals and objectives. Our tool assists the user in each phase by collecting, processing, and recommending data and analyses that are contextually relevant to their task. By offloading data collection, supporting data organization, and providing personalized recommendations, our tool allows users to focus their efforts on verifying, interpreting, and assessing the information needed to recommend the best course of action.

Messages often spread within a population through unofficial - particularly web-based - media. Such ideas have been termed “memes.” To impede the flow of terrorist messages and to promote counter messages within a population, intelligence analysts must understand how messages spread. We used statistical language processing technologies to operationalize “memes” as latent topics in electronic text and applied epidemiological techniques to describe and analyze patterns of message propagation. We developed our methods and applied them to English-language newspapers and blogs in the Arab world. We found that a relatively simple epidemiological model can reproduce some dynamics of observed empirical relationships.