
This study reports on an evaluation of responses from 10 custom-developed chatbots to assess how chatbot answer evaluation might best be done for benchmarking and real-world environments. We asked the same 16 questions to 10 configurations of our chatbot and compared the quality of the responses provided with two key metrics: how well the chatbots’ responses scored on the Flesch-Kincaid readability test, and how well they held up to scrutiny from technical communication experts. We show how the Flesch-Kincaid readability test can be misleading and should not be considered as a comprehensive measurement when it comes to evaluating the output generated by AI models. We discuss the limitations of the Flesch-Kincaid readability test and preview our current work integrating the ISO 24495-1 Plain Language standards into the assessment of AI-driven chatbots.
As artificial intelligence (AI) becomes increasingly integrated in everyday decision-making, understanding how individuals respond to AI versus human advice is critical. The present study examined whether individuals are more likely to revise their initial decisions based on recommendations’ source (AI vs. Human) and stance (Agree vs. Disagree), while holding recommendation content constant. Fifty-nine participants completed eight game scenarios from popular casino games. After each initial choice, participants received a recommendation and were prompted to re-make the same decision. Mixed-effects logistic regression revealed that there is a strong tendency for participants to switch their selection when presented with disagreeing recommendations, regardless of its source. No significant differences were found in switching behaviors between AI and human-disagree conditions, though confidence increased in presence of decision switch when the opposing recommendation came from AI. Future research should explore how individual differences and contextual factors influence adoption of AI advice.
Increased use of Artificial Intelligence raises questions around mitigating unintended consequences. Technical perspectives on these questions lead to calls for “Responsible AI,” while regulatory perspectives lead to calls for increased human oversight. Neither perspective fully addresses the core issues surrounding accountability in the sociotechnical systems in which AI operates. Human oversight is a form of supervisory control, in which the role of the human is to monitor AI performance within defined constraints, and this requires consideration of who defines the constraints, where accountability lies when the constraints are exceeded, and how AI performance can be interpreted by human supervisors. For us, constraints correspond to system’s Values and a challenge lies in aligning system operations with these Values. We demonstrate the use of Work Domain Analysis to map purpose-related functions (i.e., activity performed in a sociotechnical system) to Values (weighted according to the functional purpose of the organization).
Effective team communication depends not only on what is said but also on whether each exchange advances task execution. This study developed an utterance-level progress metric to characterize communication as forward, backward, sideways, or pause and examined its relationship with team performance. Forty-eight participants completed a cooperative, time-constrained defusal task, yielding 95 cases. Speech acts and progress were coded from transcripts with strong interrater reliability for speech acts (κ = 0.83) and moderate reliability for progress (κ = 0.60). Good-performance cases showed more forward progress and less backward and sideways progress than poor-performance cases. Poor performance was also associated with a higher prevalence of information requests, while pause utterances were longer during good performance. Adding progress variables significantly improved a speech-act-only classification model, although cross-validated improvement was inconclusive. These findings suggest that speech acts and utterance-by-utterance progress provide complementary indicators of team coordination and performance.
This paper presents the Catalyst UX framework, a five-stage participatory design process developed to support the creation of highly technical Geographic Information System (GIS) platforms. GIS user experience can suffer from steep learning curves, fragmented workflows, and declining engagement once development shifts toward technical implementation. Catalyst UX addresses this challenge through sustained engagement with domain experts across workshops, a Technical Working Group (TWG), individual interviews, usability testing, and product launch activities. The framework was implemented during the development of a water data observatory integrating 21 datasets for hydrology and water resource management. This paper discusses qualitative and quantitative findings from a mixed-methods usability study ( n = 12) demonstrating a 93% task completion rate and a System Usability Scale (SUS) score of 60 for the minimum viable product. Results suggest that continuous expert engagement improved feature prioritization, reduced scope creep, and preserved alignment between system architecture and analytical workflows in GIS environments.