As large language models (LLMs) are increasingly taken up in professional settings, understanding how workers adopt and develop LLM-related competence is essential. While LLMs offer the potential for enhanced efficiency, they also come with several risks, related to e.g., misinformation and bias. To enable the efficient and responsible uptake of LLMs in organizations, there is a need to better understand how generative AI tools are taken up in practice. Addressing this need, we draw on questionnaire data from 138 bank employees following their access to an internally deployed GPT tool, to explore how workers take LLMs into use and develop competence for effective utilization. Specifically, we examine users’ experiences with task identification and prompting. Our case study shows that LLM adoption is largely characterized by learning-by-doing, with workers identifying use cases and refining prompt strategies through experimentation rather than structured training. The results indicate the need for targeted organizational strategies, including accessible training and knowledge-sharing initiatives, to support broad, equitable and responsible uptake of LLMs in the workplace. The work contributes to the growing body of research on human-AI interaction in organizational contexts.
The recent rise of Large Language Model (LLM)-driven chatbot architectures has opened a new pathway for political chatbots, which hold the potential to shape users’ political knowledge, interest, and engagement. Yet their flexibility also raises risks: biased, misleading, or hallucinatory outputs can erode trust and harm usability. While human evaluation remains the gold standard, automatic approaches are increasingly needed to provide scalable, replicable, and cost-effective assessments of real-world applications. However, existing methods are not tailored to the unique challenges of the political domain. This study proposes the Political Chatbot Evaluation Metric (PCEM), a novel multi-dimensional framework for systematically evaluating political chatbots. PCEM incorporates six weighted dimensions—accuracy, source reliance, sentiment, neutrality, offensiveness, and readability—allowing researchers to adapt evaluations to specific priorities. The framework is tested using 261 queries from a recent political chatbot study, with PCEM scores compared against expert human evaluations. Results show strong alignment overall, validating most underlying automatic measures such as BERT-based models and ROUGE, while also highlighting the need for further refinement of LLM-Judges and classifiers. By offering this tailored and scalable evaluation toolkit for political chatbots, PCEM enables systematic benchmarking, comparison, and continuous monitoring. While not a replacement for human evaluation, it provides a transparent and replicable complement, fostering the development of trustworthy and safe political chatbots for public use.
Despite the rise of large language models (LLMs) and their general-purpose conversational capabilities, human involvement continues to play a critical role in designing conversational AI interactions. We examine how human-AI partnerships reshape conversational experience design (CxD), using expert interviews to see how practices adapt when LLMs enter the dialogue designer’s toolkit. To ground the interviews, we created a custom context-engineered GPT bot that assists conversation designers. Our findings indicate that LLM collaboration often accelerates early ideation, shifts effort from exhaustive scripting to editorial judgment and guardrails, and pushes CxD toward end-to-end orchestration. We discuss implications for skills, workflows, and tools as the profession evolves in the era of LLMs and propose the 3-O Framework with a design checklist for CxD.
The usage of chatbots based on Artificial Intelligence (AI) and Large Language Models (LLMs) is exponentially increasing among students. This younger generation is also known to show symptoms of poor mental well-being. Mental well-being can be influenced by different factors and it fluctuates over time. Because of its complexity, a Student Mental Well-Being (StMWB) chatbot could be a potential intervention targeting students and supporting them at the right moment and in the right way. However, to develop such a chatbot in an evidence-based manner, the needs, preferences, and requirements of students must be explored first. This paper will focus on the student counsellors’ perspective, interviewing 14 student counsellors from various universities. From the analysis, six themes emerged: needs and characteristics of university students, expectations from the StMWB chatbot, features of the StMWB chatbot, behavior of the StMWB chatbot, trust in the StMWB chatbot, and engaging with the StMWB chatbot. From these themes, the student counsellors’ perspective on the needs of students towards their mental well-being and a supporting StMWB chatbot was identified. Moreover, students’ preferences - according to the student counsellors - for the StWMB chatbot were identified. Linked to these needs and preferences, different requirements for the chatbot were formulated. The results of this study can contribute to the development of a StMWB chatbot. Finally, considering the student counsellors’ perspective can facilitate the implementation of the StMWB chatbot in university.
We present FairBot-Live, an interactive demonstrator for exploring ethical challenges of large language model (LLM)-based chatbots in e-commerce. The system enables users to simulate advisory dialogues with state-of-the-art LLMs and to evaluate the generated responses along the metrics bias, personalization, and toxicity. The evaluation is performed using the LLM-as-a-Judge approach, where one LLM assesses the outputs of another, providing both numerical scores and textual justifications.FairBot-Live provides a reproducible and transparent environment for investigating ethical evaluation methods and model behavior in realistic e-commerce interactions.
