Comparing LLM and Human Expert Responses to Problem Gambling Questions

Large language models (LLMs) are being increasingly used for mental health and informational support, yet little is known about their effectiveness in addressing problem gambling inquiries. This study evaluates how two general-purpose LLMs—OpenAI’s GPT-4o and Meta’s Llama 3.1 (405B)—respond to nine problem gambling questions constructed based on the Problem Gambling Severity Index (PGSI). We collected responses by prompting each LLM via its respective chatbot interface (ChatGPT-4o and Meta AI) and recruited professional gambling counselors (n = 23) to provide their own responses via an online survey. We asked counselors which chatbot responses they preferred and whether exposure to chatbot responses influenced their willingness to alter their answers. We compared LLM and human responses by analyzing several linguistic and readability metrics. Our results reveal that LLMs generate more verbose responses and that counselors prefer Llama’s responses over GPT’s. Most counselors reported that they would not change their own responses after reviewing the LLM-generated responses. This preliminary comparison highlights important considerations for integrating LLM-based tools into gambling harm prevention strategies by comparing their responses to those of experts with extensive experience in providing in-person treatment for gambling problems.