Validation of Social Skills Training System: Four-Week Randomized Controlled Pilot Trial (Preprint)

BACKGROUND Social skills training by human trainers is a well-established method of teaching appropriate social and communication skills and strengthening social self-efficacy. Specifically, human social skills training is a fundamental approach to teaching and learning the rules of social interaction. However, it is cost-ineffective and offers low accessibility, since professional trainers are limited. In our previous work, we attempted to automate social skills training by developing a conversational agent that taught social skills through interaction. Long-term validation of such an automated training system is needed. OBJECTIVE This study aims to validate the training effect of a social skills training system during a four-week randomized controlled pilot trial. We compare two groups (with and without training) and hypothesize that trained people will improve their social skills and overcome their social anxiety to a greater extent than the non-trained group. Furthermore, this study seeks to clarify the optimal sample size and effect size for future larger-scale evaluations. METHODS For the study, 26 Japanese participants are separated into the two groups, where we hypothesize that Group 1 (with training) will make greater improvement than Group 2 (without training). In addition to questionnaires, we conduct a Role-play Performance Test that requires the social cognition and expression of participants in new role-play scenarios. Blind ratings by third-party trainers are made by watching recorded role-play videos. A non-parametric Wilcoxson rank sum test is performed for each variable. Improvement between pre- and post-training evaluations are used to compare the two groups. Moreover, we compare the statistical significance from the questionnaires and ratings between the two groups. RESULTS Out of 26 recruited participants, 18 individuals completed this experiment: 9 people in Group 1 and 9 in Group 2. Those in Group 1 achieved significant improvement in generalized self-efficacy (P=0.02). We also found a significant decrease in state anxiety presence (P=0.04) measured by the State-Trait Anxiety Inventory. For ratings by third-party trainers, clarity of speech was significantly strengthened in Group 1 (P=0.03). CONCLUSIONS Our findings reveal the usefulness of the social skills training system after a one-month training period. This study confirms a large effect size between groups, thus indicating the need for further larger-scale evaluations.