New Approach to Estimating Mental Health Score Using a Communication Agent

It is important for monitoring systems for the elderly to estimate the state of the user during an interaction. In the field of elderly welfare, quality of life (QOL) is a useful indicator for comprehensive treatment of physical suffering as well as mental health and social wellbeing. In this study, we propose a new approach to QOL estimation that integrates facial expressions, head fluctuations, and eye movements captured during interactions with a communication agent. To this end, we implemented a communication agent and constructed a database based on the information collected through interpersonal experiments with 14 participants. In addition, we implemented a multimodal learning estimator and performed training with facial expressions, head fluctuation, and gaze feature extraction. Our results show that integration learning realized with all of the features integrated provided less error than single-modal learning based on facial expression alone. From our experimental results, we concluded that the proposed system can be used adequately as a QOL estimator.