Modelling Multi-Channel Emotions Using Facial Expression and Trajectory Cues for Improving Socially-Aware Robot Navigation

Using facial expressions and trajectory signals, we present an emotion-aware navigation algorithm for social robots. Our approach uses a combination of Bayesian-inference, CNN-based learning and the Pleasure-Arousal-Dominance model from psychology to estimate time-varying emotional behaviors of pedestrians from their faces and trajectories. For each pedestrian, these PAD characteristics are used to generate proxemic constraints. We use a multi-channel model to classify pedestrian features into four categories of emotions (happy, sad, angry, neutral). We observe an emotional detection accuracy of 85.33% in our validation results. In low-to medium-density environments, we formulate emotion-based proxemic constraints to perform socially conscious robot navigation. With Pepper, a social humanoid robot, we demonstrate the benefits of our algorithm in simulated environments with tens of pedestrians as well as in a real world setting.