Multi-Still: A lightweight Multi-modal Cross Attention Knowledge Distillation method for the Real-Time Emotion Recognition Service in Edge-to-Cloud Continuum

Recent advances in big data and artificial intelligence have led to active research in emotion recognition based on multimodal transformer models. Although these multimodal transformer models demonstrate high performance, their applications into real-time services are challenging due to their heavy computational requirements. Therefore, this study proposes a Multi-Still method, which transfers the multimodal knowledge of a teacher model to a student model using the knowledge distillation method supporting edge to cloud continuum environment. Multi-Still trained by text and voice data from Korean multimodal emotional datasets (KEMDy19, KEMDy20) both teacher and student. As a result, the student model transferred knowledge from the teacher model showed a 21% increase in number of inferences per second compared to the teacher model, 70.31% reduction in network size, and 65% reduction in the number of parameters. Nevertheless, it shows similar accuracy to the teacher model. We provide real-time emotion recognition services for the lightweight resources in edge continuum by efficiently learning multimodal data through knowledge distillation.