Multi-modal Sentiment Analysis Using Text and Audio for Customer Support Centers

Customer service has become crucial for every organization looking to grow and improve its clientele in today’s cutthroat marketplace. Companies cannot afford to fall short of consumer expectations. With the recent progress in Artificial Intelligence, companies have adopted AI techniques like sentiment analysis to measure customer satisfaction. Chatbots are the foundation for most AI applications in call centers trained for either question-answering tasks or calculating sentiment out of user feedback via surveys. These existing expert systems only utilize text mining techniques to classify sentiment as positive or negative. In this paper, we propose a Multimodal learning framework for tackling the sentiment classification task, employing acoustic and linguistic modalities from real-world conversations between support representatives and customers and survey feedback data using the decision-level fusion technique. Leveraging the classification abilities and feature representations from both modalities, our model achieves the best results amongst all implementations and enables us to better analyze the sentiment expressed by the user than using a single modality like text or audio.