Federated Multimodal Learning for Privacy-Preserving Driver Break Recommendations in Consumer Electronics

In recent years, driver distraction behaviors, such as eating, drinking, and making phone calls, have become more and more frequent, especially during continuous driving. This results in a significant increase in traffic accidents caused by those behaviors. Therefore, it is crucial to recommend driver breaks when frequent instances of driver distraction are detected. Despite numerous single modal-based approaches, such as computer vision, were proposed for driver distraction detection, driver break recommendations still suffer from low accuracy and mandatory requirement of preinstalled cameras. To make the full use the power of ambient information including images, audios, and postures, we introduce Multimodal Learning (ML) to identify driver distraction. Our model is capable of utilizing information contained in images, audios, and postures through consumer electronic devices such as mobile phones. Moreover, transfer learning is utilized to take advantage of previously trained models, thereby significantly enhancing the accuracy. However, training an personalized model requires a substantial amount of data from entities and individuals which raises privacy concerns. Henceforth, we integrate Federated Learning into Multimodal Deep Learning framework to protect participants’ privacy while achieving better performance in recommending driver breaks. The experimental findings conclusively demonstrate that the proposed method not only outperforms other existing approaches in terms of providing effective driver break recommendations but also prioritizes the privacy of the individuals involved.