SHAD: Privacy-Friendly Shared Activity Detection and Data Sharing

Nowadays, there is a growing demand for sharing multimedia data among participants in the same activity. With existing social applications, users need to conduct friending and data sharing operations manually, which is troublesome due to changing attendees and highly diverse data content of different activities. To tackle this issue, in this work we propose a novel system SHAD to achieve privacy-friendly shared activity detection and multimedia data auto-sharing based on users' historical multimodal data. Facing noisy, incomplete and asynchronous data, as well as inaccurate recognition results of machine learning models, we design an algorithm to aggregate multimodal data relevant to the same activity and propose an activity-semantic graph to comprehensively characterize each activity by fusing knowledge of multimodal data. Based on the activity-semantic graph, the privacy-preserving shared activity detection and data sharing method is designed, which protects both raw data and semantic information of data. We implemented our system and conducted comprehensive evaluations with real-life multimodal data (including photos and motion sensor data). The results show the efficacy of our system. We can achieve 94.9% precision and 91.5% recall for shared activity detection.