Identifying Causal Relationships Between Behavior and Local Brain Activity During Natural Conversation.
INTERSPEECH(2020)
摘要
Characterizing precisely neurophysiological activity involved in natural conversations remains a major challenge. We explore in this paper the relationship between multimodal conversational behavior and brain activity during natural conversations. This is challenging due to Functional Magnetic Resonance Imaging (fMRI) time resolution and to the diversity of the recorded multimodal signals.\r\nWe use a unique corpus including localized brain activity and behavior recorded during a fMRI experiment when several participants had natural conversations alternatively with a human and a conversational robot. \r\nThe corpus includes fMRI responses as well as conversational signals that consist of synchronized raw audio and their transcripts, video and eye-tracking recordings.\r\nThe proposed approach includes a first step to extract discrete neurophysiological time-series from functionally well defined brain areas, as well as behavioral time-series describing specific behaviors. Then, machine learning models are applied to predict neurophysiological time-series based on the extracted behavioral features.\r\nThe results show promising prediction scores, and specific causal relationships are found between behaviors and the activity in functional brain areas for both conditions, i.e., human-human and human-robot conversations.
更多查看译文
关键词
multimodal signals processing, natural conversation, machine learning, human-human and human-machine interactions, Functional MRI
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络