Exploring The Dependencies Between Behavioral And Neuro-Physiological Time-Series Extracted From Conversations Between Humans And Artificial Agents

Whole-brain neuroimaging using functional Magnetic Resonance Imaging (fMRI) provides valuable data to localize brain activity in space and time. Here, we use a unique corpus including fMRI and behavior recorded when participants discussed with a human or a conversational robot. Temporal dynamic is crucial when studying conversation, yet identifying relationship between the participants' behavior and their brain activity is technically challenging given the time resolution of fMRI. We propose here an approach developed to extract neurophysiological and behavioral time-series from the corpus and analyse their causal relationships. Pre-processing entails the construction of discrete neurophysiological time-series from functionally well defined brain areas, as well as behavioral time-series describing higher-order behaviors extracted from synchronized raw audio, video and eyetracking recordings. The second step consists in applying machine learning models to predict brain activity on the basis of various aspects of behavior given knowledge about the functional role of the areas under scrutiny. Results demonstrate the specificity of the behaviors allowing the predictions of the activity in functional brain areas.