Embodied salience for gaze analysis in ecologically valid environments

The brain is a dynamical system, mapping sensory inputs to motor actions. This relationship has been widely characterised by reductionist controlled lab experiments. However, with the emergence of mobile eye-tracking, increasing emphasis has been placed on the ecological validity of gaze studies, taking them out of the lab and into the “wild” (Hayhoe & Ballard, 2005; Kingstone et al., 2003; Land & Tatler, 2009). Here we build on this by capturing, rather than constraining, sensory inputs and motor outputs in natural behaviour. We record 90% of sensory inputs using head mounted eye-tracking, scene camera and microphone. Simultaneously, recording 95% of skeletal motor outputs by motion tracking 51 degrees of freedom in the body and a total of 40 degrees of freedom in the hands. All tracking equipment is markerless and thus allows unconstrained behavioural monitoring “in the wild”. The eye-tracker data is processed post-hoc to give 3D gaze position relative to the subjects’ head and limb endpoints using our GT3D decoding method (Abbott & Faisal, 2011; Abbott & Faisal, 2012) and the motion capture data. This enables us to evaluate classical relationships in ecologically valid environments including 3 daily scenarios: breakfast in the kitchen, evening activities in the home and in-door ambulation. We find that the classical categorisation of gaze data to saccades and fixations is insufficient to capture eye-movement repertoire in natural behaviour. We spend the majority of daily life making smooth eye-movements directly coupled to body movements (eg VOR). Classically, allocation of gaze has been attributed to both bottom up scene salience and top down ongoing task demands. We propose a new method for analysing and interpreting eye-movements in the wild, by relating them directly to body posture. Thus, the interpretation of gaze and attention is integrated, not disembodied, from the kinematics of motor behaviour. Meeting abstract presented at VSS 2015