Early effects of word surprisal on pupil size during reading.

Early effects of word surprisal on pupil size during reading Stefan L. Frank (s.frank@ucl.ac.uk) Department of Cognitive, Perceptual and Brain Sciences University College London 26 Bedford Way, London WC1H 0AP, United Kingdom Robin L. Thompson (robin.thompson@ucl.ac.uk) Deafness, Cognition and Language Research Centre Department of Cognitive, Perceptual and Brain Sciences University College London 49 Gordon Square, London WC1H 0PD, United Kingdom Abstract This study investigated the relation between word surprisal and pupil dilation during reading. Participants’ eye movements and pupil size were recorded while they read single sentences. Surprisal values for each word in the sentence stimuli were estimated by both a recurrent neural network and a phrase- structure grammar. Higher surprisal corresponded to longer word-reading time, and this effect was stronger when surprisal values were estimated by the neural network. In addition, there was an early, positive effect of surprisal on pupil size, from about 250 ms before word fixation until 100 ms after fixation. This early effect, which was only significant for the network- based surprisal estimates, is suggestive of a preparation-based account of surprisal. Keywords: Reading; Eye tracking; Pupillometry; Sentence comprehension; Surprisal; Recurrent neural network; Phrase- structure grammar Introduction Language comprehension is mostly incremental: When lis- tening to or reading a sentence, each word is immediately integrated with information from the sentence so far (e.g., Just, Carpenter, & Woolley, 1982). It has been argued that the amount of cognitive effort required to process a given word can be quantified by its surprisal (Hale, 2001; Levy, 2008), an information-theoretic measure of the extent to which the word’s occurrence was unexpected. Formally, if w 1...t de- notes the sentence’s first t words, the surprisal of the fol- lowing word is: surprisal(w t+1 ) = − log P(w t+1 |w 1...t ). These values can be estimated by any language model that assigns probabilities to word sequences. The relationship between surprisal and cognitive load (i.e., relative difficulty in processing) has indeed been observed in reading studies: Words with higher surprisal values take longer to read, which accounts for several phenomena in sen- tence comprehension, such as garden-path effects (Brouwer, Fitz, & Hoeks, 2010) and anti-locality effects (Levy, 2008). More generally, reading times at each word in sentences or texts have been shown to correlate with surprisal (e.g., Boston, Hale, Patil, Kliegl, & Vasishth, 2008; Demberg & Keller, 2008; Fernandez Monsalve, Frank, & Vigliocco, 2012; Frank & Bod, 2011; Smith & Levy, 2008). Here, we investigate an alternative empirical index of cog- nitive load; one that can be measured continuously and with precise time-resolution: pupil size. By analyzing how and when effects of word surprisal appear in pupillometry data, we are able to use a physiological measure to investigate the fine-grained time course of sentence-comprehension pro- cesses. A large number of studies, using a variety of tasks, have looked at the relationship between cognitive load and pupil dilation (for a recent overview, see Laeng, Sirois, & Gre- deb¨ack, 2012). Although these studies differ in how cogni- tive load is operationalized, increased cognitive load is invari- ably found to result in larger pupil size. In a non-linguistic context, Preuschoff, ’t Hart, and Einh¨auser (2011) showed that pupil size (and therefore, presumably, cognitive load) in- creases when a stimulus is less expected. They had partici- pants perform a simple gambling task and found that expe- riencing surprise causes pupil dilation: Pupil size correlated not with the gambling outcome itself but with its unexpected- ness. Whether unexpectedness of words in sentences also results in pupil dilation is still an open question. In fact, there has been only very little pupillometry research in psycholinguis- tics. Engelhardt, Ferreira, and Patsenko (2010) found that a mismatch between syntactic and prosodic structure of audito- rily presented sentences results in larger pupil size compared to a condition in which the two structures matched. In another sentence-listening study, Piquado, Isaacowitz, and Wingfield (2010) found a pupil response to both syntactic complexity and sentence length. To the best of our knowledge, there ex- ists only two published studies in which pupillometry is ap- plied during sentence reading: Raisig, Hagendorf, and Van der Meer (2012) presented participants with written descrip- tions of simple events in everyday activities and found in- creased pupil dilation when the order of presentation was incongruent with the actual temporal order of the described activities. Just and Carpenter (1993) compared object- and subject-relative clauses and found increased reading times and pupil dilation on the object-relatives, which have long been known to be more difficult to process (Hakes, Evans, & Brannon, 1976). Moreover, the occurrence of a semantically implausible word resulted in increased pupil size compared to a plausible-word condition. Here, we did not compare particular sentence pairs but, in- stead, investigate the general relation between word surprisal