Perceptual Clauses As Units of Production in Visual Descriptions

Describing our visual environments is challenging because although an enormous amount of information is simultaneously available to the visual system, the language channel must impose a linear order on that information. Moreover, the production system is at least moderately incremental, meaning that it interleaves planning and speaking processes. Here we address how the operations of these two cognitive systems are coordinated given their different characteristics. We propose the concept of a perceptual clause, defined as an interface representation that allows the visual and linguistic systems to exchange information. The perceptual clause serves as the input to the language formulator, which translates the representation into a linguistic sequence. Perceptual clauses capture speakers’ ability to describe visual scenes coherently while at the same time taking advantage of the incremental abilities of the language production system.