Situated language understanding for a spoken dialog system within vehicles

HighlightsWe implemented and analyzed issues in situated language understanding in moving car.We analyzed timing of utterances, spatial relationships between the car and targets.Our algorithms improved the target identification rate by 24.1%. In this paper, we address issues in situated language understanding in a moving car, which has the additional challenge of being a rapidly changing environment. More specifically, we propose methods for understanding user queries regarding specific target buildings in their surroundings. Unlike previous studies on physically situated interactions, such as interactions with mobile robots, the task at hand is very time sensitive because the spatial relationship between the car and target changes while the user is speaking. We collected situated utterances from drivers using our research system called Townsurfer, which was embedded in a real vehicle. Based on this data, we analyzed the timing of user queries, the spatial relationships between the car and the targets, the head pose of the user, and linguistic cues. Based on this analysis, we further propose methods to optimize timing and spatial distances and to make use of linguistic cues. Finally, we demonstrate that our algorithms improved the target identification rate by 24.1% absolute.