A Multimodal Home Entertainment Interface via a Mobile Device

We describe a multimodal dialogue system for interacting with a home entertainment center via a mobile device. In our working proto- type, users may utilize both a graphical and speech user interface to search TV listings, record and play television programs, and listen to music. The developed framework is quite generic, potentially supporting a wide variety of applications, as we demonstrate by integrat- ing a weather forecast application. In the pro- totype, the mobile device serves as the locus of interaction, providing both a small touch- screen display, and speech input and output; while the TV screen features a larger, richer GUI. The system architecture is agnostic to the location of the natural language process- ing components: a consistent user experience is maintained regardless of whether they run on a remote server or on the device itself.