Exploring Conversational Search With Humans, Assistants, and Wizards.

Chatbots and conversational assistants are becoming increasingly popular. However, for information seeking scenarios, these systems still have very limited conversational abilities, and primarily serve as proxies to existing web search engines. In this work, we ask: what would conversational search look like with a truly intelligent assistant? To begin answering this question empirically, we conduct a user study, in which 21 participants are each given 3 information seeking tasks to solve using a text-based chat interface. To complete each task, participants conversed with three conversational agents: an existing commercial system, a human expert, and a perceived experimental automatic system, backed by a human 'wizard' behind the curtain. The observations and insights of our study help us understand the aspirations of users and the limitations of the current conversational agents -- and to sharpen a frontier of work required to improve conversational assistants for search scenarios.