The Language Of Action: A New Tool For Human-Centric Interfaces

One of the major goals of human-centric interfaces is for humans to interact with robots or other machines as they do with other humans. To achieve this we need to analyze and interpret human activity sensed by a variety of sensors. In order to develop useful technology and a subsequent industry around human-centric interfaces, we need to proceed in a principled fashion. This chapter suggests that human activity can be expressed in a language, a special language with its own phonemes, its own morphemes ( words), and its own syntax. This language can be inferred using machine learning techniques applied to gargantuan amounts of data collected by cameras or other sensor networks. Developing sensory-motor language will create bridges among several disciplines. It will also provide a hierarchical structure that can lead to a successful industry. We point to a research program and the application of its early realizations to a few domains.