Real-time environmental tracking has become a fundamental capability in modern mobile phones and AR/VR devices. However, it only allows user interfaces to be anchored at a static location. Although fiducial and natural-feature tracking overlays interfaces with specific visual features, they typically require developers to define the pattern before deployment. In this paper, we introduce opportunistic interfaces to grant users complete freedom to summon virtual interfaces on everyday objects via voice commands or tapping gestures. We present the workflow and technical details of Ad hoc UI (AhUI), a prototyping toolkit to empower users to turn everyday objects into opportunistic interfaces on the fly. We showcase a set of demos with real-time tracking, voice activation, 6DoF interactions, and mid-air gestures and prospect the future of opportunistic interfaces.

While many systems have been designed to support collaboration around visual thinking tools, much less work has investigated how to share and collaboratively design physical prototypes—an important part of the design process. We describe preliminary results from a formative study on how designers communicate and collaborate in design meetings around physical and digital artifacts. Addressing some limitations in current collaboration platforms and drawing guidelines from our study, we introduce a new prototype platform for remote collaboration. This platform leverages the use of augmented reality (AR) for rendering of the remote participant and a pair of linked actuated tabletop tangible interfaces that acts as the participant’s shared physical workspace. We propose the use of actuated tabletop tangibles to synchronously render complex shapes and to act as physical input.

With the emergence of quantified-self, smart devices, Internet of Things and ubiquitous robotics, we envision new opportunities to create dynamic embedded physicalizations. In particular, we see new challenges arising in the context of personal and casual physicalizations at home. In this paper, we discuss the research directions and potential benefits of dynamic embedded physicalizations in the residential context, or Embedded Personal Physicalizations.

This paper introduces dynamic composite physicalizations, a new class of physical visualizations that use collections of self-propelled objects to represent data. Dynamic composite physicalizations can be used both to give physical form to well-known interactive visualization techniques, and to explore new visualizations and interaction paradigms. We first propose a design space characterizing composite physicalizations based on previous work in the fields of Information Visualization and Human Computer Interaction. We illustrate dynamic composite physicalizations in two scenarios demonstrating potential benefits for collaboration and decision making, as well as new opportunities for physical interaction. We then describe our implementation using wheeled micro-robots capable of locating themselves and sensing user input, before discussing limitations and opportunities for future work.

We present a first prototype of an open-ended interactive physical game aiming at developing children's understanding of dynamic systems in a playful and embodied way. We use a swarm user interface, Zooids, developed by Le Goc et al., made of independent self-propelled elements that move collectively and react to user input. Papert promoted an active way of developing a computational literacy, through programming a turtle with LOGO, from which Resnick proposed StarLogo, a "multi-turtles" language to simulate complex systems behaviors. Our interface is positioned in between these two perspectives: it allows to physically interact with multiple "turtles", each having its own dynamic. Each Zooid can be assigned an action that will affect the system behavior. Based on this principle, our first prototype invites children to resolve situations by changing individual actions in a dynamic system.

Passive haptic proxy objects allow for rich tangible interaction, and this is especially true in VR applications. However, this requires users to have many physical objects at hand. Our paper proposes robotic assembly at run-time of low-resolution haptic proxies for tangible interaction and virtual reality. These assembled physical proxy objects are composed of magnetically attached blocks which are assembled by a small multi robot system, specifically Zooids. We explore the design of the basic building blocks and illustrate two approaches to assembling physical proxies: using multi-robot systems to (1) self-assemble into structures and (2) assemble 2.5D structure with passive blocks of various heights. The success rate and completion time are evaluated for both approaches. Finally, we demonstrate the potential of assembled proxy objects for tangible interaction and virtual reality through a set of demonstrations.

We introduce a novel method based on physical proxies for investigating fundamental differences between touch and tangible interfaces. This method uses physical chips to emulate the flat, non-graspable objects that make up touch interfaces, in a way that supports direct comparison with tangible interfaces. We ran an experiment to test the effect of object thickness on participants' behavior, performance and subjective experience in spatial rearrangement tasks. We found that for the tasks tested, thick objects are faster but less accurate to operate, and that their graspability is only used occasionally. We also found that coarse manipulation of multiple thin objects is error-prone, an issue that only thick objects may allow to alleviate.

We introduce a novel method based on physical proxies for investigating fundamental differences between touch and tangible interfaces. This method uses physical chips to emulate the flat, non-graspable objects that make up touch interfaces, in a way that supports direct comparison with tangible interfaces. We ran an experiment to test the effect of object thickness on participants' behavior, performance and subjective experience in spatial rearrangement tasks. We found that for the tasks tested, thick objects are faster but less accurate to operate, and that their graspability is only used occasionally. We also found that coarse manipulation of multiple thin objects is error-prone, an issue that only thick objects may allow to alleviate.

This paper introduces swarm user interfaces, a new class of human-computer interfaces comprised of many autonomous robots that handle both display and interaction. We describe the design of Zooids, an open-source open-hardware platform for developing tabletop swarm interfaces. The platform consists of a collection of custom-designed wheeled micro robots each 2.6 cm in diameter, a radio base-station, a highspeed DLP structured light projector for optical tracking, and a software framework for application development and control. We illustrate the potential of tabletop swarm user interfaces through a set of application scenarios developed with Zooids, and discuss general design considerations unique to swarm user interfaces.

In this paper, we relate the iterative fabrication of a physical Bertin Matrix. Jacques Bertin designed and refined such devices over 10 years (1970–1980) and five iterations of what he called Dominos 1–5. For the purpose of an exhibit dedicated to Bertin's work during VIS 2014 in Paris, we designed an improved version of such device by leveraging modern fabrication possibilities and in particular a laser cutter. We describe the process, iterations and improvements of our matrix, and report lessons we learnt.

SmartTokens are small-sized tangible tokens that can sense multiple types of motion, multiple types of touch/grip, and send input events wirelessly as state-machine transitions. By providing an open platform for embedding basic sensing capabilities within small form-factors, SmartTokens extend the design space of tangible user interfaces. We describe the design and implementation of SmartTokens and illustrate how they can be used in practice by introducing a novel TUI design for event notification and personal task management.

While physical visualizations have existed for many years, most of them remain monolithic and static. We identify a promising category of physical visualizations we call composite physical visualizations. Composite physical visualizations are combinations of multiple physical objects and can be designed to better leverage both human and technological capabilities. We show that two important properties have to be considered when designing such visualizations: their level of actuation and their manipulability. Through examples, we illustrate the tradeoffs between these two dimensions, and identify the need for more research in this particular area.

We contribute a thin, transparent, and low-cost design for electric field sensing, allowing for 3D finger and hand tracking and gestures on mobile devices. Our approach requires no direct instrumentation of the hand or body, and is non-optical, allowing for a compact form-factor that is resilient to ambient illumination. Our simple driver electronics are based on an off-the-shelf chip that removes the need for building custom analog electronics. We describe the design of our transparent electrode array, and present a machine learning algorithm for mapping from signal measurements at the receivers to 3D positions. We demonstrate non-contact motion gestures, and precise 3D hand and finger localization. We conclude by discussing limitations and future work.

Revel is a new wearable tactile technology that modifies the user's tactile perception of the physical world. Current tactile technologies enhance objects and devices with various actuators to create rich tactile sensations, limiting the experience to the interaction with instrumented devices. In contrast, REVEL can add artificial tactile sensations to almost any surface or object with very little if any instrumentation of the environment. As a result, REVEL can provide dynamic tactile sensations on touch screens as well as everyday objects and surfaces in the environment, such as furniture, walls, wooden and plastic objects, and even human skin. Revel can be used in many new and exciting applications, including adding tactile feedback to projected content, enhancing the environment with tactile guidance for the visually impaired or providing personal tactile feedback for multi-user touch surfaces.