Asynchronous reflections: theory and practice in the design of multimedia mirror systems

In this paper, we present a theoretical framing of the functions of a mirror by breaking the synchrony between the state of a reference object and its reflection. This framing provides a new conceptualization of the uses of reflections for various applications. We describe the fundamental technical components of such systems and illustrate the technical challenges in two different forms of electronic mirror systems for apparel shopping. The first example, the Responsive Mirror, is an intelligent video capture and access system for clothes shopping in physical stores that provides personalized asynchronous reflections of clothing items through an implicitly controlled human–computer interface. The Responsive Mirror employs computer vision and machine learning techniques to interpret the visual cues of the shopper’s behavior from cameras to then display two different reflections of the shopper on digital displays: (1) the shopper in previously worn clothing with matching pose and orientation and (2) other people in similar and dissimilar shirts with matching pose and orientation. The second example system is a Countertop Responsive Mirror that differs from the first in that the images do not respond to the real-time movement of the shopper but to frames in a recorded video so that the motion of the shopper in the different recordings are matched non-sequentially. These instantiations of the mirror systems in fitting room and jewelry shopping scenarios are described, focusing on the system architecture and the intelligent computer vision components. The effectiveness of Responsive Mirror is demonstrated by the user study. The paper contributes a conceptualization of reflection and examples of systems illustrating new applications in multimedia systems that break traditional reflective synchronies.