An investigation of dynamic crossmodal instantiation in TUIs.

There is a growing research interest in combining crossmodal research with tangible interaction design. However, most behavioural research on crossmodal perception and on multimodal tangible interaction evaluation are based on static and non-interactive signals. Grounded in the cognitive neuroscience studies and behavioural research on multisensory perception, we present two interactive, dynamic crossmodal studies based on unimodal and crossmodal physical dimensions through an interactive table-top. We tested the implementation in two user studies. Results indicate that first, dynamic crossmodal congruency was spontaneously invoked during the tangible interaction and elicited better user performance than uni-modal instantiation in the conditions involving vision and haptics, second, crossmodal instantiations using all three modalities may not produce the best user performance in the interaction, and third, participant's subconscious reaction towards perceived stimuli during interaction may not always consistent with their explicit verbalization.