Mind the Switch: Measuring Modality-Switch Cognitive Costs (MSCC) As a Metric for Multimodal Interaction.

Multimodal interfaces increasingly blend voice, touch, and other inputs to enable natural interaction. However, transitions between modalities often introduce hidden cognitive costs not captured by traditional usability metrics. This Work-in-Progress paper presents the Modality-Switch Cognitive Cost (MSCC), a validated metric for quantifying the performance and workload penalties of switching between modalities. Building on an empirical study with 65 participants performing smart assistant tasks, we observed asymmetric switching costs, with Touch -> Voice transitions producing higher cognitive load than Voice -> Touch. We discuss MSCC as a foundation for adaptive multimodal design and outline ongoing work extending the metric to AR/VR and cross-device contexts.