Speech modulated typography: towards an affective representation model

ABSTRACTThe transcription of expressive speech into text is a lossy process, since traditional textual resources are typically not capable of fully representing prosody. Speech modulated typography aims to narrow the gap between expressive speech and its textual transcription, with potential applications in affect-sensitive text interfaces, closed-captioning, and automated voice transcriptions. This paper proposes and evaluates two different representation models of prosody-related acoustic features of expressive speech mapped as axes of a variable font. Our experiment tested its participants' preferences for four of these modulations: font weight, letter width, letter slant, and baseline shift. Each of these represented utterances expressed in one of five emotions (anger, happiness, neutrality, sadness, and surprise). Participants preferred font-weight for sentences spoken with intensity and baseline shift for quieter utterances. In both cases, the distance between each syllable's fundamental frequency and centroid frequency was a good predictor of these preferences.