A bi-directional attention guided cross-modal network for music based dance generation

AI choreography has been an active task over the past few years. A fundamental problem of generating dance from music is to effectively learn the correlation between two modalities and then to create long-term body movements. To bridge the gap between music and motion, we propose a novel transformer-based cross-modal framework with a bi-directional attention module, aiming at generating high-quality dance sequences according to the music. The bi-directional attention module contains two layers: one is from music to dance, and the other is from dance to music, bringing benefits to the information exchange and alignment between two modalities. Furthermore, to alleviate exposure bias in sequence-to-sequence generation, we exploit a scheduled sampling strategy that is compatible with the transformer, which reduces error accumulation in long dance generation. Experimental results on 3D dance datasets demonstrate that our approach can achieve rhythmic natural long dance sequences.