SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
NIPS 2020, 2020.
We propose Sparse Adaptive Connection — a sparse connection method to accelerate and structure the self-attention mechanism that adapts to various downstream tasks
While the self-attention mechanism has been widely used in a wide variety of tasks, it has the unfortunate property of a quadratic cost with respect to the input length, which makes it difficult to deal with long inputs. In this paper, we present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection (SAC). ...More
PPT (Upload PPT)