SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection

Xiaoya Li
Xiaoya Li
Yuxian Meng
Yuxian Meng
Mingxin Zhou
Mingxin Zhou

NIPS 2020, 2020.

Cited by: 2|Views94
EI
Weibo:
We propose Sparse Adaptive Connection — a sparse connection method to accelerate and structure the self-attention mechanism that adapts to various downstream tasks

Abstract:

While the self-attention mechanism has been widely used in a wide variety of tasks, it has the unfortunate property of a quadratic cost with respect to the input length, which makes it difficult to deal with long inputs. In this paper, we present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection (SAC). ...More

Code:

Data:

0
Your rating :
0

 

Tags
Comments