Blockwise Self-Attention for Long Document Understanding

EMNLP, pp. 2555-2565, 2019.

EI
Other Links: arxiv.org|dblp.uni-trier.de|academic.microsoft.com

Abstract:

We present BlockBERT, a lightweight and efficient BERT model that is designed to better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training time, which also enables attention heads to capture either short- or long-rang...More

Code:

Data:

Full Text
Your rating :
0

 

Tags
Comments