Blockwise Self-Attention for Long Document Understanding
EMNLP, pp. 2555-2565, 2019.
EI
Abstract:
We present BlockBERT, a lightweight and efficient BERT model that is designed to better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training time, which also enables attention heads to capture either short- or long-rang...More
Code:
Data:
Full Text
Tags
Comments