Hardware Accelerator for Bidirectional Encoder Representations from Transformers (BERT)

Recently Bidirectional Encoder Representations from Transformers (BERT) model has gained lots of attention because of its state-of-the-art performance in multiple natu-ral language processing (NLP) tasks. However, just like many other deep learning based tasks, large model size and intensive computation load of BERT make it difficult and expensive to run and implement on general purpose processors. The proposed hardware accelerator for BERT model realizes faster inference speed and higher energy efficiency. Design procedure is elaborated with two stages: model compression and hardware architecture. Quantization is chosen as the compression technique because of good speed-up performance as well as small model size and less complexity. In the hardware design, systolic tensor array (STA) is applied as processing elements (PE) array to achieve lower area and power consumption by reducing the ratio between registers and number of Floating-point operations per second (FLOPS). Dedicated hardware is designed for Softmax and layer normalization operations. Mathematical transformation is used to replace complicate nonlinear functions with simple operations to reduce the required hardware resources. Performance is evaluated based on transformer-base model. The maximum speed of overall hardware design is 125 MHz and the total latency is 165.9 us. Compared to the same task run on GPU, 22.4x and 7.5x speed up are achieved in multi-head-attention (MHA) and feed-forward networks(FFN) separately. The peak performance of this design is 4.1 TOPs/s and the maximum required memory bandwidth is 80 GB/s.