F or large nlp model

Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, putting them into production is very expensive, requiring comprehensive optimization techniques to reduce inference costs. This paper introduces a series of transformer inference optimization techniques at the algorithm and implementation levels. These techniques include a pre-padding decoding mechanism that improves token parallelism for generation and highly optimized kernels designed for very long inputs and large hidden sizes. On this basis, we propose a transformer inference acceleration library – Easy and Efficient Transformer (EET), which has a significant performance improvement over existing libraries. Compared to Faster Transformer v4.0’s implementation for transformer decoder layer on A100, EET achieves an average 2-4.5x state-of-art speedup. EET is available at https://github.com/NetEase-FuXi/EET. A demo video is available at https://youtu.be/22UPcNGcErg.