Auto-regressive large language models have dramatically improved performance in natural language generation tasks. Popular architectures such as the transformer have enabled parallel training across tokens and scaled to large corpora of datasets. Generation--however--remains a fundamentally serial task where a token must be fully predicted before processing of the next token begins. In this work, we propose a framework for partially-parallelized large model inference by striding autoregressive dependencies between model layers, yielding strategies to improve latency in either memory or compute bound workflows, while preserving fully parallel training. The associated models require a simple modification in training by rolling representations along the sequence axes and create a favorable setup in inference with only minor degredation in accuracy.
分享
引用
