BLoad: Enhancing Neural Network Training with Efficient Sequential Data Handling
arxiv(2023)
摘要
The increasing complexity of modern deep neural network models and the
expanding sizes of datasets necessitate the development of optimized and
scalable training methods. In this white paper, we addressed the challenge of
efficiently training neural network models using sequences of varying sizes. To
address this challenge, we propose a novel training scheme that enables
efficient distributed data-parallel training on sequences of different sizes
with minimal overhead. By using this scheme we were able to reduce the padding
amount by more than 100x while not deleting a single frame, resulting in an
overall increased performance on both training time and Recall in our
experiments.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要