Dynamic Tensor Rematerialization

Marisa Kirisame
Marisa Kirisame
Steven Lyubomirsky
Steven Lyubomirsky
Altan Haan
Altan Haan
Jennifer Brennan
Jennifer Brennan
Mike He
Mike He

ICLR 2021, 2021.

Cited by: 2|Views324
Weibo:
We present an online algorithm for rematerialization (recomputing intermediate activations during backpropagation instead of storing them), which enables training under low memory, finding that it is competitive with offline techniques.

Abstract:

Checkpointing enables training larger models by freeing intermediate activations and recomputing them on demand. Previous checkpointing techniques are difficult to generalize to dynamic models because they statically plan recomputations offline. We present Dynamic Tensor Rematerialization (DTR), a greedy online algorithm for heuristical...More
0
Full Text
Bibtex
Weibo
Your rating :
0

 

Tags
Comments