Dynamic Tensor Rematerialization

Marisa Kirisame

Steven Lyubomirsky

Altan Haan

Jennifer Brennan

Mike He

Jared Roesch

[0]

Tianqi Chen

[0]

Zachary Tatlock

[0]

ICLR 2021, 2021.

Cited by: 2|Views324

Keywords:

intermediate activationonline algorithmtensor rematerializationmemory efficient backpropagationcheckpointing techniqueMore(13+)

Weibo:

We present an online algorithm for rematerialization (recomputing intermediate activations during backpropagation instead of storing them), which enables training under low memory, finding that it is competitive with offline techniques.

Abstract:

Checkpointing enables training larger models by freeing intermediate activations and recomputing them on demand. Previous checkpointing techniques are difficult to generalize to dynamic models because they statically plan recomputations offline. We present Dynamic Tensor Rematerialization (DTR), a greedy online algorithm for heuristical...More

Code:

https://github.com/uwsampl/dtr-prototype

Data:

Full Text

View via Publisher