The Bootstrap Framework: Generalization Through the Lens of Online Optimization
international conference on learning representations, 2021.
We show empirical evidence that the performance gap between offline generalization and online optimization is small and propose an alternative framework for studying generalization.
We propose a new framework for reasoning about generalization in deep learning. The core idea is to couple the Real World, where optimizers take stochastic gradient steps on the empirical loss, to an Ideal World, where optimizers take steps on the population loss. This leads to an alternate decomposition of test error into: (1) the Ideal ...More
PPT (Upload PPT)