Extreme Memorization via Scale of Initialization
international conference on learning representations, 2021.
Through studying the effect of scale of initialization on generalization, we come up with an alignment measure that correlations with generalization of deep models.
We construct an experimental setup in which changing the scale of initialization strongly impacts the implicit regularization induced by SGD, interpolating from good generalization performance to completely memorizing the training set while making little progress on the test set. Moreover, we find that the extent and manner in which gener...More
PPT (Upload PPT)