On the Variance of Neural Network Training with respect to Test Sets and Distributions
arxiv(2023)
摘要
Typical neural network trainings have substantial variance in test-set
performance between repeated runs, impeding hyperparameter comparison and
training reproducibility. In this work we present the following results towards
understanding this variation. (1) Despite having significant variance on their
test-sets, we demonstrate that standard CIFAR-10 and ImageNet trainings have
little variance in performance on the underlying test-distributions from which
their test-sets are sampled. (2) We show that these trainings make
approximately independent errors on their test-sets. That is, the event that a
trained network makes an error on one particular example does not affect its
chances of making errors on other examples, relative to their average rates
over repeated runs of training with the same hyperparameters. (3) We prove that
the variance of neural network trainings on their test-sets is a downstream
consequence of the class-calibration property discovered by Jiang et al.
(2021). Our analysis yields a simple formula which accurately predicts variance
for the binary classification case. (4) We conduct preliminary studies of data
augmentation, learning rate, finetuning instability and distribution-shift
through the lens of variance between runs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要