Markov chain importance sampling for minibatches

Machine Learning(2023)

引用 0|浏览2
暂无评分
摘要
This study investigates importance sampling under the scheme of minibatch stochastic gradient descent, under which the contributions are twofold. First, theoretically, we develop a neat tilting formula, which can be regarded as a general device for asymptotically optimal importance sampling. Second, practically, guided by the formula, we present an effective algorithm for importance sampling which accounts for the effects of minibatches and leverages the Markovian property of the gradients between iterations. Experiments conducted on artificial data confirm that our algorithm consistently delivers superior performance in terms of variance reduction. Furthermore, experiments carried out on real-world data demonstrate that our method, when paired with relatively straightforward models like multilayer perceptron and convolutional neural networks, outperforms in terms of training loss and testing error.
更多
查看译文
关键词
Importance sampling,Stochastic gradient descent,Variance reduction,Minibatch
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要