It's Hard for Neural Networks to Learn the Game of Life
2021 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN)(2021)
摘要
Efforts to improve the learning abilities of neural networks have focused mostly on the role of optimization methods rather than on weight initializations. Recent findings, however, suggest that neural networks rely on lucky random initial weights of subnetworks called "lottery tickets" that converge quickly to a solution [1]. To investigate how weight initializations affect performance, we examine small convolutional networks that are trained to predict n steps of the two-dimensional cellular automaton Conway's Game of Life, the update rules of which can be implemented efficiently in a small CNN. We find that networks of this architecture trained on this task rarely converge. Rather, networks require substantially more parameters to consistently converge. Furthermore, we find that the initialization parameters that gradient descent converges to a solution are sensitive to small perturbations, such as a single sign change. Finally, we observe a critical value d(0) such that training minimal networks with examples in which cells are alive with probability d(0) dramatically increases the chance of convergence to a solution. Our results are consistent with the lottery ticket hypothesis [1].
更多查看译文
关键词
neural networks, game of life, lottery ticket hypothesis
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络