Piecewise linear activations substantially shape the loss surfaces of neural networks
Based on a recent result that the loss surface has a smooth and multilinear partition, we draw a big picture of the loss surface from the following aspects: local minima in any cell are good, and they are all global minima in the cell; all local minima in one cell constitute an e...
Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning. This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks. We first prove that the loss surfaces of many neural networks have infinite spurious local minima, whic...More
PPT (Upload PPT)