Piecewise linear activations substantially shape the loss surfaces of neural networks

Fengxiang He

[0]

Bohan Wang

Dacheng Tao (陶大程)

[0]

ICLR, 2020.

Cited by: 2|Views22

Keywords:

neural network nonlinear activation loss surface spurious local minimum

Weibo:

Based on a recent result that the loss surface has a smooth and multilinear partition, we draw a big picture of the loss surface from the following aspects: local minima in any cell are good, and they are all global minima in the cell; all local minima in one cell constitute an e...

Abstract:

Understanding the loss surface of a neural network is fundamentally important to the understanding of deep learning. This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks. We first prove that the loss surfaces of many neural networks have infinite spurious local minima, whic...More

Code:

Data:

Full Text

View via Publisher