Revise Saturated Activation Functions
arXiv: Learning, Volume abs/1602.05980, 2016.
In this paper, we revise two commonly used saturated functions, the logistic sigmoid and the hyperbolic tangent (tanh). point out that, besides the well-known non-zero centered property, slope of the activation function near the origin is another possible reason making training deep networks with the logistic function difficult to train....More
PPT (Upload PPT)