Learning with Evolved Activation Functions

The choice of activation function can have a large effect on the final accuracy of a neural network. While there have been some attempts to hand-engineer novel activation functions, there remain few activation functions that are commonly used in practice. We show that evolutionary algorithms can discover novel activation functions that outperform commonlyused functions such as ReLU. We define a tree-based search space of potential activation functions and explore the space with exhaustive search, mutation, and crossover. Experiments on the MNIST and CIFAR-10 image datasets show our approach is effective for both fully-connected and convolutional architectures. Finally, we verify that our evolutionary algorithm outperforms random search.