Smart-Init of Neural Networks

The initialization of neural networks is of significant importance for their performance. Currently, the prevalent initialization method is a random sample based on the network’s structure. This work presents a general yet effective method to initialize neural networks. We provide repeated experiments of training variants of Mobile-Net over down-sampled variants of Image-Net, demonstrating accuracy gain and loss decrease across most of the test sets and validation sets. E.g., for Mobile-Net (v1) and Image-Net ($32 \times 32$), we had $2.5 \%$ accuracy improvement over both the test and validation sets.