Adaptive Input Normalization for Quantized Neural Networks.

Neural networks with quantized activation functions cannot adapt the quantization at the input of their first layer. Preprocessing is therefore required to adapt the range of input data to the quantization range. Such preprocessing usually includes an activation-wise linear transformation and is steered by the properties of the training set. We suggest to include the linear transform into the training process. Using the Jet Stream Classification task and an evaluation architecture of three quantized dense layers, we document that it improves accuracy, requires the same resources as standard preprocessing, plays a role in network pruning, and is reasonably stable with respect to initialization.