Differential Entropy Estimation under Gaussian Noise

There is a recent growing interest in measuring mutual information between the data X and an internal representation T of a deep neural network (DNN). In particular, the evolution of I (X; T) during training attracted much attention in the context of the Information Bottleneck theory. However, in deterministic networks with strictly monotone nonlinearities (eg, tanh or sigmoid) I (X; T) is either a constant independent of the network’s parameters (discrete X) or infinite (continuous X), making the mutual information a vacuous quantity. A possible remedy for this issue is the recently proposed paradigm of noisy DNNs, where the outputs of the hidden activities are perturbed by (small) Gaussian noises, making the X↦→ T map a stochastic parameterized channel. This work focuses on the nonparametric differential entropy estimation problem that arises in this setup: the estimation of h (S+ Z), where S is the sampled variable while Z is an isotropic Gaussian with known parameters. Our main motivation it to provide estimation techniques and error bounds that are applicable in practice for real-life DNNs. We first show that the sample complexity of any good estimator must scale exponentially with dimension. Then, a natural estimator for h (S+ Z) is proposed which approximates it via a the entropy of a Gaussian mixture. A convergence rate of O ((log n) d/4√ n) is derived for the absolute-error risk, with all constants explicit and the dependence on dimension and noise parameters made clear. We observe that (i) the inherent smoothness of the convolved distribution does not require any additional smoothness assumptions on the nonparametric class of distributions …