Variational Representations and Neural Network Estimation of Renyi Divergences

We derive a new variational formula for the Renyi family of divergences, R-alpha(Q parallel to P), between probability measures Q and P. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this Renyi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for Renyi divergence estimators. By applying this theory to neural network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property, then the corresponding Renyi divergence estimator is consistent. In contrast to density estimator based methods, our estimators involve only expectations under Q and P and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5,000 dimensions.