Quantifying Uncertainty In Neural Network Ensembles Using U-Statistics

Quantifying uncertainty is critically important to many applications of predictive modeling. In this paper we apply a recently developed method that uses U-statistics as a basis for estimating uncertainty in ensemble regressors to the case of neural network ensembles. U-statistics generalize the notion of a sample mean and provide distributional properties to estimates obtained by ensembles of estimators. With this method, we train neural networks on subsamples of the data and use the resulting ensemble to estimate the variance of the point estimates from the ensemble. We demonstrate that neural networks predicting a regression function exhibit the required theoretical properties for use in this ensemble method, and we then perform a coverage probability study of three simulated data sets to show that the empirical coverage probabilities match the theoretical values.