Sample Efficient Uncertainty Estimation using Probabilistic Neighborhood Component Analysis

Deep Neural Networks are known to make highly overconfident predictions on out-of-distribution data. Recent research has shown that probabilistic models-Bayesian Neural Network (BNNs) and Deep Ensembles are less susceptible to this issue. However research in this area has been largely confined to the big data setting. In this work we show that even BNNs and Ensembles tend to make overconfident predictions when the amount of training data is insufficient. This is especially relevant for settings like science and medicine where overconfident and inaccurate predictions can lead to disastrous consequences. To address this issue we propose a probabilistic generalization of the non-parametric kNN approach. Our model projects data into probability distributions in a latent space and then performs kNN classification on these distributions. The nonparametric classification algorithm coupled with the probabilistic nature of the model enables it to outperform previous approaches on small data classification of lung X-Rays for COVID-19 detection.