Decision-Tailored Uncertainty Quantification for Probability Estimates

Proper loss functions are used in decision analysis to elicit probability estimates that reflect an individual's beliefs. In this paper, we consider a decision framework in which the probability estimates come from a statistical or machine learning model. We regard the probability estimates themselves as random variables with respect to the underlying distribution of the data set from which the estimates were derived. We derive a decision-tailored proper loss function, and we show that the generalized variance (based on a generalized bias-variance decomposition that we also derive in the paper) of the probability estimates under this decision-tailored loss quantifies the uncertainty in the probability estimates in a way that is relevant to the decision problem. In particular, points with low decision-tailored generalized variance correspond to points whose optimal decisions are robust with respect to the distribution of the probability estimate at that point, whereas high decision-tailored generalized variance corresponds to higher uncertainty under the distribution of the probability estimate.