Alpha Model Domination in Multiple Choice Learning

Multiple Choice Learning (MCL) algorithms involve an 'oracle' user with unmodeled biases that selects a preferred output among several possible hypotheses. We identify a shortcoming and training obstacle of the recent Stochastic Multiple Choice Learning (sMCL) algorithm. When an ensemble of neural networks trains under sMCL, the best performing model receives the majority of parameter updates at the expense of other ensemble members. We refer to this sMCL training issue as Alpha Model Domination (AMD) and empirically demonstrate that AMD does not resolve itself with longer training time. We introduce several novel MCL loss functions that both avoid AMD and yield statistically significant improvements in oracle accuracy (OA) compared to sMCL. Using the MNIST, CIFAR-10, and ImageNet classification datasets, we empirically demonstrate the superior performance of our proposed loss functions.