Comparing Few to Rank Many: Active Human Preference Learning Using Randomized Frank-Wolfe Method

We study learning human preferences from limited comparison feedback, a core machine learning problem that is at the center of reinforcement learning from human feedback (RLHF). We formulate the problem as learning a Plackett-Luce (PL) model from a limited number of K-subset comparisons over a universe of N items, where typically K << N. Our objective is to select the K-subsets such that all items can be ranked with minimal mistakes within the budget. We solve the problem using the D-optimal design, which minimizes the worst-case ranking loss under the estimated PL model. Lambda ll known algorithms for this problem are computationally infeasible in our setting because we consider exponentially many subsets in K. To address this challenge, we propose a randomized Frank-Wolfe algorithm with memoization and sparse updates that has a low O(N-2 + K-2) per-iteration complexity. We analyze it and demonstrate its empirical superiority on synthetic and open-source NLP datasets.