Designing Interfaces to Help Stakeholders Comprehend, Navigate, and Manage Algorithmic Trade-Offs

Artificial intelligence algorithms have been applied to a wide variety of tasks, including assisting human decision making in high-stake contexts. However, these algorithms are complex and have trade-offs, notably between prediction accuracy and fairness to population subgroups. This makes it hard for domain stakeholders to understand algorithms and deploy them in a way that respects their goals and values. We created an interactive interface to help stakeholders explore algorithms, visualize their trade-offs, and select algorithms with trade-offs consistent with their values. We evaluated our interface on the problem of predicting criminal defendants' likelihood to re-offend through (i) a large-scale Amazon Mechanical Turk experiment, and (ii) in-depth interviews with domain experts. The interface proved effective at communicating algorithm trade-offs and supporting human decision making and also affected participants' trust in algorithmically aided decision making. Our results have implications for the deployment of intelligent algorithms and suggest important directions for future work.