# Balancing Competing Objectives with Noisy Data: Score-Based Classifiers for Welfare-Aware Machine Learning

ICML, pp. 8158-8168, 2020.

EI

Weibo:

Abstract:

While real-world decisions involve many competing objectives, algorithmic decisions are often evaluated with a single objective function. In this paper, we study algorithmic policies which explicitly trade off between a private objective (such as profit) and a public objective (such as social welfare). We analyze a natural class of poli...More

Code:

Data:

Introduction

- From medical diagnosis and criminal justice to financial loans and humanitarian aid, consequential decisions increasingly rely on data-driven algorithms.
- The field of fair machine learning proposes algorithmic approaches that mitigate the adverse effects of single objective maximization.
- Far it has predominantly done so by defining various fairness criteria that an algorithm ought to satisfy.
- The impossibility of satisfying all desirable criteria Kleinberg et al (2017) and the unintended consequences of enforcing parity constraints based on sensitive attributes Kearns et al (2017) indicate that existing fairness solutions are not a panacea for these adverse effects.
- Recent work (Liu et al, 2018; Hu & Chen, 2020) contend that while social welfare is of primary concern in many applications, common fairness constraints may be at odds with the relevant notion of welfare

Highlights

- From medical diagnosis and criminal justice to financial loans and humanitarian aid, consequential decisions increasingly rely on data-driven algorithms
- The field of fair machine learning proposes algorithmic approaches that mitigate the adverse effects of single objective maximization
- We present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective
- Taking care to consider data-limited regimes, we develop theory around the optimality of using learned predictors to make decisions
- Score-based policies can trade off multiple objectives with scalar predictions, with error bounded by a weighted sum of the errors in the learned scores
- The score-based approach shifts much of the difficulty of welfare-aware machine learning toward defining and predicting welfare, which is an area of active academic and policy debate Griffin (1986); Kahneman & Krueger (2006)

Methods

- In Section 5.1 the authors corroborate the theoretical results under different simulated distributions on scores and prediction errors.
- The authors' second experiment studies empirical Pareto frontiers from learned scores with realistic degradation of training data, in the context of sustainable abalone collection in Section 5.2.
- The results for different pairs (σεw , σε2p are shown in Figure 2a.
- Higher noise in the predicted scores imposes a wider distribution of empirical Pareto frontiers

Conclusion

- The authors present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective.
- The plug-in policy is not guaranteed to be the optimal policy learned from data.
- When further assumptions on the problem structure are appropriate, it may be worthwhile to consider more general policy classes learned from data.
- The score-based approach shifts much of the difficulty of welfare-aware machine learning toward defining and predicting welfare, which is an area of active academic and policy debate Griffin (1986); Kahneman & Krueger (2006)

Summary

## Introduction:

From medical diagnosis and criminal justice to financial loans and humanitarian aid, consequential decisions increasingly rely on data-driven algorithms.- The field of fair machine learning proposes algorithmic approaches that mitigate the adverse effects of single objective maximization.
- Far it has predominantly done so by defining various fairness criteria that an algorithm ought to satisfy.
- The impossibility of satisfying all desirable criteria Kleinberg et al (2017) and the unintended consequences of enforcing parity constraints based on sensitive attributes Kearns et al (2017) indicate that existing fairness solutions are not a panacea for these adverse effects.
- Recent work (Liu et al, 2018; Hu & Chen, 2020) contend that while social welfare is of primary concern in many applications, common fairness constraints may be at odds with the relevant notion of welfare
## Methods:

In Section 5.1 the authors corroborate the theoretical results under different simulated distributions on scores and prediction errors.- The authors' second experiment studies empirical Pareto frontiers from learned scores with realistic degradation of training data, in the context of sustainable abalone collection in Section 5.2.
- The results for different pairs (σεw , σε2p are shown in Figure 2a.
- Higher noise in the predicted scores imposes a wider distribution of empirical Pareto frontiers
## Conclusion:

The authors present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective.- The plug-in policy is not guaranteed to be the optimal policy learned from data.
- When further assumptions on the problem structure are appropriate, it may be worthwhile to consider more general policy classes learned from data.
- The score-based approach shifts much of the difficulty of welfare-aware machine learning toward defining and predicting welfare, which is an area of active academic and policy debate Griffin (1986); Kahneman & Krueger (2006)

- Table1: Hyperparameter configurations to generate

Related work

- Fair and Welfare-Aware Machine Learning

The growing subfield of fairness in machine learning has investigated the implementation and implications of machine learning algorithms that satisfy definitions of fairness (Dwork et al, 2012; Barocas & Selbst, 2016; Barocas et al, 2019). Machine learning systems in general cannot satisfy multiple definitions of group fairness (Chouldechova, 2017; Kleinberg et al, 2017), and there are inherent limitations to using observational criteria (Kilbertus et al, 2017). Alternative notions of fairness more directly encode specific trade-offs between separate objectives, such as per-group accuracies (Kim et al, 2019) and overall accuracy versus a continuous fairness score Zliobaite (2015). These fairness strategies represent trade-offs with domain specific implications, for example in tax policy (Fleurbaey & Maniquet, 2018) or targeted poverty prediction (Noriega et al, 2018).

An emerging line of work is concerned with the long-term impact of algorithmic decisions on societal welfare and fairness (Ensign et al, 2017; Hu & Chen, 2018; Mouzannar et al, 2019; Liu et al, 2020). Liu et al (2018) investigated the potentially harmful delayed impact that a fairness-satisfying decision policy has on the well-being of different subpopulations. In a similar spirit, Hu & Chen (2020) showed that always preferring “more fair” classifiers does not abide by the Pareto Principle (the principle that a policy must be preferable for at least one of multiple groups) in terms of welfare. Motivated by these findings, our work acknowledges that algorithmic policies affect individuals and institutions in many dimensions, and explicitly encodes these dimensions in policy optimization.

Funding

- This work was supported by NSF grant DGE1752814, the Bill and Melinda Gates Foundation, the Center for Effective Global Action, and DARPA and NIWC under contract N66001-15-C-4066
- MS is supported by the Open Philanthropy AI Fellowship
- LTL is supported by the Open Philanthropy AI Fellowship and the Microsoft Ada Lovelace Fellowship
- DB was partly supported by Microsoft Research

Reference

- Awasthi, P., Kleindessner, M., and Morgenstern, J. Equalized odds postprocessing under imperfect group information, 2019.
- Balkanski, E. and Singer, Y. The sample complexity of optimizing a convex function. In Conference on Learning Theory, pp. 275–301, 2017.
- Barocas, S. and Selbst, A. D. Big data’s disparate impact. UCLA Law Review, 2016.
- Barocas, S., Hardt, M., and Narayanan, A. Fairness and Machine Learning. fairmlbook.org, 2019. http://www.fairmlbook.org.
- Chen, J., Kallus, N., Mao, X., Svacha, G., and Udell, M. Fairness under unawareness: Assessing disparity when protected class is unobserved. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 339–348, 2019.
- Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5, 2017.
- Corbett-Davies, S. and Goel, S. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023, 2018.
- Deb, K. and Kalyanmoy, D. Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley & Sons, Inc., New York, NY, USA, 2001. ISBN 047187339X.
- Desideri, J.-A. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313–318, 2012.
- Donahue, K. and Kleinberg, J. Fairness and utilization in allocating resources with uncertain demand. arXiv preprint arXiv:1906.09050, 2019.
- Dua, D. and Graff, C. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
- Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, pp. 214–226, New York, NY, USA, 20ACM. ISBN 978-1-4503-1115-1. doi: 10.1145/2090236.2090255. URL http://doi.acm.org/10.1145/2090236.2090255.
- Elzayn, H., Jabbari, S., Jung, C., Kearns, M., Neel, S., Roth, A., and Schutzman, Z. Fair algorithms for learning in allocation problems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 170–179, 2019.
- Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., and Venkatasubramanian, S. Runaway Feedback Loops in Predictive Policing. arXiv:1706.09847 [cs, stat], June 2017. URL http://arxiv.org/abs/1706.09847.arXiv:1706.09847.
- Faddoul, M., Chaslot, G., and Farid, H. A longitudinal analysis of youtube’s promotion of conspiracy videos. In Preparation, 2020.
- Fleurbaey, M. and Maniquet, F. Optimal income taxation theory and principles of fairness. Journal of Economic Literature, 56(3):1029–79, 2018.
- Griffin, J. Well-being: Its meaning, measurement and moral importance. 1986.
- Gupta, M. R., Cotter, A., Fard, M. M., and Wang, S. Proxy fairness. CoRR, abs/1806.11212, 20URL http://arxiv.org/abs/1806.11212.
- Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3315–3323, 2016.
- Hu, L. and Chen, Y. Welfare and Distributional Impacts of Fair Classification. arXiv:1807.01134 [cs, stat], July 2018. URL http://arxiv.org/abs/1807.01134.arXiv:1807.01134.
- Hu, L. and Chen, Y. Fair classification and social welfare. ACM FAT*, 2020.
- Jin, Y. Multi-objective machine learning, volume 16. Springer Science & Business Media, 2006.
- Jin, Y. and Sendhoff, B. Pareto-based multiobjective machine learning: An overview and case studies. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(3):397–415, 2008.
- Kahneman, D. and Krueger, A. B. Developments in the measurement of subjective well-being. Journal of Economic perspectives, 20(1):3–24, 2006.
- Kallus, N., Mao, X., and Zhou, A. Assessing algorithmic fairness with unobserved protected class using data combination. arXiv preprint arXiv:1906.00285, 2019.
- Kearns, M., Neel, S., Roth, A., and Wu, Z. S. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. arXiv preprint arXiv:1711.05144, 2017.
- Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., and Scholkopf, B. Avoiding discrimination through causal reasoning. In In Proc. 30th NIPS, pp. 656–666, 2017.
- Kim, I. Y. and de Weck, O. L. Adaptive weighted-sum method for bi-objective optimization: Pareto front generation. Structural and multidisciplinary optimization, 29(2):149–158, 2005.
- Kim, M. P., Ghorbani, A., and Zou, J. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 247–254. ACM, 2019.
- Kleinberg, J. M., Mullainathan, S., and Raghavan, M. Inherent trade-offs in the fair determination of risk scores. Proc. 8th ITCS, 2017.
- Knowles, J. Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1):50–66, 2006.
- Komiya, H. Elementary proof for sion’s minimax theorem. Kodai Mathematical Journal, 11(1):5–7, 1988.
- Lamy, A., Zhong, Z., Menon, A. K., and Verma, N. Noise-tolerant fair classification. In Advances in Neural Information Processing Systems 32, pp. 294–305. Curran Associates, Inc., 2019.
- Liu, C., Xu, X., and Hu, D. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385–398, 2014.
- Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., and Hardt, M. Delayed impact of fair machine learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3156–3164, Stockholm, Sweden, 2018.
- Liu, L. T., Simchowitz, M., and Hardt, M. The implicit fairness criterion of unconstrained learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4051–4060, Long Beach, California, USA, 2019. PMLR.
- Liu, L. T., Wilson, A., Haghtalab, N., Kalai, A. T., Borgs, C., and Chayes, J. The disparate equilibria of algorithmic decision making when individuals invest rationally. ACM FAT*, 2020.
- Loshchilov, I., Schoenauer, M., and Sebag, M. A mono surrogate for multiobjective optimization. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp. 471–478. ACM, 2010.
- Mouzannar, H., Ohannessian, M. I., and Srebro, N. From Fair Decision Making to Social Equality. ACM FAT*, 2019.
- Nash, W. J., Sellers, T. L., Talbot, S. R., Cawthorn, A. J., and Ford, W. B. The population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report, 48:p411, 1994.
- Noriega, A., Garcia-Bulle, B., Tejerina, L., and Pentland, A. Algorithmic fairness and efficiency in targeting social welfare programs at scale. Bloomberg Data for Good Exchange Conference, 2018.
- Pareto, V. Manuale di economia politica, volume 13. Societa Editrice, 1906.
- Paria, B., Kandasamy, K., and Poczos, B. A flexible multi-objective bayesian optimization approach using random scalarizations. arXiv preprint arXiv:1805.12168, 2018.
- Peitz, S. and Dellnitz, M. Gradient-based multiobjective optimization with uncertainties. In NEO 2016, pp. 159–182.
- Persily, N. The 2016 US Election: Can democracy survive the internet? Journal of democracy, 28(2):63–76, 2017.
- Roijers, D. M. and Whiteson, S. Multi-objective decision making. Synthesis Lectures on Artificial Intelligence and Machine Learning, 11(1):1–129, 2017.
- Skiba, P. M. and Tobacman, J. Do Payday Loans Cause Bankruptcy? SSRN Scholarly Paper ID 1266215, Social Science Research Network, Rochester, NY, November 2009. URL https://papers.ssrn.com/abstract=1266215.
- US Federal Reserve. Report to the congress on credit scoring and its effects on the availability and affordability of credit, 2007.
- Van Moffaert, K. and Nowe, A. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483–3512, 2014.
- Wainwright, M. J. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
- Zliobaite, I. On the relation between accuracy and fairness in binary classification. arXiv preprint arXiv:1505.05723, 2015.
- The map UP(π) and UW(π) are both linear functions in π. Hence, if Π is a a convex, and compact in a topology in which π → UP(π) and UW(π) are continuous, Sion’s minimax theorem Komiya (1988) ensures that strong duality holds, which means that we can switch order of the minimization over t and maximization over π. Thus, for some t ≥ 0, as needed.
- We remark that in general, optimizing arbitrary loss functions for function value states (e.g. estimating α-utilities for all α directly from features) requires a prohibitively large sample (Balkanski & Singer, 2017). The structures of the combined learning problems and α-utility in our setting allow us to circumvent this lower bound.
- Uα(πα) − Uα(παplug) = E[|y| · I(y(y + z) < 0)], where y ∼ N (0, α2σw2 + (1 − α)2σp2 + 2ρα(1 − α)σwσp) and z is sub-Gaussian with squared parameter σ2 = 4(α2σε2w + (1 − α)2σε2p ) (Wainwright, 2019).5 By assumption, the errors are independent of the scores, so that
- When w and p are assumed to be independent, σ2 = (α2σε2w + (1 − α)2σε2p ) Wainwright (2019).

Tags

Comments