Balancing Competing Objectives with Noisy Data: Score-Based Classifiers for Welfare-Aware Machine Learning

Rolf Esther
Rolf Esther
Dean Sarah
Dean Sarah
Björkegren Daniel
Björkegren Daniel

ICML, pp. 8158-8168, 2020.

Cited by: 0|Views69
EI
Weibo:
We present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective

Abstract:

While real-world decisions involve many competing objectives, algorithmic decisions are often evaluated with a single objective function. In this paper, we study algorithmic policies which explicitly trade off between a private objective (such as profit) and a public objective (such as social welfare). We analyze a natural class of poli...More

Code:

Data:

0
Full Text
Bibtex
Weibo
Introduction
  • From medical diagnosis and criminal justice to financial loans and humanitarian aid, consequential decisions increasingly rely on data-driven algorithms.
  • The field of fair machine learning proposes algorithmic approaches that mitigate the adverse effects of single objective maximization.
  • Far it has predominantly done so by defining various fairness criteria that an algorithm ought to satisfy.
  • The impossibility of satisfying all desirable criteria Kleinberg et al (2017) and the unintended consequences of enforcing parity constraints based on sensitive attributes Kearns et al (2017) indicate that existing fairness solutions are not a panacea for these adverse effects.
  • Recent work (Liu et al, 2018; Hu & Chen, 2020) contend that while social welfare is of primary concern in many applications, common fairness constraints may be at odds with the relevant notion of welfare
Highlights
  • From medical diagnosis and criminal justice to financial loans and humanitarian aid, consequential decisions increasingly rely on data-driven algorithms
  • The field of fair machine learning proposes algorithmic approaches that mitigate the adverse effects of single objective maximization
  • We present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective
  • Taking care to consider data-limited regimes, we develop theory around the optimality of using learned predictors to make decisions
  • Score-based policies can trade off multiple objectives with scalar predictions, with error bounded by a weighted sum of the errors in the learned scores
  • The score-based approach shifts much of the difficulty of welfare-aware machine learning toward defining and predicting welfare, which is an area of active academic and policy debate Griffin (1986); Kahneman & Krueger (2006)
Methods
  • In Section 5.1 the authors corroborate the theoretical results under different simulated distributions on scores and prediction errors.
  • The authors' second experiment studies empirical Pareto frontiers from learned scores with realistic degradation of training data, in the context of sustainable abalone collection in Section 5.2.
  • The results for different pairs (σεw , σε2p are shown in Figure 2a.
  • Higher noise in the predicted scores imposes a wider distribution of empirical Pareto frontiers
Conclusion
  • The authors present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective.
  • The plug-in policy is not guaranteed to be the optimal policy learned from data.
  • When further assumptions on the problem structure are appropriate, it may be worthwhile to consider more general policy classes learned from data.
  • The score-based approach shifts much of the difficulty of welfare-aware machine learning toward defining and predicting welfare, which is an area of active academic and policy debate Griffin (1986); Kahneman & Krueger (2006)
Summary
  • Introduction:

    From medical diagnosis and criminal justice to financial loans and humanitarian aid, consequential decisions increasingly rely on data-driven algorithms.
  • The field of fair machine learning proposes algorithmic approaches that mitigate the adverse effects of single objective maximization.
  • Far it has predominantly done so by defining various fairness criteria that an algorithm ought to satisfy.
  • The impossibility of satisfying all desirable criteria Kleinberg et al (2017) and the unintended consequences of enforcing parity constraints based on sensitive attributes Kearns et al (2017) indicate that existing fairness solutions are not a panacea for these adverse effects.
  • Recent work (Liu et al, 2018; Hu & Chen, 2020) contend that while social welfare is of primary concern in many applications, common fairness constraints may be at odds with the relevant notion of welfare
  • Methods:

    In Section 5.1 the authors corroborate the theoretical results under different simulated distributions on scores and prediction errors.
  • The authors' second experiment studies empirical Pareto frontiers from learned scores with realistic degradation of training data, in the context of sustainable abalone collection in Section 5.2.
  • The results for different pairs (σεw , σε2p are shown in Figure 2a.
  • Higher noise in the predicted scores imposes a wider distribution of empirical Pareto frontiers
  • Conclusion:

    The authors present a methodology for developing welfare-aware policies that jointly optimize a private return with a public objective.
  • The plug-in policy is not guaranteed to be the optimal policy learned from data.
  • When further assumptions on the problem structure are appropriate, it may be worthwhile to consider more general policy classes learned from data.
  • The score-based approach shifts much of the difficulty of welfare-aware machine learning toward defining and predicting welfare, which is an area of active academic and policy debate Griffin (1986); Kahneman & Krueger (2006)
Tables
  • Table1: Hyperparameter configurations to generate
Download tables as Excel
Related work
  • Fair and Welfare-Aware Machine Learning

    The growing subfield of fairness in machine learning has investigated the implementation and implications of machine learning algorithms that satisfy definitions of fairness (Dwork et al, 2012; Barocas & Selbst, 2016; Barocas et al, 2019). Machine learning systems in general cannot satisfy multiple definitions of group fairness (Chouldechova, 2017; Kleinberg et al, 2017), and there are inherent limitations to using observational criteria (Kilbertus et al, 2017). Alternative notions of fairness more directly encode specific trade-offs between separate objectives, such as per-group accuracies (Kim et al, 2019) and overall accuracy versus a continuous fairness score Zliobaite (2015). These fairness strategies represent trade-offs with domain specific implications, for example in tax policy (Fleurbaey & Maniquet, 2018) or targeted poverty prediction (Noriega et al, 2018).

    An emerging line of work is concerned with the long-term impact of algorithmic decisions on societal welfare and fairness (Ensign et al, 2017; Hu & Chen, 2018; Mouzannar et al, 2019; Liu et al, 2020). Liu et al (2018) investigated the potentially harmful delayed impact that a fairness-satisfying decision policy has on the well-being of different subpopulations. In a similar spirit, Hu & Chen (2020) showed that always preferring “more fair” classifiers does not abide by the Pareto Principle (the principle that a policy must be preferable for at least one of multiple groups) in terms of welfare. Motivated by these findings, our work acknowledges that algorithmic policies affect individuals and institutions in many dimensions, and explicitly encodes these dimensions in policy optimization.
Funding
  • This work was supported by NSF grant DGE1752814, the Bill and Melinda Gates Foundation, the Center for Effective Global Action, and DARPA and NIWC under contract N66001-15-C-4066
  • MS is supported by the Open Philanthropy AI Fellowship
  • LTL is supported by the Open Philanthropy AI Fellowship and the Microsoft Ada Lovelace Fellowship
  • DB was partly supported by Microsoft Research
Reference
  • Awasthi, P., Kleindessner, M., and Morgenstern, J. Equalized odds postprocessing under imperfect group information, 2019.
    Google ScholarFindings
  • Balkanski, E. and Singer, Y. The sample complexity of optimizing a convex function. In Conference on Learning Theory, pp. 275–301, 2017.
    Google ScholarLocate open access versionFindings
  • Barocas, S. and Selbst, A. D. Big data’s disparate impact. UCLA Law Review, 2016.
    Google ScholarLocate open access versionFindings
  • Barocas, S., Hardt, M., and Narayanan, A. Fairness and Machine Learning. fairmlbook.org, 2019. http://www.fairmlbook.org.
    Locate open access versionFindings
  • Chen, J., Kallus, N., Mao, X., Svacha, G., and Udell, M. Fairness under unawareness: Assessing disparity when protected class is unobserved. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 339–348, 2019.
    Google ScholarLocate open access versionFindings
  • Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5, 2017.
    Google ScholarLocate open access versionFindings
  • Corbett-Davies, S. and Goel, S. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023, 2018.
    Findings
  • Deb, K. and Kalyanmoy, D. Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley & Sons, Inc., New York, NY, USA, 2001. ISBN 047187339X.
    Google ScholarFindings
  • Desideri, J.-A. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313–318, 2012.
    Google ScholarLocate open access versionFindings
  • Donahue, K. and Kleinberg, J. Fairness and utilization in allocating resources with uncertain demand. arXiv preprint arXiv:1906.09050, 2019.
    Findings
  • Dua, D. and Graff, C. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.
    Findings
  • Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, pp. 214–226, New York, NY, USA, 20ACM. ISBN 978-1-4503-1115-1. doi: 10.1145/2090236.2090255. URL http://doi.acm.org/10.1145/2090236.2090255.
    Locate open access versionFindings
  • Elzayn, H., Jabbari, S., Jung, C., Kearns, M., Neel, S., Roth, A., and Schutzman, Z. Fair algorithms for learning in allocation problems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 170–179, 2019.
    Google ScholarLocate open access versionFindings
  • Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., and Venkatasubramanian, S. Runaway Feedback Loops in Predictive Policing. arXiv:1706.09847 [cs, stat], June 2017. URL http://arxiv.org/abs/1706.09847.arXiv:1706.09847.
    Findings
  • Faddoul, M., Chaslot, G., and Farid, H. A longitudinal analysis of youtube’s promotion of conspiracy videos. In Preparation, 2020.
    Google ScholarLocate open access versionFindings
  • Fleurbaey, M. and Maniquet, F. Optimal income taxation theory and principles of fairness. Journal of Economic Literature, 56(3):1029–79, 2018.
    Google ScholarLocate open access versionFindings
  • Griffin, J. Well-being: Its meaning, measurement and moral importance. 1986.
    Google ScholarFindings
  • Gupta, M. R., Cotter, A., Fard, M. M., and Wang, S. Proxy fairness. CoRR, abs/1806.11212, 20URL http://arxiv.org/abs/1806.11212.
    Findings
  • Hardt, M., Price, E., and Srebro, N. Equality of opportunity in supervised learning. In Advances in neural information processing systems, pp. 3315–3323, 2016.
    Google ScholarLocate open access versionFindings
  • Hu, L. and Chen, Y. Welfare and Distributional Impacts of Fair Classification. arXiv:1807.01134 [cs, stat], July 2018. URL http://arxiv.org/abs/1807.01134.arXiv:1807.01134.
    Findings
  • Hu, L. and Chen, Y. Fair classification and social welfare. ACM FAT*, 2020.
    Google ScholarLocate open access versionFindings
  • Jin, Y. Multi-objective machine learning, volume 16. Springer Science & Business Media, 2006.
    Google ScholarFindings
  • Jin, Y. and Sendhoff, B. Pareto-based multiobjective machine learning: An overview and case studies. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(3):397–415, 2008.
    Google ScholarLocate open access versionFindings
  • Kahneman, D. and Krueger, A. B. Developments in the measurement of subjective well-being. Journal of Economic perspectives, 20(1):3–24, 2006.
    Google ScholarLocate open access versionFindings
  • Kallus, N., Mao, X., and Zhou, A. Assessing algorithmic fairness with unobserved protected class using data combination. arXiv preprint arXiv:1906.00285, 2019.
    Findings
  • Kearns, M., Neel, S., Roth, A., and Wu, Z. S. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. arXiv preprint arXiv:1711.05144, 2017.
    Findings
  • Kilbertus, N., Rojas-Carulla, M., Parascandolo, G., Hardt, M., Janzing, D., and Scholkopf, B. Avoiding discrimination through causal reasoning. In In Proc. 30th NIPS, pp. 656–666, 2017.
    Google ScholarLocate open access versionFindings
  • Kim, I. Y. and de Weck, O. L. Adaptive weighted-sum method for bi-objective optimization: Pareto front generation. Structural and multidisciplinary optimization, 29(2):149–158, 2005.
    Google ScholarLocate open access versionFindings
  • Kim, M. P., Ghorbani, A., and Zou, J. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 247–254. ACM, 2019.
    Google ScholarLocate open access versionFindings
  • Kleinberg, J. M., Mullainathan, S., and Raghavan, M. Inherent trade-offs in the fair determination of risk scores. Proc. 8th ITCS, 2017.
    Google ScholarLocate open access versionFindings
  • Knowles, J. Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1):50–66, 2006.
    Google ScholarLocate open access versionFindings
  • Komiya, H. Elementary proof for sion’s minimax theorem. Kodai Mathematical Journal, 11(1):5–7, 1988.
    Google ScholarLocate open access versionFindings
  • Lamy, A., Zhong, Z., Menon, A. K., and Verma, N. Noise-tolerant fair classification. In Advances in Neural Information Processing Systems 32, pp. 294–305. Curran Associates, Inc., 2019.
    Google ScholarLocate open access versionFindings
  • Liu, C., Xu, X., and Hu, D. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385–398, 2014.
    Google ScholarLocate open access versionFindings
  • Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., and Hardt, M. Delayed impact of fair machine learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3156–3164, Stockholm, Sweden, 2018.
    Google ScholarLocate open access versionFindings
  • Liu, L. T., Simchowitz, M., and Hardt, M. The implicit fairness criterion of unconstrained learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 4051–4060, Long Beach, California, USA, 2019. PMLR.
    Google ScholarLocate open access versionFindings
  • Liu, L. T., Wilson, A., Haghtalab, N., Kalai, A. T., Borgs, C., and Chayes, J. The disparate equilibria of algorithmic decision making when individuals invest rationally. ACM FAT*, 2020.
    Google ScholarLocate open access versionFindings
  • Loshchilov, I., Schoenauer, M., and Sebag, M. A mono surrogate for multiobjective optimization. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp. 471–478. ACM, 2010.
    Google ScholarLocate open access versionFindings
  • Mouzannar, H., Ohannessian, M. I., and Srebro, N. From Fair Decision Making to Social Equality. ACM FAT*, 2019.
    Google ScholarLocate open access versionFindings
  • Nash, W. J., Sellers, T. L., Talbot, S. R., Cawthorn, A. J., and Ford, W. B. The population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report, 48:p411, 1994.
    Google ScholarLocate open access versionFindings
  • Noriega, A., Garcia-Bulle, B., Tejerina, L., and Pentland, A. Algorithmic fairness and efficiency in targeting social welfare programs at scale. Bloomberg Data for Good Exchange Conference, 2018.
    Google ScholarFindings
  • Pareto, V. Manuale di economia politica, volume 13. Societa Editrice, 1906.
    Google ScholarFindings
  • Paria, B., Kandasamy, K., and Poczos, B. A flexible multi-objective bayesian optimization approach using random scalarizations. arXiv preprint arXiv:1805.12168, 2018.
    Findings
  • Peitz, S. and Dellnitz, M. Gradient-based multiobjective optimization with uncertainties. In NEO 2016, pp. 159–182.
    Google ScholarLocate open access versionFindings
  • Persily, N. The 2016 US Election: Can democracy survive the internet? Journal of democracy, 28(2):63–76, 2017.
    Google ScholarLocate open access versionFindings
  • Roijers, D. M. and Whiteson, S. Multi-objective decision making. Synthesis Lectures on Artificial Intelligence and Machine Learning, 11(1):1–129, 2017.
    Google ScholarLocate open access versionFindings
  • Skiba, P. M. and Tobacman, J. Do Payday Loans Cause Bankruptcy? SSRN Scholarly Paper ID 1266215, Social Science Research Network, Rochester, NY, November 2009. URL https://papers.ssrn.com/abstract=1266215.
    Findings
  • US Federal Reserve. Report to the congress on credit scoring and its effects on the availability and affordability of credit, 2007.
    Google ScholarFindings
  • Van Moffaert, K. and Nowe, A. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research, 15(1):3483–3512, 2014.
    Google ScholarLocate open access versionFindings
  • Wainwright, M. J. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
    Google ScholarFindings
  • Zliobaite, I. On the relation between accuracy and fairness in binary classification. arXiv preprint arXiv:1505.05723, 2015.
    Findings
  • The map UP(π) and UW(π) are both linear functions in π. Hence, if Π is a a convex, and compact in a topology in which π → UP(π) and UW(π) are continuous, Sion’s minimax theorem Komiya (1988) ensures that strong duality holds, which means that we can switch order of the minimization over t and maximization over π. Thus, for some t ≥ 0, as needed.
    Google ScholarLocate open access versionFindings
  • We remark that in general, optimizing arbitrary loss functions for function value states (e.g. estimating α-utilities for all α directly from features) requires a prohibitively large sample (Balkanski & Singer, 2017). The structures of the combined learning problems and α-utility in our setting allow us to circumvent this lower bound.
    Google ScholarLocate open access versionFindings
  • Uα(πα) − Uα(παplug) = E[|y| · I(y(y + z) < 0)], where y ∼ N (0, α2σw2 + (1 − α)2σp2 + 2ρα(1 − α)σwσp) and z is sub-Gaussian with squared parameter σ2 = 4(α2σε2w + (1 − α)2σε2p ) (Wainwright, 2019).5 By assumption, the errors are independent of the scores, so that
    Google ScholarLocate open access versionFindings
  • When w and p are assumed to be independent, σ2 = (α2σε2w + (1 − α)2σε2p ) Wainwright (2019).
    Google ScholarFindings
Your rating :
0

 

Tags
Comments