A Brief Tutorial on Sample Size Calculations for Fairness Audits
CoRR(2023)
摘要
In fairness audits, a standard objective is to detect whether a given
algorithm performs substantially differently between subgroups. Properly
powering the statistical analysis of such audits is crucial for obtaining
informative fairness assessments, as it ensures a high probability of detecting
unfairness when it exists. However, limited guidance is available on the amount
of data necessary for a fairness audit, lacking directly applicable results
concerning commonly used fairness metrics. Additionally, the consideration of
unequal subgroup sample sizes is also missing. In this tutorial, we address
these issues by providing guidance on how to determine the required subgroup
sample sizes to maximize the statistical power of hypothesis tests for
detecting unfairness. Our findings are applicable to audits of binary
classification models and multiple fairness metrics derived as summaries of the
confusion matrix. Furthermore, we discuss other aspects of audit study designs
that can increase the reliability of audit results.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要