Assessing data diversity and model fairness in machine learning (ML) requires access to sensitive demographic attributes, which are often unavailable due to privacy constraints. While several methods have been proposed to estimate these properties, the field lacks a unified and reproducible evaluation framework. To fill this gap, we introduce Proxy-based Assessment for Inclusion, Representation, and Equity (PAIRE), a standardized benchmark for evaluating Fairness and Diversity (FD) estimators that operate without individual-level sensitive attributes. Leveraging PAIRE, we evaluate state-of-the-art demographic estimators on binary classification (UCI Adult, 45k instances) and multiclass ranking (TREC Fair Ranking, 1.15M instances, 21 regions), measuring estimation accuracy and vulnerability to attribute inference. Advanced methods demonstrate superior diversity estimation, reducing estimation error by up to 81%. However, this performance can be inverted in fairness assessment, with naive counting-based methods achieving up to 41% lower error than advanced quantification-based estimators, highlighting that strong performance on direct prevalence estimation does not guarantee reliability for downstream fairness assessment. Finally, privacy attacks formalized with PAIRE highlight that aggregate demographic estimators can be exploited to infer individual sensitive attributes with high accuracy (F1macro>0.9). Overall, PAIRE establishes a challenging benchmark for attribute-unaware FD estimation, providing a holistic evaluation in sensitive applications.
