## AI helps you reading Science

## AI Insight

AI extracts a summary of this paper

Weibo:

# On Testing of Samplers

NIPS 2020, (2020)

EI

Keywords

Abstract

Given a set of items $\mathcal{F}$ and a weight function $\mathtt{wt}: \mathcal{F} \mapsto (0,1)$, the problem of sampling seeks to sample an item proportional to its weight. Sampling is a fundamental problem in machine learning. The daunting computational complexity of sampling with formal guarantees leads designers to propose heuristi...More

Code:

Data:

Introduction

- Motivated by the success of statistical techniques, automated decision-making systems are increasingly employed in critical domains such as medical [19], aeronautics [33], criminal sentencing [20], and military [2].
- There has been a call for the design of randomized and quantitative formal methods [35] to verify the basic building blocks of the modern AI systems.
- The authors focus on one such core building block: constrained sampling.
- Given a set of constraints φ over a set of variables X and a weight function wt over assignments to X, the problem of constrained sampling is to sample a satisfying assignment σ of φ with probability proportional to wt(σ).

Highlights

- Motivated by the success of statistical techniques, automated decision-making systems are increasingly employed in critical domains such as medical [19], aeronautics [33], criminal sentencing [20], and military [2]
- We study the problem of verifying whether a probabilistic sampler samples from a given discrete distribution
- We present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set
- We noticed that the analytical upper bound on the sample complexity is significantly weak compared to our observed values; this suggests that the bounds could be further tightened
- Our algorithm can only deal with those discrete distributions for which the relative probabilities of any two points is computable
- Since our algorithm does not deal with all possible discrete distributions, extending the approach to other distributions would enable the testing of a broader set of samplers

Results

- The objective of the evaluation was to answer the following questions: RQ1. Is Barbarik2 able to distinguish between off-the-shelf samplers by returning ACCEPT for samplers ε-close to the ideal distribution and REJECT for the η-far samplers?

RQ2. - To evaluate the runtime performance of Barbarik2 and test the quality of some state of the art samplers, the authors implemented a prototype of Barbarik2 in Python.
- The authors' algorithm utilizes an ideal sampler, for which the authors use the state of the art sampler WAPS [25].
- The authors use a single core with a timeout of 24 hours.
- The detailed logs along with list of benchmarks and the runtime code employed to run the experiments are available at http://doi.org/10.5281/zenodo.4107136

Conclusion

- The authors study the problem of verifying whether a probabilistic sampler samples from a given discrete distribution.
- The authors present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set.
- The authors' algorithm can only deal with those discrete distributions for which the relative probabilities of any two points is computable.
- Since the algorithm does not deal with all possible discrete distributions, extending the approach to other distributions would enable the testing of a broader set of samplers

Summary

## Introduction:

Motivated by the success of statistical techniques, automated decision-making systems are increasingly employed in critical domains such as medical [19], aeronautics [33], criminal sentencing [20], and military [2].- There has been a call for the design of randomized and quantitative formal methods [35] to verify the basic building blocks of the modern AI systems.
- The authors focus on one such core building block: constrained sampling.
- Given a set of constraints φ over a set of variables X and a weight function wt over assignments to X, the problem of constrained sampling is to sample a satisfying assignment σ of φ with probability proportional to wt(σ).
## Objectives:

The authors' goal is to design a program that can test the quality of a sampler with respect to an ideal sampler.## Results:

The objective of the evaluation was to answer the following questions: RQ1. Is Barbarik2 able to distinguish between off-the-shelf samplers by returning ACCEPT for samplers ε-close to the ideal distribution and REJECT for the η-far samplers?

RQ2.- To evaluate the runtime performance of Barbarik2 and test the quality of some state of the art samplers, the authors implemented a prototype of Barbarik2 in Python.
- The authors' algorithm utilizes an ideal sampler, for which the authors use the state of the art sampler WAPS [25].
- The authors use a single core with a timeout of 24 hours.
- The detailed logs along with list of benchmarks and the runtime code employed to run the experiments are available at http://doi.org/10.5281/zenodo.4107136
## Conclusion:

The authors study the problem of verifying whether a probabilistic sampler samples from a given discrete distribution.- The authors present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set.
- The authors' algorithm can only deal with those discrete distributions for which the relative probabilities of any two points is computable.
- Since the algorithm does not deal with all possible discrete distributions, extending the approach to other distributions would enable the testing of a broader set of samplers

- Table1: A"(resp. “R") represents Barbarik2 returning ACCEPT(resp. REJECT). maxSamp represents the upper bound on the number of samples required by Barbarik2 to return ACCEPT/REJECT
- Table2: The Extended Table tilt (maxSamp)
- Table3: Extended table comparing the baseline tester for wSTS with Barbarik2
- Table4: Extended table comparing the baseline tester for wQuicksampler with Barbarik2
- Table5: Extended table comparing the baseline tester for wUniGen with Barbarik2
- Table6: Number of samples required for baseline tester

Related work

- Distribution testing involves testing whether an unknown probability distribution is identical or close to a given distribution. This problem has been studied extensively in the property testing literature [11, 8, 36, 37] . The sample space is exponential, and for many fundamental distributions, including uniform, it is prohibitively expensive in terms of samples to verify closeness. This led to the development of the conditional sampling model [11, 8], which can provide sub-linear or even constant sample complexities for the testing of the above-given properties[1, 28, 5, 9, 17]. A detailed discussion on prior work in property testing and their relationship to Barbarik2 is given in Appendix A.

The first practically efficient algorithm for verification of samplers with a formal proof of correctness was presented by Chakraborty and Meel in form of Barbarik [10]. The central idea of Barbarik, building on the work of Chakraborty et al [11] and Canonne et al [8], was that if one can have conditional samples from the distribution, then one can test properties of the distribution using only a constant number of conditional samples.

Funding

- This work was supported in part by the National Research Foundation Singapore under its NRF Fellowship Programme [NRFNRFFAI1-2019-0004] and the AI Singapore Programme [AISG-RP-2018-005], and NUS ODPRT Grant [R-252-000-685-13]

Reference

- Jayadev Acharya, Clément L. Canonne, and Gautam Kamath. A chasm between identity and equivalence testing with conditional queries. Theory of Computing, 2018.
- Jürgen Altmann and Frank Sauer. Autonomous weapon systems and strategic stability. Survival, 2017.
- Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to MCMC for machine learning. Machine learning, 2003.
- Tugkan Batu, Lance Fortnow, Ronitt Rubinfeld, Warren D. Smith, and Patrick White. Testing closeness of discrete distributions. J. ACM, 2013.
- Rishiraj Bhattacharyya and Sourav Chakraborty. Property testing of joint distributions using conditional samples. TOCT, 2018.
- Stephen P Brooks and Andrew Gelman. General methods for monitoring convergence of iterative simulations. Journal of computational and graphical statistics, 1998.
- Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
- Clément L. Canonne, Dana Ron, and Rocco A. Servedio. Testing probability distributions using conditional samples. SIAM J. Comput., 2015.
- Clément L. Canonne, Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. Testing conditional independence of discrete distributions. CoRR, 2017.
- Sourav Chakraborty and Kuldeep S. Meel. On testing of uniform samplers. In Proc. of AAAI, 2019.
- Sourav Chakraborty, Eldar Fischer, Yonatan Goldhirsh, and Arie Matsliah. On the power of conditional samples in distribution testing. SIAM J. Comput., 2016.
- Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. A Scalable and Nearly Uniform Generator of SAT Witnesses. In Proc. of CAV, 2013.
- Supratik Chakraborty, Daniel J. Fremont, Kuldeep S. Meel, Sanjit A. Seshia, and Moshe Y. Vardi. Distribution-aware sampling and weighted model counting for SAT. In Proc. of AAAI, 2014.
- Supratik Chakraborty, Daniel J. Fremont, Kuldeep S. Meel, Sanjit A. Seshia, and Moshe Y. Vardi. On parallel scalable uniform SAT witness generation. In Proc. of TACAS, 2015.
- Supratik Chakraborty, Dror Fried, Kuldeep S Meel, and Moshe Y Vardi. From weighted to unweighted model counting. In Proc. of IJCAI, 2015.
- Mark Chavira and Adnan Darwiche. On probabilistic inference by weighted model counting. Artificial Intelligence, 2008.
- Xi Chen, Rajesh Jayaram, Amit Levi, and Erik Waingarten. Learning and testing junta distributions with subcube conditioning. CoRR, 2020.
- David Cohen-Steiner, Pierre Alliez, and Mathieu Desbrun. Variational shape approximation. In ACM SIGGRAPH Papers. 2004.
- Elliott Crigger and Christopher Khoury. Making policy on augmented intelligence in health care. AMA Journal of Ethics, 2019.
- Michael E Donohue. A replacement for Justitia’s scales?: Machine learning’s role in sentencing. Harvard Journal of Law & Technology, 2019.
- Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, and Koushik Sen. Efficient sampling of SAT solutions for testing. In Proc. of ICSE, 2018.
- Stefano Ermon, Carla P. Gomes, and Bart Selman. Uniform solution sampling using a constraint solver as an oracle. In Proc. of UAI, 2012.
- Stefano Ermon, Carla P Gomes, Ashish Sabharwal, and Bart Selman. Embed and project: Discrete sampling with universal hashing. In Proc. of NIPS, 2013.
- Carla P. Gomes, Ashish Sabharwal, and Bart Selman. Near-uniform sampling of combinatorial spaces using XOR constraints. In Proc. of NIPS, 2007.
- Rahul Gupta, Shubham Sharma, Subhajit Roy, and Kuldeep S. Meel. WAPS: Weighted and Projected Sampling. In Proc. of TACAS, 2019.
- Mark Jerrum. Mathematical foundations of the markov chain monte carlo method. In Probabilistic methods for algorithmic discrete mathematics. 1998.
- Mark R. Jerrum and Alistair Sinclair. The Markov Chain Monte Carlo method: an approach to approximate counting and integration. Approximation algorithms for NP-hard problems, 1996.
- Gautam Kamath and Christos Tzamos. Anaconda: A non-adaptive conditional sampling algorithm for distribution testing. SIAM, 2019.
- Scott Kirkpatrick, C. Daniel Gelatt, and Mario P. Vecchi. Optimization by simulated annealing. Science, 1983.
- Chris J Maddison, Daniel Tarlow, and Tom Minka. A* sampling. In Proc. of NIPS, 2014.
- Neal Madras. Lectures on Monte Carlo Methods, Fields Institute Monographs 16. American Mathematical Society, 2002.
- Kuldeep S. Meel, Moshe Y. Vardi, Supratik Chakraborty, Daniel J Fremont, Sanjit A Seshia, Dror Fried, Alexander Ivrii, and Sharad Malik. Constrained sampling and counting: Universal hashing meets sat solving. In AAAI Workshop: Beyond NP, 2016.
- Kathleen L Mosier and Linda J Skitka. Human decision makers and automated decision aids: Made for each other? In Automation and human performance. 2018.
- K.P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
- Sanjit A Seshia, Dorsa Sadigh, and S Shankar Sastry. Towards verified artificial intelligence. arXiv preprint arXiv:1606.08514, 2016.
- Gregory Valiant and Paul Valiant. The Power of Linear Estimators. In Proc of FOCS, 2011.
- Paul Valiant. Testing symmetric properties of distributions. SIAM J. Comput., 2011.
- Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Found. Trends Machine Learning, 2008.
- 1. If E[Yi] ≥ θ ≥ 0, then for any t ≤ θ,
- 2. If E[Yi] ≤ θ, then for any t ≥ θ,

Tags

Comments