AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set

On Testing of Samplers

NIPS 2020, (2020)

Cited by: 0|Views15
EI
Full Text
Bibtex
Weibo

Abstract

Given a set of items $\mathcal{F}$ and a weight function $\mathtt{wt}: \mathcal{F} \mapsto (0,1)$, the problem of sampling seeks to sample an item proportional to its weight. Sampling is a fundamental problem in machine learning. The daunting computational complexity of sampling with formal guarantees leads designers to propose heuristi...More

Code:

Data:

Introduction
  • Motivated by the success of statistical techniques, automated decision-making systems are increasingly employed in critical domains such as medical [19], aeronautics [33], criminal sentencing [20], and military [2].
  • There has been a call for the design of randomized and quantitative formal methods [35] to verify the basic building blocks of the modern AI systems.
  • The authors focus on one such core building block: constrained sampling.
  • Given a set of constraints φ over a set of variables X and a weight function wt over assignments to X, the problem of constrained sampling is to sample a satisfying assignment σ of φ with probability proportional to wt(σ).
Highlights
  • Motivated by the success of statistical techniques, automated decision-making systems are increasingly employed in critical domains such as medical [19], aeronautics [33], criminal sentencing [20], and military [2]
  • We study the problem of verifying whether a probabilistic sampler samples from a given discrete distribution
  • We present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set
  • We noticed that the analytical upper bound on the sample complexity is significantly weak compared to our observed values; this suggests that the bounds could be further tightened
  • Our algorithm can only deal with those discrete distributions for which the relative probabilities of any two points is computable
  • Since our algorithm does not deal with all possible discrete distributions, extending the approach to other distributions would enable the testing of a broader set of samplers
Results
  • The objective of the evaluation was to answer the following questions: RQ1. Is Barbarik2 able to distinguish between off-the-shelf samplers by returning ACCEPT for samplers ε-close to the ideal distribution and REJECT for the η-far samplers?

    RQ2.
  • To evaluate the runtime performance of Barbarik2 and test the quality of some state of the art samplers, the authors implemented a prototype of Barbarik2 in Python.
  • The authors' algorithm utilizes an ideal sampler, for which the authors use the state of the art sampler WAPS [25].
  • The authors use a single core with a timeout of 24 hours.
  • The detailed logs along with list of benchmarks and the runtime code employed to run the experiments are available at http://doi.org/10.5281/zenodo.4107136
Conclusion
  • The authors study the problem of verifying whether a probabilistic sampler samples from a given discrete distribution.
  • The authors present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set.
  • The authors' algorithm can only deal with those discrete distributions for which the relative probabilities of any two points is computable.
  • Since the algorithm does not deal with all possible discrete distributions, extending the approach to other distributions would enable the testing of a broader set of samplers
Summary
  • Introduction:

    Motivated by the success of statistical techniques, automated decision-making systems are increasingly employed in critical domains such as medical [19], aeronautics [33], criminal sentencing [20], and military [2].
  • There has been a call for the design of randomized and quantitative formal methods [35] to verify the basic building blocks of the modern AI systems.
  • The authors focus on one such core building block: constrained sampling.
  • Given a set of constraints φ over a set of variables X and a weight function wt over assignments to X, the problem of constrained sampling is to sample a satisfying assignment σ of φ with probability proportional to wt(σ).
  • Objectives:

    The authors' goal is to design a program that can test the quality of a sampler with respect to an ideal sampler.
  • Results:

    The objective of the evaluation was to answer the following questions: RQ1. Is Barbarik2 able to distinguish between off-the-shelf samplers by returning ACCEPT for samplers ε-close to the ideal distribution and REJECT for the η-far samplers?

    RQ2.
  • To evaluate the runtime performance of Barbarik2 and test the quality of some state of the art samplers, the authors implemented a prototype of Barbarik2 in Python.
  • The authors' algorithm utilizes an ideal sampler, for which the authors use the state of the art sampler WAPS [25].
  • The authors use a single core with a timeout of 24 hours.
  • The detailed logs along with list of benchmarks and the runtime code employed to run the experiments are available at http://doi.org/10.5281/zenodo.4107136
  • Conclusion:

    The authors study the problem of verifying whether a probabilistic sampler samples from a given discrete distribution.
  • The authors present a conditional sampling technique that can verify the sampler in sample complexity constant in terms of the sampling set.
  • The authors' algorithm can only deal with those discrete distributions for which the relative probabilities of any two points is computable.
  • Since the algorithm does not deal with all possible discrete distributions, extending the approach to other distributions would enable the testing of a broader set of samplers
Tables
  • Table1: A"(resp. “R") represents Barbarik2 returning ACCEPT(resp. REJECT). maxSamp represents the upper bound on the number of samples required by Barbarik2 to return ACCEPT/REJECT
  • Table2: The Extended Table tilt (maxSamp)
  • Table3: Extended table comparing the baseline tester for wSTS with Barbarik2
  • Table4: Extended table comparing the baseline tester for wQuicksampler with Barbarik2
  • Table5: Extended table comparing the baseline tester for wUniGen with Barbarik2
  • Table6: Number of samples required for baseline tester
Download tables as Excel
Related work
  • Distribution testing involves testing whether an unknown probability distribution is identical or close to a given distribution. This problem has been studied extensively in the property testing literature [11, 8, 36, 37] . The sample space is exponential, and for many fundamental distributions, including uniform, it is prohibitively expensive in terms of samples to verify closeness. This led to the development of the conditional sampling model [11, 8], which can provide sub-linear or even constant sample complexities for the testing of the above-given properties[1, 28, 5, 9, 17]. A detailed discussion on prior work in property testing and their relationship to Barbarik2 is given in Appendix A.

    The first practically efficient algorithm for verification of samplers with a formal proof of correctness was presented by Chakraborty and Meel in form of Barbarik [10]. The central idea of Barbarik, building on the work of Chakraborty et al [11] and Canonne et al [8], was that if one can have conditional samples from the distribution, then one can test properties of the distribution using only a constant number of conditional samples.
Funding
  • This work was supported in part by the National Research Foundation Singapore under its NRF Fellowship Programme [NRFNRFFAI1-2019-0004] and the AI Singapore Programme [AISG-RP-2018-005], and NUS ODPRT Grant [R-252-000-685-13]
Reference
  • Jayadev Acharya, Clément L. Canonne, and Gautam Kamath. A chasm between identity and equivalence testing with conditional queries. Theory of Computing, 2018.
    Google ScholarLocate open access versionFindings
  • Jürgen Altmann and Frank Sauer. Autonomous weapon systems and strategic stability. Survival, 2017.
    Google ScholarLocate open access versionFindings
  • Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to MCMC for machine learning. Machine learning, 2003.
    Google ScholarFindings
  • Tugkan Batu, Lance Fortnow, Ronitt Rubinfeld, Warren D. Smith, and Patrick White. Testing closeness of discrete distributions. J. ACM, 2013.
    Google ScholarLocate open access versionFindings
  • Rishiraj Bhattacharyya and Sourav Chakraborty. Property testing of joint distributions using conditional samples. TOCT, 2018.
    Google ScholarLocate open access versionFindings
  • Stephen P Brooks and Andrew Gelman. General methods for monitoring convergence of iterative simulations. Journal of computational and graphical statistics, 1998.
    Google ScholarLocate open access versionFindings
  • Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of Markov Chain Monte Carlo. CRC press, 2011.
    Google ScholarFindings
  • Clément L. Canonne, Dana Ron, and Rocco A. Servedio. Testing probability distributions using conditional samples. SIAM J. Comput., 2015.
    Google ScholarLocate open access versionFindings
  • Clément L. Canonne, Ilias Diakonikolas, Daniel M. Kane, and Alistair Stewart. Testing conditional independence of discrete distributions. CoRR, 2017.
    Google ScholarLocate open access versionFindings
  • Sourav Chakraborty and Kuldeep S. Meel. On testing of uniform samplers. In Proc. of AAAI, 2019.
    Google ScholarLocate open access versionFindings
  • Sourav Chakraborty, Eldar Fischer, Yonatan Goldhirsh, and Arie Matsliah. On the power of conditional samples in distribution testing. SIAM J. Comput., 2016.
    Google ScholarLocate open access versionFindings
  • Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. A Scalable and Nearly Uniform Generator of SAT Witnesses. In Proc. of CAV, 2013.
    Google ScholarLocate open access versionFindings
  • Supratik Chakraborty, Daniel J. Fremont, Kuldeep S. Meel, Sanjit A. Seshia, and Moshe Y. Vardi. Distribution-aware sampling and weighted model counting for SAT. In Proc. of AAAI, 2014.
    Google ScholarLocate open access versionFindings
  • Supratik Chakraborty, Daniel J. Fremont, Kuldeep S. Meel, Sanjit A. Seshia, and Moshe Y. Vardi. On parallel scalable uniform SAT witness generation. In Proc. of TACAS, 2015.
    Google ScholarLocate open access versionFindings
  • Supratik Chakraborty, Dror Fried, Kuldeep S Meel, and Moshe Y Vardi. From weighted to unweighted model counting. In Proc. of IJCAI, 2015.
    Google ScholarLocate open access versionFindings
  • Mark Chavira and Adnan Darwiche. On probabilistic inference by weighted model counting. Artificial Intelligence, 2008.
    Google ScholarLocate open access versionFindings
  • Xi Chen, Rajesh Jayaram, Amit Levi, and Erik Waingarten. Learning and testing junta distributions with subcube conditioning. CoRR, 2020.
    Google ScholarFindings
  • David Cohen-Steiner, Pierre Alliez, and Mathieu Desbrun. Variational shape approximation. In ACM SIGGRAPH Papers. 2004.
    Google ScholarLocate open access versionFindings
  • Elliott Crigger and Christopher Khoury. Making policy on augmented intelligence in health care. AMA Journal of Ethics, 2019.
    Google ScholarLocate open access versionFindings
  • Michael E Donohue. A replacement for Justitia’s scales?: Machine learning’s role in sentencing. Harvard Journal of Law & Technology, 2019.
    Google ScholarLocate open access versionFindings
  • Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, and Koushik Sen. Efficient sampling of SAT solutions for testing. In Proc. of ICSE, 2018.
    Google ScholarLocate open access versionFindings
  • Stefano Ermon, Carla P. Gomes, and Bart Selman. Uniform solution sampling using a constraint solver as an oracle. In Proc. of UAI, 2012.
    Google ScholarLocate open access versionFindings
  • Stefano Ermon, Carla P Gomes, Ashish Sabharwal, and Bart Selman. Embed and project: Discrete sampling with universal hashing. In Proc. of NIPS, 2013.
    Google ScholarLocate open access versionFindings
  • Carla P. Gomes, Ashish Sabharwal, and Bart Selman. Near-uniform sampling of combinatorial spaces using XOR constraints. In Proc. of NIPS, 2007.
    Google ScholarLocate open access versionFindings
  • Rahul Gupta, Shubham Sharma, Subhajit Roy, and Kuldeep S. Meel. WAPS: Weighted and Projected Sampling. In Proc. of TACAS, 2019.
    Google ScholarLocate open access versionFindings
  • Mark Jerrum. Mathematical foundations of the markov chain monte carlo method. In Probabilistic methods for algorithmic discrete mathematics. 1998.
    Google ScholarFindings
  • Mark R. Jerrum and Alistair Sinclair. The Markov Chain Monte Carlo method: an approach to approximate counting and integration. Approximation algorithms for NP-hard problems, 1996.
    Google ScholarFindings
  • Gautam Kamath and Christos Tzamos. Anaconda: A non-adaptive conditional sampling algorithm for distribution testing. SIAM, 2019.
    Google ScholarLocate open access versionFindings
  • Scott Kirkpatrick, C. Daniel Gelatt, and Mario P. Vecchi. Optimization by simulated annealing. Science, 1983.
    Google ScholarLocate open access versionFindings
  • Chris J Maddison, Daniel Tarlow, and Tom Minka. A* sampling. In Proc. of NIPS, 2014.
    Google ScholarLocate open access versionFindings
  • Neal Madras. Lectures on Monte Carlo Methods, Fields Institute Monographs 16. American Mathematical Society, 2002.
    Google ScholarFindings
  • Kuldeep S. Meel, Moshe Y. Vardi, Supratik Chakraborty, Daniel J Fremont, Sanjit A Seshia, Dror Fried, Alexander Ivrii, and Sharad Malik. Constrained sampling and counting: Universal hashing meets sat solving. In AAAI Workshop: Beyond NP, 2016.
    Google ScholarLocate open access versionFindings
  • Kathleen L Mosier and Linda J Skitka. Human decision makers and automated decision aids: Made for each other? In Automation and human performance. 2018.
    Google ScholarLocate open access versionFindings
  • K.P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
    Google ScholarFindings
  • Sanjit A Seshia, Dorsa Sadigh, and S Shankar Sastry. Towards verified artificial intelligence. arXiv preprint arXiv:1606.08514, 2016.
    Findings
  • Gregory Valiant and Paul Valiant. The Power of Linear Estimators. In Proc of FOCS, 2011.
    Google ScholarLocate open access versionFindings
  • Paul Valiant. Testing symmetric properties of distributions. SIAM J. Comput., 2011.
    Google ScholarLocate open access versionFindings
  • Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Found. Trends Machine Learning, 2008.
    Google ScholarLocate open access versionFindings
  • 1. If E[Yi] ≥ θ ≥ 0, then for any t ≤ θ,
    Google ScholarFindings
  • 2. If E[Yi] ≤ θ, then for any t ≥ θ,
    Google ScholarFindings
Author
Yash Pralhad Pote
Yash Pralhad Pote
Your rating :
0

 

Tags
Comments
小科