AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
The problem generalizes the well-studied work on 1-bit compressed sensing and complements the literature on learning mixtures of sparse linear regression in a similar query model

Recovery of sparse linear classifiers from mixture of responses

NIPS 2020, (2020)

Cited by: 1|Views38
EI
Full Text
Bibtex
Weibo

Abstract

In the problem of learning a {\em mixture of linear classifiers}, the aim is to learn a collection of hyperplanes from a sequence of binary responses. Each response is a result of querying with a vector and indicates the side of a randomly chosen hyperplane from the collection the query vector belong to. This model provides a rich repre...More

Code:

Data:

0
Introduction
  • One of the first and most basic tasks of machine learning is to train a binary linear classifier.
  • The authors consider a natural generalization of this problem and model a classification task as a mixture of components
  • In this generalization, each response is stochastically generated by picking a hyperplane uniformly from the set of unknown hyperplanes, and returning the side of that hyperplane the feature vector lies.
  • The goal is to learn all of these hyperplanes as accurately as possible, using the least number of responses
  • This can be termed as a mixture of binary linear classifiers [32].
  • The term sample complexity is used with a slightly generalized meaning than traditional learning theory - as the authors explain and switch to the term query complexity instead
Highlights
  • One of the first and most basic tasks of machine learning is to train a binary linear classifier
  • We show that using an Robust Union Free Family (RUFF) in combination with another similar class of union-free family (UFF), we can deduce the supports of both β1 and β2
  • Once we obtain the supports, we use an additional O( k log nk) Gaussian queries to approximately recover the individual vectors. We extend this two-step process to recover a mixture of different sparse vectors under the assumption that the support of no vector is contained in the union of supports of the remaining ones (Assumption 1)
  • Our work focuses on recovering the unknown classifiers in the query model that was used in [35, 23] to study the mixtures of sparse linear regressions
  • We primarily focus on minimizing the query complexity of the problem, i.e., minimizing the number of queries that suffice to approximately recover all the sparse unknown vectors or their supports
  • The problem generalizes the well-studied work on 1-bit compressed sensing ( = 1) and complements the literature on learning mixtures of sparse linear regression in a similar query model
Results
  • Precision and recall for three random user pairs who have together rated at least 500 movies.
  • The results show that the algorithm predicts the movie genre preferences of the user pair with high accuracy even with small m.
  • Each of the quantities are obtained by averaging over 100 runs
Conclusion
  • Conclusion and Open Questions

    In this work, the authors initiated the study of recovering a mixture of different sparse linear classifiers given query access to an oracle.
  • The problem generalizes the well-studied work on 1-bit compressed sensing ( = 1) and complements the literature on learning mixtures of sparse linear regression in a similar query model.
  • The authors' results for > 2, rely on the assumption that the supports of all the unknown vectors are separable.
  • This separability assumption translates to each classifier using a unique feature not being used in others, which happen often in practice.
  • The authors leave the problem of support recovery and -recovery without any assumptions as an open problem
Funding
  • This research is supported in part by NSF CCF 1909046 and NSF 1934846
Reference
  • Jayadev Acharya, Arnab Bhattacharyya, and Pritish Kamath. Improved bounds for universal one-bit compressive sensing. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 2353–2357. IEEE, 2017.
    Google ScholarLocate open access versionFindings
  • Albert Ai, Alex Lapanowski, Yaniv Plan, and Roman Vershynin. One-bit compressed sensing with non-gaussian measurements. Linear Algebra and its Applications, 441:222–239, 2014.
    Google ScholarLocate open access versionFindings
  • Sanjeev Arora, Rong Ge, Ravi Kannan, and Ankur Moitra. Computing a nonnegative matrix factorization—provably. SIAM Journal on Computing, 45(4):1582–1611, 2016.
    Google ScholarLocate open access versionFindings
  • Christopher M Bishop. Latent variable models. In Learning in graphical models, pages 371–403.
    Google ScholarFindings
  • Ekin Blackwell, Carlos F Mendes De Leon, and Gregory E Miller. Applying mixed regression models to the analysis of repeated-measures data in psychosomatic medicine. Psychosomatic medicine, 68(6):870–878, 2006.
    Google ScholarLocate open access versionFindings
  • Stephane Boucheron, Gabor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
    Google ScholarFindings
  • Arun Tejasvi Chaganty and Percy Liang. Spectral experts for estimating mixtures of linear regressions. In International Conference on Machine Learning, pages 1040–1048, 2013.
    Google ScholarLocate open access versionFindings
  • Yudong Chen, Xinyang Yi, and Constantine Caramanis. A convex formulation for mixed regression with two components: Minimax optimal rates. In Conference on Learning Theory, pages 560–604, 2014.
    Google ScholarLocate open access versionFindings
  • Richard D De Veaux. Mixtures of linear regressions. Computational Statistics & Data Analysis, 8(3):227–245, 1989.
    Google ScholarLocate open access versionFindings
  • Ronald De Wolf. Efficient data structures from unionfree families of sets, 2012.
    Google ScholarFindings
  • Partha Deb and Ann M Holmes. Estimates of use and costs of behavioural health care: a comparison of standard and finite mixture models. Health economics, 9(6):475–489, 2000.
    Google ScholarLocate open access versionFindings
  • David Donoho and Victoria Stodden. When does non-negative matrix factorization give a correct decomposition into parts? In Advances in neural information processing systems, pages 1141–1148, 2004.
    Google ScholarLocate open access versionFindings
  • Arkadii G D’yachkov, IV Vorobyev, NA Polyanskii, and V Yu Shchukin. Bounds on the rate of superimposed codes. In 2014 IEEE International Symposium on Information Theory, pages 2341–2345. IEEE, 2014.
    Google ScholarLocate open access versionFindings
  • Larkin Flodin, Venkata Gandikota, and Arya Mazumdar. Superset technique for approximate recovery in one-bit compressed sensing. In Advances in Neural Information Processing Systems, pages 10387–10396, 2019.
    Google ScholarLocate open access versionFindings
  • Zoltan Furedi. On r-cover-free families. Journal of Combinatorial Theory, Series A, 73(1):172– 173, 1996.
    Google ScholarLocate open access versionFindings
  • Sivakant Gopi, Praneeth Netrapalli, Prateek Jain, and Aditya Nori. One-bit compressed sensing: Provable support and vector recovery. In International Conference on Machine Learning, pages 154–162, 2013.
    Google ScholarLocate open access versionFindings
  • F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015.
    Google ScholarLocate open access versionFindings
  • Mian Huang, Runze Li, and Shaoli Wang. Nonparametric mixture of regression models. Journal of the American Statistical Association, 108(503):929–941, 2013.
    Google ScholarLocate open access versionFindings
  • Laurent Jacques, Jason N Laska, Petros T Boufounos, and Richard G Baraniuk. Robust 1-bit compressive sensing via binary stable embeddings of sparse vectors. IEEE Transactions on Information Theory, 59(4):2082–2102, 2013.
    Google ScholarLocate open access versionFindings
  • Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.
    Google ScholarLocate open access versionFindings
  • William Kautz and Roy Singleton. Nonrandom binary superimposed codes. IEEE Transactions on Information Theory, 10(4):363–377, 1964.
    Google ScholarLocate open access versionFindings
  • Abbas Khalili and Jiahua Chen. Variable selection in finite mixture of regression models. Journal of the american Statistical association, 102(479):1025–1038, 2007.
    Google ScholarLocate open access versionFindings
  • Akshay Krishnamurthy, Arya Mazumdar, Andrew McGregor, and Soumyabrata Pal. Sample complexity of learning mixture of sparse linear regressions. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
    Google ScholarLocate open access versionFindings
  • Percy Liang, Alexandre Bouchard-Cote, Dan Klein, and Ben Taskar. An end-to-end discriminative approach to machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 761–768. Association for Computational Linguistics, 2006.
    Google ScholarLocate open access versionFindings
  • Arya Mazumdar and Soumyabrata Pal. Recovery of sparse signals from a mixture of linear samples. In International Conference on Machine Learning (ICML), 2020.
    Google ScholarLocate open access versionFindings
  • Yaniv Plan and Roman Vershynin. One-bit compressed sensing by linear programming. Communications on Pure and Applied Mathematics, 66(8):1275–1297, 2013.
    Google ScholarLocate open access versionFindings
  • Ariadna Quattoni, Michael Collins, and Trevor Darrell. Conditional random fields for object recognition. In Advances in neural information processing systems, pages 1097–1104, 2005.
    Google ScholarLocate open access versionFindings
  • Miklos Ruszinko. On the upper bound of the size of the r-cover-free families. Journal of Combinatorial Theory, Series A, 66(2):302–310, 1994.
    Google ScholarLocate open access versionFindings
  • Yanyao Shen and Sujay Sanghavi. Iterative least trimmed squares for mixed linear regression. arXiv preprint arXiv:1902.03653, 2019.
    Findings
  • Martin Slawski, Matthias Hein, and Pavlo Lutsik. Matrix factorization with binary components. In Advances in Neural Information Processing Systems, pages 3210–3218, 2013.
    Google ScholarLocate open access versionFindings
  • Weixing Song, Weixin Yao, and Yanru Xing. Robust mixture regression model fitting by laplace distribution. Computational Statistics & Data Analysis, 71:128–137, 2014.
    Google ScholarLocate open access versionFindings
  • Yuekai Sun, Stratis Ioannidis, and Andrea Montanari. Learning mixtures of linear classifiers. In ICML, pages 721–729, 2014.
    Google ScholarLocate open access versionFindings
  • Taiyao Wang and Ioannis Ch Paschalidis. Convergence of parameter estimates for regularized mixed linear regression models. arXiv preprint arXiv:1903.09235, 2019.
    Findings
  • Xinyang Yi, Constantine Caramanis, and Sujay Sanghavi. Solving a mixture of many random linear equations by tensor decomposition and alternating minimization. arXiv preprint arXiv:1608.05749, 2016.
    Findings
  • Dong Yin, Ramtin Pedarsani, Yudong Chen, and Kannan Ramchandran. Learning mixtures of sparse linear regressions using sparse graph codes. IEEE Transactions on Information Theory, 65(3):1430–1451, 2019.
    Google ScholarLocate open access versionFindings
  • Hong-Tu Zhu and Heping Zhang. Hypothesis testing in mixture regression models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 66(1):3–16, 2004.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科