Fast agnostic classification

A learning algorithm is agnostic if it doesn’t presume a perfect model of how input data produce output data. Such algorithms are difficult to design, even for the basic task of classifying data as well as the best linear separator. This has led to a persistent rift between practice and theory: popular algorithms, such as SVMs and logistic regression, are susceptible to noise; provable agnostic algorithms involve brute-force sampling (which uses too much time) or fitting polynomials (which uses too much data.) We recently introduced a new classification algorithm, KG, which is both practical and agnostic. It revisits basic elements of learning: 1. What functions should the algorithm fit? Smooth lists of halfspaces are a novel generalization of halfspaces. They are more flexible than halfspaces, but do not require more data to train in the worst case. 2. How should the algorithm fit such a function to the data? The algorithm involves ‘immutable’ iterations which are fundamentally different than update rules such as gradient descent, multiplicative weights, or perceptrons. KG achieves promising experimental performance for both natural and artificial problems. We seek to deepen our theoretical understanding of the algorithm and expand its practical applications. The main question we shall answer is: when is KG provably fast? It eventually converges to the correct solution for a wide variety of input distributions. However, these intersect with a litany of hardness results, so restricting the input distribution seems necessary. Based on experimental evidence and the mechanics of the algorithm, we believe it is possible the algorithm runs in polynomial time when the inputs are normally distributed. If so, this algorithm would solve a notorious problem in computer science: learning logarithmically-sparse parities with noise. This would resolve a variety of challenges in learning theory, such as learning DNFs (encountered in 1984 by Valiant) and learning log-juntas (the subject of a prize offered in 2003 by Blum). As exciting as this possibility seems, it does not contradict known hardness results, nor does it upset the consensus on related problems in cryptography or complexity theory. We propose to gain more experimental and theoretical evidence for this possibility. In practice, many classification tasks involve multiple classes. When the number of classes is large, we do not believe fast agnostic classification is possible. We posit stronger lower bounds for classification with a growing number of classes which depend on P 6= NP rather than weaker conjectures about refuting random constraint satisfaction problems. We believe the problem remains challenging even when the inputs are normally distributed. This is due to close relations with the learning with errors (LWE) problem, which underpins much of modern cryptography. The difficulty of LWE and its variants depends heavily on its parameters, including the dimension of the inputs, the amount of data, the number of classes, and the amount of noise. For some choices, it is as hard as worst-case lattice problems; for others, it may be solved in polynomial time. We propose a generalization of KG for multiple classes. It may substantially improve upon logistic regression and multiclass SVM for many practical applications. It may also experimentally validate the range of safe parameters for lattice-based cryptography.