We analyze and improve a recent strategy to train a quantile regression model by minimizing an expected pinball loss over all quantiles. Through an asymptotic convergence analysis, we show that minimizing the expected pinball loss can be more efficient at estimating single quantiles than training with the standard pinball loss for that quantile, an insight that generalizes the known deficiencies of the sample quantile in the unconditioned setting. Then, to guarantee a legitimate inverse CDF, we propose using flexible deep lattice networks with a monotonicity constraint on the quantile input to guarantee non-crossing quantiles, and show lattice models can be regularized to the same location-scale family. Our analysis and experiments on simulated and real datasets show that the proposed method produces state-of-the-art legitimate inverse CDF estimates that are likely to be as good or better for specific target quantiles.

We investigate different methods for regularizing quantile regression when predicting either a subset of quantiles or the full inverse CDF. We show that minimizing an expected pinball loss over a continuous distribution of quantiles is a good regularizer even when only predicting a specific quantile. For predicting multiple quantiles, we propose achieving the classic goal of non-crossing quantiles by using deep lattice networks that treat the quantile as a monotonic input feature, and we discuss why monotonicity on other features is an apt regularizer for quantile regression. We show that lattice models enable regularizing the predicted distribution to a location-scale family. Lastly, we propose applying rate constraints to improve the calibration of the quantile predictions on specific subsets of interest and improve fairness metrics. We demonstrate our contributions on simulations, benchmark datasets, and real quantile regression problems.

We present fast implementations of linear interpolation operators for piecewise linear functions and multi-dimensional look-up tables. These operators are common for efficient transformations in image processing and are the core operations needed for lattice models like deep lattice networks, a popular machine learning function class for interpretable, shape-constrained machine learning. We present new strategies for an efficient compiler-based solution using MLIR to accelerate linear interpolation. For real-world machine-learned multi-layer lattice models that use multidimensional linear interpolation, we show these strategies run 5-10× faster on a standard CPU compared to an optimized C++ interpreter implementation.

We present fast implementations of linear interpolation operators for both piecewise linear functions and multi-dimensional look-up tables. We use a compiler-based solution (using MLIR) for accelerating this family of workloads. On real-world multi-layer lattice models and a standard CPU, we show these strategies deliver $5-10\times$ faster runtimes compared to a C++ interpreter implementation that uses prior techniques, producing runtimes that are 1000s of times faster than TensorFlow 2.0 for single evaluations.

Models of categorization make different representational assumptions, with categories being represented by prototypes, sets of exemplars, and everything in between. Rational models of categorization justify these representational assumptions in terms of different schemes for estimating probability distributions. However, they do not answer the question of which scheme should be used in representing a given category. We show that existing rational models of categorization are special cases of a statistical model called the hierarchical Dirichlet process, which can be used to automatically infer a representation of the appropriate complexity for a given category.

We investigate machine learning models that can provide diminishing returns and accelerating returns guarantees to capture prior knowledge or policies about how outputs should depend on inputs. We show that one can build flexible, nonlinear, multi-dimensional models using lattice functions with any combination of concavity/convexity and monotonicity constraints on any subsets of features, and compare to new shape-constrained neural networks. We demonstrate on real-world examples that these shape constrained models can provide tuning-free regularization and improve model understandability.

We propose learning deep models that are monotonic with respect to a user-specified set of inputs by alternating layers of linear embeddings, ensembles of lattices, and calibrators (piecewise linear functions), with appropriate constraints for monotonicity, and jointly training the resulting network. We implement the layers and projections with new computational graph nodes in TensorFlow and use the Adam optimizer and batched stochastic gradients. Experiments on benchmark and real-world datasets show that six-layer monotonic deep lattice networks achieve state-of-the art performance for classification and regression with monotonicity guarantees.

For many machine learning problems, there are some inputs that are known to be positively (or negatively) related to the output, and in such cases training the model to respect that monotonic relationship can provide regularization, and makes the model more interpretable. However, flexible monotonic functions are computationally challenging to learn beyond a few features. We break through this barrier by learning ensembles of monotonic calibrated interpolated look-up tables (lattices). A key contribution is an automated algorithm for selecting feature subsets for the ensemble base models. We demonstrate that compared to random forests, these ensembles produce similar or better accuracy, while providing guaranteed monotonicity consistent with prior knowledge, smaller model size and faster evaluation.

Practical applications of machine learning often involve successive training iterations with changes to features and training examples. Ideally, changes in the output of any new model should only be improvements (wins) over the previous iteration, but in practice the predictions may change neutrally for many examples, resulting in extra net-zero wins and losses, referred to as unnecessary churn. These changes in the predictions are problematic for usability for some applications, and make it harder and more expensive to measure if a change is statistically significant positive. In this paper, we formulate the problem and present a stabilization operator to regularize a classifier towards a previous classifier. We use a Markov chain Monte Carlo stabilization operator to produce a model with more consistent predictions without adversely affecting accuracy. We investigate the properties of the proposal with theoretical analysis. Experiments on benchmark datasets for different classification algorithms demonstrate the method and the resulting reduction in churn.

Real-world machine learning applications may have requirements beyond accuracy, such as fast evaluation times and interpretability. In particular, guaranteed monotonicity of the learned function with respect to some of the inputs can be critical for user confidence. We propose meeting these goals for low-dimensional machine learning problems by learning flexible, monotonic functions using calibrated interpolated look-up tables. We extend the structural risk minimization framework of lattice regression to monotonic functions by adding linear inequality constraints. In addition, we propose jointly learning interpretable calibrations of each feature to normalize continuous features and handle categorical or missing data, at the cost of making the objective non-convex. We address large-scale learning through parallelization, mini-batching, and random sampling of additive regularizer terms. Case studies on real-world problems with up to sixteen features and up to hundreds of millions of training samples demonstrate the proposed monotonic functions can achieve state-of-the-art accuracy in practice while providing greater transparency to users.

We explored people's inductive biases in category learning--that is, the factors that make learning category structures easy or hard--using iterated learning. This method uses the responses of one participant to train the next, simulating cultural transmission and converging on category structures that people find easy to learn. We applied this method to four different stimulus sets, varying in the identifiability of their underlying dimensions. The results of iterated learning provide an unusually clear picture of people's inductive biases. The category structures that emerge often correspond to a linear boundary on a single dimension, when such a dimension can be identified. However, other kinds of category structures also appear, depending on the nature of the stimuli. The results from this single experiment are consistent with previous empirical findings that were gleaned from decades of research into human category learning.

We describe a new, simplified, and general analysis of a fusion of Nesterov's accelerated gradient with parallel coordinate descent. The resulting algorithm, which we call BOOM, for boo sting with m omentum, enjoys the merits of both techniques. Namely, BOOM retains the momentum and convergence properties of the accelerated gradient method while taking into account the curvature of the objective function. We describe a distributed implementation of BOOM which is suitable for massive high dimensional datasets. We show experimentally that BOOM is especially effective in large scale learning problems with rare yet informative features.

A task of primary importance for social network users is to decide whose updates to subscribe to in order to maximize the relevance, credibility, and quality of the information received. To address this problem, we conducted an experiment designed to measure the extent to which different factors in online social networks affect both explicit and implicit judgments of credibility. The results of the study indicate that both the topical content of information sources and social network structure affect source credibility. Based on these results, we designed a novel method of automatically identifying and ranking social network users according to their relevance and expertise for a given topic. We performed empirical studies to compare a variety of alternative ranking algorithms and a proprietary service provided by a commercial website specifically designed for the same purpose. Our findings show a great potential for automatically identifying and ranking credible users for any given topic.

Grow your own representations: Computational constructivism Joseph L. Austerweil (Joseph.Austerweil@gmail.com) Robert L. Goldstone (rgoldsto@indiana.edu) Thomas L. Griffiths (Tom Griffiths@berkeley.edu) Todd Gureckis (todd.gureckis@nyu.edu) Kevin Canini (kevin@eecs.berkeley.edu) Matt Jones (mcj@colorado.edu) Keywords: representational change, Bayesian modeling, Connectionism, features, categories From a cognitivist standpoint, one main interest of psy- chology is the study of representations of the human mind as they mediate how people react to stimuli in their environment (Palmer, 1978). This can explain why two people that en- counter the same stimulus can behave in very different ways (Chomsky, 1959). For example, an art historian viewing a Jackson Pollock painting may exclaim “this is beautiful” due to her representation of his work as a rejection of painting with a brush; however, a lay person may say “this is ugly” due to his representation of the painting as a cluttered mess of col- ors. Without knowledge of the representations of each person in this example, it would be nearly impossible to explain their behavior when interacting with the Jackson Pollock painting. Over the last three decades, cognitive psychologists have demonstrated that the representations people use can change flexibly to capture changes in their environment (Hoffman & Richards, 1985; Schyns, Goldstone, & Thilbaut, 1998; Gold- stone, 2003). However, if the representations we use are de- termined by the stimuli in our environment, this threatens the explanatory utility of representations as it could be superflu- ous to use representations to explain people’s reaction to stim- uli if the representations are determined by the stimuli. Thus, cognitive psychologists need to explicitly formulate how rep- resentations change with experience. Although computational modelers, from connectionists to Bayesians, disagree on many things, one thing they do agree on is the importance of representations in their models (Mc- Clelland et al., 2010; Griffiths, Chater, Kemp, Perfors, & Tenenbaum, 2010). Recently, there has been a growing in- terest in exploring computational models that adapt their rep- resentations with experience in ways that match this human capacity. In this symposium, we explore computational mod- els that adapt their representations with experience in ways that are inspired by the human capability. Recently, there have been several proposals for computa- tional models whose representations flexibly adapt to the in- put data like people do; however, there has not been a thor- ough comparison of the different models. The goal of the symposium is the compare and contrast the different meth- ods, evaluate their ability to capture of human representation learning, and make explicit what is meant in each model by “representation change” as this can be a controversial claim (Schyns et al., 1998). Currently, it is not clear whether or not the different proposals mean the same thing by a “representa- tion” and if they are competing proposals to explain the same aspect of human cognition or different levels of explanation. Thus, the symposium will emphasize understanding what is meant by representation change and how well each model can explain human representation change. The symposium will focus on a wide variety of methods for representation learning from some of the most popular computational paradigms in computational cognitive science: nonparametric Bayesian modeling (Austerweil & Griffiths; Canini & Griffiths), connectionist modeling (Gureckis; Gold- stone), and reinforcement learning (Jones). Importantly, each presenter will focus on how their computational proposals ex- plain human experimental data and discussing what exactly is a representation in their framework and how they are inferred. Thus, the symposium should be interesting to a broad audi- ence of cognitive scientists (from computation modelers to experimentalists to philosphers). We hope it inspires a growth of new computational models and human experiments in this underdeveloped, yet incredibly important, aspect of cognitive science. Introduction and Nonparametric Bayesian Models of Fea- ture Learning Austerweil and Griffiths Cognitive psychology is concerned primarly with representations and how they mediate the re- sponse to stimuli. In this talk, we present a framework for exploring the principles underlying human feature learning using nonparametric Bayesian statistics. We show that our framework can capture how people infer features using sta- tistical information of the observed images, spatial informa- tion from the observed images, and categorization cues. Next, we extend our initial framework to infer features that are in- variant over a set of transformations and demonstrate that the model infers new invariant features like people do. Although most shapes and features can be transformed by translations and rescalings, some shapes and features lose their identity when rotated. We show how our model is easily extended to capture how people infer the allowable set of transformations of an object from their observations of the object. Finally, we conclude with the implications of our framework for refer- ence frames in shape perception and feature-based cognitive models and compare it to other approaches for inferring rep- resentations. Building flexible categorization models by grounding them in perception Goldstone One limitation of most existing models of catego- rization is that they do not start with a perceptually grounded representation of the objects that they categorize. Instead, they use dimensional or featural representations that discard information about the spatial relations among an object’s parts. This restricts the models’ ability to create psycho- logically plausible object representations that can be flexibly adapted to meet categorization demands. I will describe a

Categories are often organized into hierarchical taxonomies, that is, tree structures where each node represents a labeled category, and a node's parent and children are, respectively, the category's supertype and subtypes. A natural question is whether it is possible to reconstruct category taxonomies in cases where we are not given explicit information about how categories are related to each other, but only a sample of observations of the members of each category. In this paper, we introduce a nonparametric Bayesian model of multi-level category learning, an extension of the hierarchical Dirichlet process (HDP) that we call the tree-HDP. We demonstrate the ability of the tree-HDP to reconstruct simulated datasets of artificial taxonomies, and show that it produces similar performance to human learners on a taxonomy inference task.

Categorization, or classification, is a fundamental problem in both cognitive psychology and machine learning. Classical psychological models of categorization fall into two main groups: prototype models and exemplar models, which are equivalent, respectively, to the statistical methods of parametric density estimation and kernel density estimation. Many categorization studies in psychology attempt to understand how people solve this problem by comparing their inferences to those of formal computational models such as prototype or exemplar models. From this perspective, different models make different predictions about the representations and mechanisms people use to make categorization judgments. Instead, one can seek to understand categorization by viewing it as a problem of statistical inference and attempting to characterize the inductive biases of human learners. These inductive biases can be directly exposed using an experimental method called iterated learning, which provides direct insight into human categorization in a way that is independent of any proposed models. I describe the results of an iterated learning study of human categorization which supports previous findings by psychologists that people's representations seem to be more flexible than would be implied by either prototype or exemplar models alone. Prototype and exemplar models both use a single, fixed level of complexity in their representations of categories, with prototype models exhibiting the simplest representations, and exemplar models using the most complex representations. Treating categorization as a type of statistical inference, I describe a family of nonparametric Bayesian models of categorization based on the Dirichlet process mixture model (DPMM). These models represent categories as combinations of clusters of objects and, together, produce a continuum of representational complexities where prototype and exemplar models are special cases, occupying opposite ends of the spectrum. DPMM models allow the level of complexity of category representations to be chosen to suit the task at hand or to change over time; this flexibility can explain psychological results demonstrating that people's inferences are more congruent with prototype models at some times and exemplar models at other times. The DPMM can be generalized into a larger framework of models based on the hierarchical Dirichlet process (HDP). The HDP subsumes the DPMM and multiple previous psychological models, including prototypes, exemplars, and the Rational Model of Categorization. In addition, the HDP contains a family of previously unexplored models which make interesting predictions about how information can be shared between multiple categories. While most other categorization models learn each individual category in isolation and independently of the others, these HDP models share information between categories. This sharing of information can improve the speed and accuracy of learning and explained certain transfer learning effects that were observed in people's judgments. I introduce an extension of the HDP, called the tree-HDP, which is designed to infer systems of hierarchically related categories. The tree-HDP is able to simultaneously learn categories at multiple levels of generality and infer the taxonomic relationships between them. The original scientific contributions of this dissertation are a detailed characterization of the inductive biases of human categorization via iterated learning, a unification of previous psychological models of categorization into a common Bayesian statistical framework (the HDP), a demonstration that this framework contains interesting and previously unexplored models that predict and explain the integration of information from multiple categories, and a proposal and exploration of a new statistical model, the tree-HDP, which can simultaneously learn categories at multiple hierarchical levels and infer taxonomic relationships between those categories.