Methods for analysis of principal components in discrete data have existed for some time under various names such as grade of membership modelling, probabilistic latent semantic analysis, and genotype inference with admixture. In this paper we explore a number of extensions to the common theory, and present some application of these methods to some common statistical tasks. We show that these methods can be interpreted as a discrete version of ICA. We develop a hierarchical version yielding components at different levels of detail, and additional techniques for Gibbs sampling. We compare the algorithms on a text prediction task using support vector machines, and to information retrieval.

In this paper, we apply information theoretic measures to voting in the U.S. Senate in 2003. We assess the associations between pairs of senators and groups of senators based on the votes they cast. For pairs, we use similarity-based methods, including hierarchical clustering and multidimensional scaling. To identify groups of senators, we use principal component analysis. We also apply a discrete multinomial latent variable model that we have developed. In doing so, we identify blocs of cohesive voters within the Senate and contrast it with continuous ideal point methods. We find more nuanced blocs than simply the two-party division. Under the bloc-voting model, the Senate can be interpreted as a weighted vote system, and we are able to estimate the empirical voting power of individual blocs through what-if analysis.

We propose a new prior distribution for classical (nonhierarchical) logistic regression models, constructed by first scaling all nonbinary variables to have mean 0 and standard deviation 0.5, and then placing independent Student-t prior distributions on the coefficients. As a default choice, we recommend the Cauchy distribution with center 0 and scale 2.5, which in the simplest setting is a longer-tailed version of the distribution attained by assuming one-half additional success and one-half additional failure in a logistic regression. Cross-validation on a corpus of datasets shows the Cauchy class of prior distributions to outperform existing implementations of Gaussian and Laplace priors.We recommend this prior distribution as a default choice for routine applied use. It has the advantage of always giving answers, even when there is complete separation in logistic regression (a common problem, even when the sample size is large and the number of predictors is small), and also automatically applying more shrinkage to higher-order interactions. This can be useful in routine data analysis as well as in automated procedures such as chained equations for missing-data imputation.We implement a procedure to fit generalized linear models in R with the Student-t prior distribution by incorporating an approximate EM algorithm into the usual iteratively weighted least squares. We illustrate with several applications, including a series of logistic regressions predicting voting preferences, a small bioassay experiment, and an imputation model for a public health data set.

Art historians and archaeologists have long grappled with the regional classification of ancient Near Eastern ivory carvings. Based on the visual similarity of sculptures, individuals within these fields have proposed object assemblages linked to hypothesized regional production centers. Using quantitative rather than visual methods, we here approach this classification task by exploiting computational methods from machine learning currently used with success in a variety of statistical problems in science and engineering. We first construct a prediction function using 66 categorical features as inputs and regional style as output. The model assigns regional style group (RSG), with 98 percent prediction accuracy. We then rank these features by their mutual information with RSG, quantifying single-feature predictive power. Using the highest- ranking features in combination with nomographic visualization, we have found previously unknown relationships that may aid in the regional classification of these ivories and their interpretation in art historical context.

Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Copy DOI

With the abundance of publicly available data registering the judgements at supreme courts and parliamentary votes, we can employ various data mining techniques to identify interesting patterns. For example, we can identify explicit and implicit voting blocs, which may or may not agree with official party affiliations. We can assess the political strength of those blocs. We can examine the vote of which particular senators is the most representative of the final outcome of the vote. We can employ text mining and visualization tools to cope with a large number of issues discussed in parliaments. While the paper primarily acts as a survey, it demonstrates the utility of several techniques that have not yet been used in the context of law.

We develop a framework and tools for applying a computer‐assisted context analysis system and find that it achieves levels of accuracy comparable to humans for about 80% less effort when starting from scratch (no labeled examples). The system is presented using a case study of Congressional bill titles as a proxy for the full text of Congressional bills. We also demonstrate that the system can use information learned from previous experiments to reduce the labeling requirements still further to over 90% savings of the current human effort. This study assumes that social scientists have a need to locate individual documents in a subject area. To support this need, "Topic classification," where documents are coded according to some organizing framework, is used to facilitate search and summarization. Our proposed framework for effectively employing machine learning methods mitigates the high costs of the standard method of topic classification ‐ human labeling. We scientifically evaluate the efficacy and accuracy of the automated approach using a large corpus of 380,000 human‐labeled events and a classification system that includes 20 major policy topics, 226 subtopics, and a demonstrably strong level of human inter‐ coder agreement.

Last time we set up a game in which we went to a random position in the document and tested for a particular word. This tends to give small expected information, since the answer is usually “no”. A different question we could ask is “Is this word present anywhere in the document?” The expected information we get from this question can be computed from a table of word-presence counts. For example, this is the table for testing if the word “car” is present (in the larger collection of 200 documents):

A topic ontology applies the usual ontological constructs to the task of annotating the topic of a document. The topic is the highly summarized essence of the document. The topics are usually chosen intuitively and rarely questioned. However, we have studied several ways of allocating frequently asked questions from a legal domain into a set of topical sub-domains. Our criteria were: 1) The sub-domains should not overlap. 2) The sub-domain should be objectively identifiable from the words of the text. 3) Which words and grammatical categories can serve as keywords? 4) Can the structure of sub-domains be induced semi-automatically from the text itself?

We propose a simple and efficient approach to building undirected probabilistic classification models (Markov networks) that extend naïve Bayes classifiers and outperform existing directed probabilistic classifiers (Bayesian networks) of similar complexity. Our Markov network model is represented as a set of consistent probability distributions on subsets of variables. Inference with such a model can be done efficiently in closed form for problems like class probability estimation. We also propose a highly efficient Bayesian structure learning algorithm for conditional prediction problems, based on integrating along a hill-climb in the structure space. Our prior based on the degrees of freedom effectively prevents overfitting.

This article presents a unified theory for analysis of components in discrete data, and compares the methods with techniques such as independent component analysis, non-negative matrix factorisation and latent Dirichlet allocation. The main families of algorithms discussed are a variational approximation, Gibbs sampling, and Rao-Blackwellised Gibbs sampling. Applications are presented for voting records from the United States Senate for 2003, and for the Reuters-21578 newswire collection.

In this paper, we present the hierarchical variable dependencies that were obtained from raw data with the use of two machine learning techniques on an ecological data set. The data set contains features of field margins and the corresponding number of spider species inhabiting them. This data set was used before by domain experts to construct a fuzzy qualitative model with hierarchical variable dependencies, which we use for comparison with our results. One of the machine learning methods constructs a hierarchical structure similar to the one in the experts’ model, while revealing some additional interesting relations of environmental features with respect to the number of spider species. The other method constructs a different hierarchy from the one proposed by the experts, which, according to our classification performance experiments, might be even more appropriate.

An ontology is a structured semantic model, composed of concepts, relations and instances. Data is a more primitive but concrete assembly of instances described by their attributes. An example of topic ontology is Open Directory of Web documents used in Google organizing documents into topics and subtopics, such as “Science”, “Arts”, etc. However, many ontologies are originally manually developed without making an explicit connection to the data. We hereby introduce the concept of ontology grounding, where the concepts and relations from the ontology become associated with the data. This enables us both to explain concepts in more concrete terms and to recognize them in the data.

Two attributes $A$ and $B$ are said to interact when it helps to observe the attribute values of both attributes together. This is an example of a $2$-way interaction. In general, a group of attributes ${\cal X}$ is involved in a $k$-way interaction when we cannot reconstruct their relationship merely with $\ell$-way interactions, $\ell < k$. These two definitions formalize the notion of an interaction in a nutshell. An additional notion is the one of context. We interpret context as just another attribute. There are two ways in which we can consider context. Context can be something that specifies our focus: we may examine interactions only in a given context, only for the instances that are in the context. Alternatively, context can be something that we are interested in: if we seek to predict weather, only the interactions involving the weather will be interesting to us. This is especially relevant for classification: we only want to examine the interactions involving the labelled class attribute and other attributes (unless there are missing or uncertain attribute values). But the definitions are not complete. We need to specify the model that assumes the interaction: how to we represent the pattern of co-appearance of several attributes? We also need to specify a model that does not assume the interaction: how do we reconstruct the pattern of co-appearance of several attributes without actually observing them all simultaneously? We need to specify a loss function that measures how good a particular model is, with respect to another model or with respect to the data. We need an algorithm that builds both models from the data. Finally, we need the data in order to assess whether it supports the hypothesis of interaction. The present work shows that mutual information, information gain, correlation, attribute importance, association and many other concepts, are all merely special cases of the above principle. Furthermore, the analysis of interactions generalizes the notions of analysis of variance, variable clustering, structure learning of Bayesian networks, and several other problems. There is an intriguing history of reinvention in the area of information theory on the topic of interactions. In our work, we focus on models founded on probability theory, and employ entropy and Kullback-Leibler divergence as our loss functions. Generally, whether an interaction exists or not, and to what extent, depends on what kind of models we are working with. The concept of McGill's interaction information in information theory, for example, is based upon Kullback-Leibler divergence as the loss function, and non-normalized Kirkwood superposition approximation models. Pearson's correlation coefficient is based on the proportion of explained standard deviation as the loss function, and on the multivariate Gaussian model. Most applications of mutual information are based on Kullback-Leibler divergence and the multinomial model. When there is a limited amount of data, it becomes unclear what model can be used to interpret it. Even if we fix the family of models, we remain uncertain about what would be the best choice of a model in the family. In all, uncertainty pervades the choice of the model. The underlying idea of Bayesian statistics is that the uncertainty about the model is to be handled in the same was as the uncertainty about the correct prediction in nondeterministic domains. The uncertainty, however, implies that we know neither if is an interaction with complete certainty, nor how important is the interaction. We propose a Bayesian approach to performing significance tests: an interaction is significant if it is very unlikely that a model assuming the interaction would suffer a greater loss than a model not assuming it, even if the interaction truly exists, among all the foreseeable posterior models. We also propose Bayesian confidence intervals to assess the probability distribution of the expected loss of assuming that an interaction does not exist. We compare significance tests based on permutations, bootstrapping, cross-validation, Bayesian statistics and asymptotic theory, and find that they often disagree. It is important, therefore, to understand the assumptions that underlie the tests. Interactions are a natural way of understanding the regularities in the data. We propose interaction analysis, a methodology for analyzing the data. It has a long history, but our novel contribution is a series of diagrams that illustrate the discovered interactions in data. The diagrams include information graphs, interaction graphs and dendrograms. We use interactions to identify concept drift and ignorability of missing data. We use interactions to cluster attribute values and build taxonomies automatically. When we say that there is an interaction, we still need to explain what it looks like. Generally, the interaction can be explained by inferring a higher-order construction. For that purpose, we provide visualizations for several models that allow for interactions. We also provide a probabilistic account of rule inference: a rule can be interpreted as a constructed attribute. We also describe interactions involving individual attribute values with other attributes: this can help us break complex attributes down into simpler components. We also provide an approach to handling the curse of dimensionality: we dynamically maintain a structure of attributes as individual attributes are entering our model one by one. We conclude this work by presenting two practical algorithms: an efficient heuristic for selecting attributes within the naive Bayesian classifier, and a complete approach to prediction with interaction models, the Kikuchi-Bayes model. Kikuchi-Bayes combines Bayesian model averaging, a parsimonious prior, and search for interactions that determine the model. Kikuchi-Bayes outperforms most popular machine learning methods, such as classification trees, logistic regression, the naive Bayesian classifier, and sometimes even the support vector machines. However, Kikuchi-Bayes models are highly interpretable and can be easily visualized as interaction graphs.

We propose a simple yet potentially very effective way of visualizing trained support vector machines. Nomograms are an established model visualization technique that can graphically encode the complete model on a single page. The dimensionality of the visualization does not depend on the number of attributes, but merely on the properties of the kernel. To represent the effect of each predictive feature on the log odds ratio scale as required for the nomograms, we employ logistic regression to convert the distance from the separating hyperplane into a probability. Case studies on selected data sets show that for a technique thought to be a black-box, nomograms can clearly expose its internal structure. By providing an easy-to-interpret visualization the analysts can gain insight and study the effects of predictive factors.