In this work, we consider the Bayesian optimization (BO) approach for parametric tuning of complex chaotic systems. Such problems arise, for instance, in tuning the sub-grid-scale parameterizations in weather and climate models. For such problems, the tuning procedure is generally based on a performance metric which measures how well the tuned model fits the data. This tuning is often a computationally expensive task. We show that BO, as a tool for finding the extrema of computationally expensive objective functions, is suitable for such tuning tasks. In the experiments, we consider tuning parameters of two systems: a simplified atmospheric model and a low-dimensional chaotic system. We show that BO is able to tune parameters of both the systems with a low number of objective function evaluations.

Cluster analysis by nonnegative low-rank approximations has experienced a remarkable progress in the past decade. However, the majority of such approximation approaches are still restricted to nonnegative matrix factorization (NMF) and suffer from the following two drawbacks: 1) they are unable to produce balanced partitions for large-scale manifold data which are common in real-world clustering tasks; 2) most existing NMF-type clustering methods cannot automatically determine the number of clusters. We propose a new low-rank learning method to address these two problems, which is beyond matrix factorization. Our method approximately decomposes a sparse input similarity in a normalized way and its objective can be used to learn both cluster assignments and the number of clusters. For efficient optimization, we use a relaxed formulation based on Data-Cluster-Data random walk, which is also shown to be equivalent to low-rank factorization of the doubly-stochastically normalized cluster incidence matrix. The probabilistic cluster assignments can thus be learned with a multiplicative majorization-minimization algorithm. Experimental results show that the new method is more accurate both in terms of clustering large-scale manifold data sets and of selecting the number of clusters.

Affective classification and retrieval of multimedia such as audio, image, and video have become emerging research areas in recent years. The previous research focused on designing features and developing feature extraction methods. Generally, a multimedia content can be represented with different feature representations (i.e., views). However, the most suitable feature representation related to people׳s emotions is usually not known a priori. We propose here a novel Bayesian multiple kernel learning algorithm for affective classification and retrieval tasks. The proposed method can make use of different representations simultaneously (i.e., multiview learning) to obtain a better prediction performance than using a single feature representation (i.e., single-view learning) or a subset of features, with the advantage of automatic feature selections. In particular, our algorithm has been implemented within a multilabel setup to capture the correlation between emotions, and the Bayesian formulation enables our method to produce probabilistic outputs for measuring a set of emotions triggered by a single image. As a case study, we perform classification and retrieval experiments with our algorithm for predicting people׳s emotional states evoked by images, using generic low-level image features. The empirical results with our approach on the widely-used International Affective Picture System (IAPS) data set outperform several existing methods in terms of classification performance and results interpretability.

Data visualization is one of the major applications of nonlinear dimensionality reduction. From the information retrieval perspective, the quality of a visualization can be evaluated by considering the extent that the neighborhood relation of each data point is maintained while the number of unrelated points that are retrieved is minimized. This property can be quantified as a trade-off between the mean precision and mean recall of the visualization. While there have been some approaches to formulate the visualization objective directly as a weighted sum of the precision and recall, there is no systematic way to determine the optimal trade-off between these two nor a clear interpretation of the optimal value. In this paper, we investigate the properties of $α$-divergence for information visualization, focusing our attention on a particular range of $α$ values. We show that the minimization of the new cost function corresponds to maximizing a geometric mean between precision and recall, parameterized by $α$. Contrary to some earlier methods, no hand-tuning is needed, but we can rigorously estimate the optimal value of $α$ for a given input data. For this, we provide a statistical framework using a novel distribution called Exponential Divergence with Augmentation (EDA). By the extensive set of experiments, we show that the optimal value of $α$, obtained by EDA corresponds to the optimal trade-off between the precision and recall for a given data distribution.

Information divergence that measures the difference between two nonnegative matrices or tensors has found its use in a variety of machine learning problems. Examples are Nonnegative Matrix/Tensor Factorization, Stochastic Neighbor Embedding, topic models, and Bayesian network optimization. The success of such a learning task depends heavily on a suitable divergence. A large variety of divergences have been suggested and analyzed, but very few results are available for an objective choice of the optimal divergence for a given task. Here we present a framework that facilitates automatic selection of the best divergence among a given family, based on standard maximum likelihood estimation. We first propose an approximated Tweedie distribution for the β-divergence family. Selecting the best β then becomes a machine learning problem solved by maximum likelihood. Next, we reformulate α-divergence in terms of β-divergence, which enables automatic selection of α by maximum likelihood with reuse of the learning principle for β-divergence. Furthermore, we show the connections between γ- and β-divergences as well as Renyi- and α-divergences, such that our automatic selection framework is extended to non-separable divergences. Experiments on both synthetic and real-world data demonstrate that our method can quite accurately select information divergence across different learning problems and various divergence families.

Many modern clustering methods employ a non-convex objective function and use iterative optimization algorithms to find local minima. Thus initialization of the algorithms is very important. Conventionally the starting guess of the iterations is randomly chosen; however, such a simple initialization often leads to poor clusterings. Here we propose a new method to improve cluster analysis by combining a set of clustering methods. Different from other aggregation approaches, which seek for consensus partitions, the participating methods in our method are used consequently, providing initializations for each other. We present a hierarchy, from simple to comprehensive, for different levels of such co-initializations. Extensive experimental results on real- world datasets show that a higher level of initialization often leads to better clusterings. Especially, the proposed strategy is more effective for complex clustering objectives such as our recent cluster analysis method by low-rank doubly stochastic matrix decomposition (called DCD). Empirical comparison with three ensemble clustering methods that seek consensus clusters confirms the superiority of improved DCD using co- initialization. (C) 2014 Elsevier B.V. All rights reserved.

In Nonnegative Matrix Factorization (NMF), a nonnegative matrix is approximated by a product of lower-rank factorizing matrices. Quadratic Nonnegative Matrix Factorization (QNMF) is a new class of NMF methods where some factorizing matrices occur twice in the approximation. QNMF finds its applications in graph partition, bi-clustering, graph matching, etc. However, the original QNMF algorithms employ constant multiplicative update rules and thus have mediocre convergence speed. Here we propose an adaptive multiplicative algorithm for QNMF which is not only theoretically convergent but also significantly faster than the original implementation. An adaptive exponent scheme has been adopted for our method instead of the old constant ones, which enables larger learning steps for improved efficiency. The proposed method is general and thus can be applied to QNMF with a variety of factorization forms and with the most commonly used approximation error measures. We have performed extensive experiments, where the results demonstrate that the new method is effective in various QNMF applications on both synthetic and real-world datasets.

With an ever growing number of published scientific studies, there is a need for automated search methods, able to collect and extract as much information as possible from those articles. We propose a framework for the extraction and characterization of brain activity areas published in neuroscientific reports, as well as a suitable clustering strategy of said areas. We further show that it is possible to obtain three-dimensional summarizing brain maps, accounting for a particular topic within those studies. After, using the text information from the articles, we characterize such maps. As an illustrative experiment, we demonstrate the proposed mining approach in fMRI reports of default mode networks. The proposed method hints at the possibility of searching for both visual and textual keywords in neuro atlases.

Abstract. In this work, we consider the Bayesian optimization (BO) approach for tuning parameters of complex chaotic systems. Such problems arise, for instance, in tuning the sub-grid scale parameterizations in weather and climate models. For such problems, the tuning procedure is generally based on a performance metric which measures how well the tuned model fits the data. This tuning is often a computationally expensive task. We show that BO, as a tool for finding the extrema of computationally expensive objective functions, is suitable for such tuning tasks. In the experiments, we consider tuning parameters of two systems: a simplified atmospheric model and a low-dimensional chaotic system. We show that BO is able to tune parameters of both the systems with a low number of objective function evaluations and without the need of any gradient information.

In this paper we introduce a novel approach to the problem of decomposition of auditory evoked fields (AEF), measured by magnetoencephalography (MEG), into basic components. This approach is based on independent component analysis (ICA), that separates components according to the kurtosis of their amplitude distribution over time. The fixed-point algorithm used extracts one independent component at a time, allowing the combination of a high resolution 122-channel whole-scalp neuromagnetometer, to a fast and very efficient implementation of ICA.We show that the independent components found on averaged auditory evoked fields isolate the main response (around 100ms) from subsequent components (around 200ms), and can discriminate between the contra- and the ipsilateral responses from the brain, in a clear dipolar manner, agreeing with the classical source modeling in AEF's.

Several neural algorithms for Independent Component Analysis (ICA) have been introduced lately, but their computational properties have not yet been systematically studied. In this paper, we compare the accuracy, convergence speed, computational load, and other properties of five prominent neural or semi-neural ICA algorithms. The comparison reveals some interesting differences between the algorithms.

Results on a comparison of adaptive recognition techniques for on-line recognition of handwritten Latin alphabets are presented. The classification strategies compared are based on first compressing or distilling a large database of handwritten characters to a small set of character prototypes. Each adaptive classifier then either modifies the original prototypes or conditionally adds new prototypes when they become available from the user of the system. In each case, the classification decision uses the 1-Nearest Neighbor (1-NN) rule for the distances between the input character and the stored prototypes. The distances are calculated using Dynamic Time Warping (DTW). One of the adaptive learning strategies features an extension of the neural Learning Vector Quantization (LVQ) algorithm to the DTW distance metric. All the methods concerned exhibit automatic unsupervised learning from user input simultaneously with the normal mode of operation. The presented experiments show that the assessed methods produce different tradeoffs between the accuracy and complexity of classification. Every version is, however, able to adapt to the user’s writing style with only a very few — say some tens of — handwritten characters.

Images usually convey information that can influence people’s emotional states. Such affective information can be used by search engines and social networks for better understanding the user’s preferences. We propose here a novel Bayesian multiple kernel learning method for predicting the emotions evoked by images. The proposed method can make use of different image features simultaneously to obtain a better prediction performance, with the advantage of automatically selecting important features. Specifically, our method has been implemented within a multilabel setup in order to capture the correlations between emotions. Due to its probabilistic nature, our method is also able to produce probabilistic outputs for measuring a distribution of emotional intensities. The experimental results on the International Affective Picture System (IAPS) dataset show that the proposed approach achieves a bette classification performance and provides a more interpretable feature selection capability than the state-of-the-art methods.