We propose a knowledge-informed variant for learning covariance-dependent data representations using unsupervised vector quantization. In particular, we consider linear data mappings included in vector quantization models, such as c-means++, neural gas, or self-organizing maps, to achieve representations in a lower-dimensional data space. To this end, we show how additional data knowledge can be integrated into the models. The additional data structure information is used to generate an appropriate data mapping depending on the corresponding structured data covariances.
We introduce SOUMAP (Self-Organizing Uniform Manifold Approximation and Projection) as an alternative to regular Self-Organizing Map (SOM) learning intended to improve clustering of the resulting output space. Improvement is achieved by abandoning the SOM's rigid lattice structure in favor of a more expressive output topology afforded by Uniform Manifold Approximation and Projection (UMAP), which is incrementally learned in conjunction with SOUMAP prototypes. As in regular SOM learning, the Hebbian connection formed between input and output spaces results in a topologically trustworthy low-dimensional embedding amenable to clustering. Through controlled experiments we show that SOUMAP's more expressive "lattice" improves the quality of clusterings obtained from it.
While high dimensionality and the selection of meaningful features is usually a burden in machine learning, it is even more so in the case of unsupervised learning and particularly in clustering. The presence of uninformative features may bias significantly the results of distance-based methods such as K-means for instance. For tackling this issue, different versions of sparse K-means have been introduced, building on the idea of adding some penalty terms in the loss function and resulting into automatic feature selection and/or weighting. This paper investigates the connections between some of these methods, and particularly the differences induced by the choices of the penalty terms. It also illustrates the algorithms and model selection tools made available through a recently implemented R package, vimpclust.
Data quality control is fundamental in data-driven analysis with machine learning (ML) models. In the domain of drug research, there is an increasing interest in the prediction of relevant biocompounds physicochemical properties with ML. In order to build predictive models of good quality, it is important to adequately select representative datasets. In this work, we combine ML prediction and Self-Organizing Maps-based exploration to build an interpretable machine learning model and to characterize those data that are most difficult to predict in the validation stage.
In this paper we propose the application of min-max-neurons for the use in generalized learning vector quantization (GLVQ) models, which correspond to min-max-prototypes. These prototypes can be identified with hyperboxes in the data space. Keeping the general GLVQ cost function, we redefine the Hebb-responsibilities for min-max-prototypes and derive consistent learning rules for stochastic gradient descent learning. We demonstrate that the resulting hyperbox-based GLVQ is capable to solve two illustrating toy classification tasks in robust manner, which can be dedicated to the use of robust min-max-prototypes. Finally, we give suggestions for future research for GLVQ based on min-max-prototypes.
Outlier detection means to characterize the distribution of inliers exactly enough such that outliers stand out. A natural and interpretable model is an enclosing ball that includes non-outliers but excludes outliers. However, a single enclosing ball is insufficient for data with multiple, distinct clusters or modes. Therefore, we propose a model with K enclosing balls, each of which can cover a separate cluster of data. We investigate two optimization schemes, one based on stochastic gradient descent and one based on alternating optimization and quadratic programming. On synthetic and real-world data we compare our proposed K minimum enclosing ball (K-MEB) approach to baselines from the literature.
A recent paradigm called Regeneration Learning addresses generative problems where the target data (e.g., images) is more complex than the available input source. While current cross-modal representation and regeneration learning rely on supervised deep learning models, this paper aims to revisit the adequacy of unsupervised models in this field. In this regard, we propose a new unsupervised approach that utilizes the SOM as a heteroassociative memory model to learn cross-modal representations in a topologically coherent map. This approach enables bidirectional predictive/regenerative mapping between domains. We evaluate the potential of this method for an unsolved (so far!) practical problem in petroleum geoscience.
This proof-of-concept work generalizes the concept of invariance, as used in contrastive learning, to fully probabilistic models (such as, e.g., mixture models) that explicitly describe data distributions in an interpretable fashion, and whose main applications are density estimation (e.g., outlier detection), sampling and tractable inference. Invariance allows allows probabilistic models to operate at a lower effective model complexity, and therefore to deal with more complex (image) data. In this article, we propose iGMM, a Gaussian Mixture Model (GMM) that explicitly incorporates invariance into its loss, which is a generalization of the conventional GMM log-likelihood. When constructing hierarchies of conventional GMM and iGMM instances, we obtain invariance properties that are reminiscent of simple and complex cells in the mammalian visual cortex. We show, by experiments on the MNIST and FashionMNIST dataset, that GMM-iGMM hierarchies can faithfully sample from learned data distributions even if the iGMM is invariant to some aspects of the data, and demonstrate that outlier detection performance is strongly enhanced in GMM-iGMM hierarchies.
In remote data collection from sampling stations, a vehicle must be within sufficient distance from a particular station for a predefined minimal time to retrieve required data from the site. The planning task is to find a cost-efficient data collection plan to retrieve data from all the stations. For a fixed-wing aerial vehicle flying with a constant forward velocity, the problem is to determine the shortest feasible path that visits every sensing site and ensure the vehicle is within a reliable communication distance from the station for a sufficient period. We propose to formulate the planning problem as a variant of the Close Enough Dubins Traveling Salesman Problem with Time Constraints (CEDTSP-TC) that is heuristically solved by unsupervised learning of the Growing Self-Organizing Array (GSOA) modified to address the constrained minimal data retrieving time. The proposed method is compared with a baseline based on a sampling-based decoupled approach, and the results support the feasibility of both proposed solvers in random instances.
We introduce a framework for selecting the number of codebook vectors in a vector quantizer based on local characteristics of the data density, the degree to which the process of VQ distorts the representation of this density, and the theoretical efficiency of estimators of these densities. In our analysis, L-2 theory from kernel density estimation relates the number of VQ prototypes to observed sample size, dimension, and complexity, all of which intuitively influence codebook sizing.
Variable importance determination refers to the challenge to identify the most relevant input dimensions or features for a given learning task and quantify their relevance, either with respect to a local decision or a global model. Feature relevance determination constitutes a foundation for feature selection, and it enables an intuitive insight into the rational of model decisions. Indeed, it constitutes one of the oldest and most prominent explanation technologies for machine learning models with relevance for both, deep and shallow networks. A huge number of measures have been proposed such as mutual information, permutation feature importance, deep lift, LIME, GMLVQ, or Shapley values, to name just a few. Within the talk, I will address recent extensions of feature relevance determination, which occur as machine learning models are increasingly used in everyday life. Here, models face an open environment, possibly changing dynamics, and the necessity of model adaptation to account for changes of the underlying distribution. At present, feature relevance determination almost solely focusses on static scenarios and batch training. In the talk, I will target the question of how to efficiently and effectively accompany a model which learns incrementally by feature relevance determination methods [1, 3]. As a second challenge, features are often not mutually independent, and the relevance of groups rather than single features should be judged. While mathematical models such as Shapley values take feature correlations into account for individual additive feature relevance terms, it is unclear how to efficiently and effectively extend those to groups of features. In the talk, I will discuss novel methods for the efficient computation of feature interaction indices [2, 4].
In numerous machine learning algorithms, the computation of eigenvectors and eigenvalues plays a pivotal role in optimizing models. Depending on the specific use case, it is frequently adequate to compute only a subset of dominant eigenvectors or utilize estimations. Handling this task for large matrices poses a challenge, as standard machine learning packages often lack suitable implementations. We explores various techniques for approximating dominant eigenvectors in the context of potentially large symmetric, real-valued matrices and offer an overview of established methods, analyzing their potentials and limitations, including implementation details.
Knowledge Graph Completion via Link Prediction (LP) is an important topic for current studies. In this paper, we present a state-of-the-art method to support decision-making within the context of project planning in small and medium-sized enterprises. A particular focus is on creating a suitable ontology based on real company data, which serves as a basis for LP. This work introduces a novel approach to assist managers in decision making for better use of human resources. By accurately predicting which teams are suitable to work on a new project, we aim to significantly reduce the time needed for manual assignment.
Topic modeling is a key technique for understanding the content of collections of scientific papers. However, commonly used methods like LDA (Latent Dirichlet Allocation) have significant drawbacks, including complex parameter settings. Additionally, these methods often yield low-quality results. Therefore, improving the outcomes of topic modeling is a crucial goal. In this paper, we compare the performance of LDA with a recent topic modeling approach we have developed, which relies on a combination of neural clustering and feature maximization (abbreviated as CFMf). Subsequently, we demonstrate how the feature ranking component of the CFMf method can be used to substantially enhance the performance of LDA, regardless of the number of topics. We also highlight the benefits of post-processing the clustering results before modeling topics in the CFMf approach. Our reference dataset consists of 16,917 full-text articles on the philosophy of science.
The curse of dimensionality refers to phenomena occurring with increasing dimensionality such as marginal differences in distances. Projection pursuit solves this issue by projecting high-dimensional data into a low-dimensional space where meaningful distances allow unbiased function estimation. However, projection pursuit only considers projections onto lines and the unbiased function depends on the sample size. We introduce deep projection pursuit (DPP) to remedy these limitations by using an ensemble of projections on parameterized surfaces combined with neural networks to solve the learning task. Furthermore, we demonstrate the capabilities of the DPP framework by training principal component curves and solving supervised tasks with interpretable models. Finally, we show the ability to maintain group properties in the projection space. Due to these applications, deep projection pursuit is a flexible design paradigm with various use cases.
In this work, we will investigate two approaches to deploy learning rules. A combination of these approaches is used to create a generic learning rule for prototype-based models with the emphasis on interpretability. In this regard, we will show how the learning rules are associated to the underlying decision making of such models. Moreover, the work concludes by giving possible interpretations of these rules and anchor points for developing related explanations and designing comprehensible learning rules.
To achieve state-of-the-art performance, deep learning models are becoming increasingly complex, leading to a significant increase in demand for high-performance computing resources and, in turn, concerns about environmental impact. In this context, the concept of Green AI has been proposed, which advocates optimizing performance by improving model efficiency, rather than relying solely on an increase in computing resources, thereby reducing the impact on the environment. Object detection is a research hotspot in the field of computer vision. This paper focuses on the commonly used YOLOv5 network in object detection, optimizing the YOLOv5 model through pruning and the use of lightweight networks. Implemented on a campus image dataset, a balance between reducing computational load and maintaining accuracy was achieved. The experimental results confirm that strategic model pruning and thoughtful network architecture selection can produce environmentally responsible and computationally efficient deep learning models without significantly reducing performance, aligning with the goals of GreenAI. All related code of the project is available at: https://github.com/xbgthws/Green-AI-project.git.
The use of urinary steroid metabolomics (USM) in combination with machine learning in endocrinology is briefly introduced. We demonstrate the usefulness of the approach for the detection and differential diagnosis of Primary Aldosteronism (PA), which has been addressed in a recent retrospective study. Here, we mainly present results for the application of the prototype based Generalized Matrix Relevance Learning Vector Quantization for the classification of steroid metabolomics profiles. The method allows for the successful diagnosis of PA and provides insights into the importance of the available markers. Moreover, it facilitates the non-invasive identification of a subtype of PA which is associated with adrenal adenoma harboring a particular mutation in the tumor tissue.
This contribution shows that Gaussian Mixture Models can be considered generalizations of self-organizing maps. More precisely, we demonstrate that the training of self-organizing maps is an approximation to the training of Gaussian Mixture Models by gradient descent. As a consequence, the scores of a trained SOM can be treated as log-likelihoods of a GMM with tied, spherical covariance and used, e.g., for outlier detection, whereas sampling from trained SOMs is not welldefined. Furthermore, we outline how SGD-trained GMMs can be generalized to diagonal and more expressive covariance matrices and how this benefits typical data science applications such as outlier detection, sampling and generative classification. Source codes are available on the author's web site or upon request.
