As the most popular unsupervised dimensionality reduction method, the existence of outliers easily distorts the subspace projection matrix learned in principal component analysis (PCA). Though many studies have attempted to improve its robustness, it is still facing challenges in how to characterize the sample quality in a more fine-grained way and how to enhance the capability to model data with non-linear structures. In this work, a novel kernel space-based dual-weighted robust principal component analysis (KDRPCA) is proposed by mapping data into kernel space and evaluating samples by employing the sample reconstruction errors as their quality metric. To be specific, samples are partitioned into outliers and normal samples by a binary weight descriptor, and the normal samples are further refined as positive and hard ones by a probabilistic weight vector, leading to a dual-weight strategy. This is completed in the mapped RKHS (reproducing kernel Hilbert space) by jointly measuring the reconstruction error of each sample and the optimal mean of all the weighted samples. As a consequence, the coupled improvements of noise control and kernel mapping enable KDRPCA to have enhanced robustness and non-linear modeling ability. To evaluate the effectiveness of KDRPCA, extensive experiments are conducted on both synthetic and benchmark datasets across multiple aspects, including non-linear modeling capability, evaluation on the rationality of sample weights, analysis of reconstruction errors, clustering performance, and robustness under different noise levels. The experimental results demonstrate that KDRPCA exhibits competitive performance compared to popular PCA variants.
We introduce a new framework for Bayesian inference in mixtures of normal linear regression models with random covariates. Such types of mixtures belong to the category of cluster-weighted models. The proposed Bayesian cluster-weighted model aims to encompass potential heterogeneity in the distribution of the response variable as well as in the multivariate distribution of the covariates for detecting signals relevant to the underlying latent structure. Of particular interest are potential signals originating from: (i) the linear predictor structures of the regression models and (ii) the covariance structures of the covariates. We model these two components using a lasso shrinkage prior for the regression coefficients and a graphical lasso shrinkage prior for the covariance matrices. A fully Bayesian approach is followed for estimating the number of clusters by treating the number of mixture components as random and implementing a trans-dimensional telescoping sampler. Alternative Bayesian approaches based on overfitting mixture models or using information criteria to select the number of components are also considered. The proposed methodology is compared to mixtures of regressions, mixtures of experts, and existing cluster-weighted models in simulation studies and an application to a biomedical dataset.
This study presents methods for automatically detecting target texts in images through temporal and spatial alignment techniques. The temporal alignment is achieved by projecting each image onto its horizontal and vertical axes, followed by applying the dynamic time-warping algorithm to align the two projections independently. For spatial alignment, these projections are utilized to create an aligned image, from which the fixed rank kriging method is employed to extract the covariance structure of the aligned image. Both alignment procedures yield aligned feature sequences for each image. A compressed learning approach is then utilized to identify abnormal signals within these aligned feature sequences. Two specific applications are explored to demonstrate the effectiveness of the methods: handwritten traditional Chinese character classification and RoHS logo classification. The first application faces challenges due to the diverse shapes and styles of handwritten characters, while the second contends with background noise and varying forms of logos. Our numerical results demonstrate that the proposed methods consistently achieve superior or at least competitive identification performance compared with several widely used approaches, including convolutional neural networks, Tesseract optical character recognition (OCR), transformer OCR, EfficientNet-B5, and ResNet50, when the angles of text images are consistent.
Regression coefficients clustering has gained increasing importance in various applications. However, most of the existing studies fail to accommodate skewed or asymmetrically distributed data with heteroscedasticity. In the context of longitudinal data, this paper investigates regression coefficients clustering by combining the asymmetric least squares loss with the multi-direction separation penalty method. The proposed method enables the identification of subgroups in which individuals share similar covariate effects, while allowing different important variables to be selected for different individuals. Meanwhile, the proposed method effectively addresses heteroscedasticity issues and captures more comprehensive distributional characteristics compared to ordinary least squares regression. The paper establishes theoretical properties, including the consistency of the estimator and its oracle property. To efficiently compute the proposed estimator, we develop an algorithm that combines cyclic coordinate descent with the alternating direction method of multipliers. Simulation studies and a practical example are provided to demonstrate the superiority of the proposed approach.
A non-Markovian framework for detecting and modelling observations that change groups across variables is introduced. This framework takes advantage of the concept of missing data, allowing straightforward implementation using the expectation-maximization algorithm for Gaussian mixture models. Importantly, the components used to model group-switching behaviour require only one additional free parameter to be estimated: making it a feasible approach for detecting and modelling rare group-switching events. We motivate the concepts underlying the techniques through simple examples, provide details for the model-fitting algorithms, and discuss results on real data.
Network data capture relationships among actors across multiple contexts, often forming clusters of individuals. These relationships frequently involve multiple types of interactions, requiring multidimensional networks, or multi-graphs, to fully capture their complexity. Latent position models (LPMs) embed nodes based on connection probabilities but cannot uncover heterogeneous clustering structures such as disassortative patterns. In contrast, stochastic block models (SBMs) excel at clustering but lack interpretable latent representations. To address these limitations, the deep latent position block model (Deep-LPBM) was introduced to provide both clustering and continuous latent space representations in unidimensional networks. In this paper, we extend this approach to multidimensional networks by introducing the deep latent position block model for multidimensional networks (Deep-LPBMM). Deep-LPBMM integrates block modeling and latent embedding across multiple interaction types, allowing nodes to partially belong to several groups, thereby capturing overlapping clustering structures more effectively. Our model employs a deep variational autoencoder with graph convolutional networks (GCNs) for each layer and a multilayer perceptron to merge the latent representations into a unified embedding that represents partial cluster membership probabilities, enabling both effective clustering and enhanced visualization.
In the literature, most distance functions used for distance-based machine learning algorithms are based on the usual absolute value function on the field ℚ of rational numbers. On the other hand, Ostrowski’s theorem states that every non-trivial absolute value on ℚ is equivalent to either the usual absolute value or a p-adic absolute value for some prime p. In this study, a new p-adic distance function, the p-adic Euclidean distance, is defined based on the p-adic absolute value. This paper represents the first systematic investigation of both the p-adic Euclidean distance and the p-adic Chebyshev distance (referred to in the literature as the p-adic max-norm) within the k-nearest neighbor (k-NN) framework. Together with the previously proposed p-adic Manhattan distance, these distance functions are employed in k-NN models and evaluated on 30 publicly available datasets. Their performance is compared with that of k-NN models using 14 conventional distance functions commonly found in the literature. In the analyses, binary and multi-class classification were performed with datasets containing categorical, numerical, and mixed types of predictive attributes. To find the best performance values of the models, the number of neighbors k was examined for numbers varied from 2 to 20, and the prime number parameter p of p-adic distance was tested for numbers less than 29. The performance of the 17 distance functions used in the models was evaluated in terms of accuracy, recall, precision, and F1-score. In 13 of the 30 datasets, the models with p-adic distances were among the top five performers. When the datasets were analyzed separately as categorical, numerical, and mixed types, models employing p-adic distances achieved the highest classification accuracy in 14 out of 30 datasets. In numerical and mixed datasets, it was observed that the precision of decimal values influenced the performance of the analysis. These results suggest that the use of p-adic distances in the k-NN algorithm is particularly effective in categorical and mixed data scenarios, often surpassing many commonly used distance functions in the literature.
In multi-class prediction tasks with interpretative goals, covariates that help distinguish individual classes, termed “class-related covariates,” can be of particular interest. Conventional variable importance measures (VIMs) from random forests, such as permutation and Gini importance, rank covariates by overall predictive contribution and thus also assign high importance to covariates that differentiate between groups of classes. We propose a novel VIM, the class-focused VIM, which ranks covariates by their ability to distinguish individual outcome classes. It evaluates covariates using hypothetical multi-way class-based partitions at each node, without altering tree construction. As a complement, we introduce the discriminatory VIM, which measures general covariate influence based on the actual node splits. Simulations show that, unlike conventional VIMs, the class-focused VIM specifically ranks class-related covariates high. Real data examples illustrate how both suggested VIMs behave on real datasets and how their results can be interpreted.
Latent Gaussian processes are widely used in latent variable models because they offer a flexible nonlinear mapping from a low-dimensional latent space to a high-dimensional space via Gaussian processes. However, these models are computationally intensive and do not scale well. In this paper, we propose a temporal categorical model that utilizes latent Gaussian processes, inducing-input approximation, and a regularization framework to model multivariate categorical processes, with and without priors on hyperparameters. We analyze the underlying properties and introduce two variational inference approaches: one based on a Monte Carlo method and the other on the delta method. We also found that latent dynamics tend to collapse into a constant zero, which hinders the reflection of dynamic information. To address this, we propose two strategies to regularize the latent dynamics for better alignment with observation dynamics: introducing a regularization term based on inducing variables and incorporating a scale prior for latent Gaussian processes. Additionally, we propose an efficient and effective stochastic variational inference technique. Finally, we demonstrate our model and inference methods using both synthetic data and real financial data.
The main goal of this paper is to address scenarios in which the distribution of multivariate real-valued data exhibits skewness and diverse tail behavior across dimensions. The dimension-wise scaled mixtures of normal (DSMN) distributions have been shown to be effective in modeling data with varying degrees of tail heaviness by dimension. An extension of the DSMN distribution is introduced by incorporating a vector of shape parameters, leading to the skew dimension-wise scaled mixtures of normal (SDSMN) distributions. The SDSMN family offers flexibility in expressing a range of shapes by allowing control over tailedness and skewness in each dimension. This study examines the characteristics and probabilistic properties of SDSMN distributions, as well as explores their extension to finite mixtures thereof. An ECME algorithm is developed utilizing a selection mechanism to compute the maximum likelihood estimates of model parameters. Numerical experiments conducted on simulated data and four real datasets from various fields, including biometry and biomedicine, demonstrate the effectiveness and practicality of the proposed methodology.
Hidden Markov models (HMMs) are gaining popularity for analyzing multivariate longitudinal datasets wherein state switching of subjects is desirable; however, over-parameterization remains an issue. Thus, parsimonious HMMs are essential for the analysis of such data. In model-based clustering, it is common to introduce parsimony via a series of constraints on decomposed state covariance matrices. This approach for handling over-parameterization is applied to HMMs for longitudinal data. Specifically, two families of HMMs are developed: one arising from an eigen-decomposition of the state covariance matrices and another resulting from a latent Gaussian mixture. In the latter case, further parsimony is introduced by imposing constraints on the resulting factor analysis covariance structure. The performance of the introduced approaches is compared on various simulated and real datasets.
Training data optimization, also known as instance selection, is a critical preprocessing step in machine learning that enhances model performance by removing redundant or noisy data, reducing computational cost, and improving the handling of class imbalances. It ensures that the most informative and representative examples are retained, optimizing both the training process and the generalizability of the model. In this paper, we introduce a novel instance selection method designed to optimize the efficiency and effectiveness of machine learning models by enhancing the training dataset’s quality without resorting to artificial data manipulation. This method strategically selects representative instances that preserve the intrinsic value of the data, addressing challenges such as computational demand, noise, redundancy, and class imbalance. By incorporating adaptive clustering and a mechanism that adjusts the selection process based on the degree of class imbalance, our method offers a robust solution suitable for both balanced and imbalanced datasets. Extensive evaluations on numerous datasets demonstrate the competitiveness of our approach while maintaining computational efficiency.
This paper introduces a novel version of the quadratic discriminant analysis rule for spatial functional data, based on the Mahalanobis distance for spatially correlated functional data. This approach is relevant when the differences among the groups are not only in the mean function but also in the covariance structure. We estimate the mean function and a valid model for the covariance structure of each group and perform discriminant analysis for spatial functional data. Our proposal is inspired by brain signals from the language area, although not limited to this field. The accurate discriminant analysis of silent vowels obtained through electroencephalography (EEG) signals holds great potential to enhance communication in individuals. EEG signals represent curves measured at a finite number of locations in the brain, and can be treated as spatial functional data, allowing for a comprehensive analysis of their spatial characteristics. The methodology is applied to a dataset of EEG brain signals of people while they were thinking about each of the five vowels of the Spanish language in a silent voice. An additional application to climate data, and a simulation study to evaluate the performance, are included.
This study introduces a novel upper bound for the Bayes error that is a lower bound for the traditional Bhattacharyya bound for binary classifiers. We demonstrate its superior tightness through analytical comparisons and empirical evaluations against known bounds.
This paper demonstrates that the Kullback–Leibler (KL) divergence expression can be used to calculate distances between sample centroids when the probability distributions are replaced by classification probabilities. To test the properties of the KL distances, we used a large number of real and artificial datasets and many classification methods, the most important of which are as follows: linear discriminant analysis (LDA), flexible discriminant analysis (FDA), linear discriminant analysis using the generalized singular value decomposition (GSVD), regularized discriminant analysis (RDA), multinomial logistic regression (MLR), mixture discriminant analysis (MDA), naïve Bayes classification (NBC), artificial neural networks (ANN), support vector machines (SVM), and k-nearest neighbors (kNN). It was found that in general, there is a high correlation between Mahalanobis and KL distances exhibiting typical Mahalanobis-like properties, especially when using LDA, FDA, GSVD, RDA, and in many cases MDA. A Euclidean-like KL distance arises only from the NBC classification method in continuous datasets. The remaining KL distances, except for kNN in many datasets, tend to give clusters similar to those of MD, but significant deviations from MD can also be observed. The KL distances obtained mainly from kNN show serious problems when used in hierarchical cluster analysis. Finally, the majority of the classification methods examined in this paper can handle all types of data, continuous, ordinal, categorical, and binary, and therefore the corresponding KL distances can be calculated using any type of data or combination of data.
This paper proposes clustering methods for large-scale stationary time series using a fuzzy approach. Adopting partitioning around centroids (PAC) and partitioning around medoids (PAM), and focusing on distributional properties of individual series, we classify a large set of time series by transforming the series into probability density functions via nonparametric density estimation, such as the kernel estimation, and using a proper distance measure, such as the Hellinger distance, between density functions. We use simulations and two real applications to demonstrate the good performance and effectiveness of the proposed clustering methods in finite samples. The proposed methods are also applicable to the spectral density functions if one focuses on the serial dependence of individual series.
Bayesian deep learning (BDL) has become a potent technique for quantifying uncertainty in classification tasks, outperforming the traditional deep learning (DL) networks in terms of providing reliable results. Approximate Bayesian inference techniques have proven useful in making uncertainty estimation in the results of DL networks not intractable. This BDL gives more confidence in the results of DL networks and in adopting them in real-world applications, thus making informed decisions. Application of BDL based on appropriate Bayesian inference technique in important areas such as agriculture, especially regarding plant diseases, is essential to ensure responsible decisions. This paper presents a reliable approach to further improve the quantification of uncertainty in wheat disease detection called explainable Bayesian convolutional neural network (EBCNN). This approach relies on one of the DL networks, which is the convolutional neural network (CNN), and to make it able to detect the level of confidence in its results, a Bayesian inference technique was used. The Bayesian inference technique used is based on the Gaussian noise (GN) method, which is a method commonly used to prevent overfitting of DL networks. The proposed EBCNN can classify wheat diseases with high efficiency and accuracy of 93.44
Cluster-weighted models (CWMs) are a powerful approach in model-based clustering, widely used for capturing complex relationships in regression data. We proposed a new class of CWMs to accommodate functional data, addressing the regression of a functional response on one or more functional predictors across different groups of subjects. Each functional random variable can be modeled as a Gaussian or Student’s T process. Projecting the response curve and functional predictors onto their eigenspaces simplifies the new regression model into a structure similar to classical CWMs. By imposing constraints on the covariance matrices of these projected data, we develop a family of parsimonious models. It leads to the proposed functional cluster-weighted model (FunCWM) approach. The maximum likelihood estimation of parameters is facilitated through an expectation-maximization (EM) algorithm. The consistency properties of these estimates are examined. The FunCWM approach is evaluated on several simulated and real datasets, and the results confirm its efficiency.
We propose a latent class model for ordinal data with CUB (combination of discrete uniform and shifted binomial) distributions in the case of multilevel structures of the data. The CUB model is a powerful approach to the analysis of ordinal data, where the elicitation process is thought to be governed by a feeling parameter and an uncertainty parameter. Ordinal data are common across different research fields and may present a multilevel structure with units nested within groups. The model we present extends the framework of multivariate CUB models for model-based clustering to multilevel data, either hierarchical or cross-classified. Numerical experiments on simulated data highlight the added value of assuming a CUB model to account for ordinal information; the procedure’s interest is also shown through a real data application.
