Finding an optimal decision tree that minimizes classification error is known to be NP-hard. While exact algorithms based on MILP, CP, SAT, or dynamic programming guarantee optimality, they often suffer from poor anytime behavior—meaning they struggle to find high-quality decision trees quickly when the search is stopped before completion—due to unbalanced search space exploration. To address this, several anytime extensions of exact methods have been proposed, such as LDS-DL8.5, Top-k-DL8.5, and Blossom, but they have not been systematically compared, making it difficult to assess their relative effectiveness. In this paper, we propose CA-DL8.5, a generic, complete, and anytime beam search algorithm that extends the DL8.5 framework and unifies some existing anytime strategies. In particular, CA-DL8.5 generalizes previous approaches LDS-DL8.5 and Top-k-DL8.5, by allowing the integration of various heuristics and relaxation mechanisms through a modular design. The algorithm reuses DL8.5’s efficient branch-and-bound pruning and trie-based caching, combined with a restart-based beam search that gradually relaxes pruning criteria to improve solution quality over time. Our contributions are twofold: (1) We introduce this new generic framework for exact and anytime decision tree learning, enabling the incorporation of diverse heuristics and search strategies; (2) We conduct a rigorous empirical comparison of several instantiations of CA-DL8.5—based on Purity, Gain, Discrepancy, and Top-k heuristics—using an anytime evaluation metric called the primal gap integral. Experimental results on standard classification benchmarks show that CA-DL8.5 using LDS (limited discrepancy) consistently provides the best anytime performance, outperforming both other CA-DL8.5 variants and the Blossom algorithm while maintaining completeness and optimality guarantees.
Large Language Models (LLMs) are increasingly used to solve machine learning tasks on tabular data, such as classification tasks. Using standard benchmarks, recent studies have shown impressive performance for such tasks. However, in this paper we study a major concern with the use of standard benchmarks: LLMs may have also been trained using these benchmarks, that is, these LLMs are contaminated. While previous work mostly focused on the GPT models, we propose a methodology to evaluate whether a large range of LLMs is contaminated. We propose new tests to detect contamination with tabular data over two aspects: knowledge and memorization. We also design an algorithm to parse the answers of the LLMs and detect hints of contamination. Our experiments conclude that the bigger and more closed-source an LLM is, the more likely it is contaminated.
Researchers and practitioners of machine learning nowadays rarely overlook the potential of using explainable AI methods to understand models and their predictions. These explainable AI methods mainly focus on the importance of individual input features. However, as important as the input features themselves, are the interactions between them. Methods such as the model-agnostic but computationally expensive Friedman's H-statistic and SHAP investigate and estimate the impact of interactions between the features. Due to computational constraints, the investigation is often limited to second-order interactions. In this paper, we present a novel, model-specific method to explain the impact of feature interactions in SVM classifiers with polynomial kernels. The method is computationally frugal and calculates the interaction importance exactly for any order of interaction. Explainability is achieved by mathematical transformation to a linear model with full fidelity to the original model. Further, we show how the model provides for both global and local explanations, and facilitates post-hoc feature selection. We demonstrate the method on two datasets; one is an artificial dataset where H-statistics requires extra care to provide useful interpretation; and one on the real-world scenario of the Wisconsin Breast Cancer dataset. Our experiments show that the method provides reasonable, easy to interpret and fast to compute explanations of the trained model.
Probabilistic Circuits (PCs) are a class of tractable models that allow a range of efficient and exact computations while achieving state-of-the-art performance in some domains. In this work, we propose a sample-based procedure to let the distribution encoded by a PC satisfy probabilistic propositional logic constraints. This sample-based method is proposed as a direct competitor to a mathematical approach previously introduced, which is based on optimizing a convex upper bound of the KL divergence. In our empirical study, we compare both methods in two different scenarios, where constraints are utilized to: i) apply fairness to a distribution; and ii) improve the performance of a PC model under scarce data. Our results indicate that although both methods are competitive to one another in the case of fairness, sample-based method has an advantage in scenarios with scarce data.
Trajectory representation learning (TRL) is an intermediate step in handling trajectory data to realize various downstream machine-learning tasks. While most previous TRL research focuses on modeling structured movements in large-scale urban spaces (e.g., cars or pedestrians on streets), this paper focuses on a more challenging scenario of modeling free movement in small-scale social spaces (e.g., children playing in a schoolyard). We present a TRL model, SiamCircle, to process raw trajectories without additional feature extraction to prevent information loss. SiamCircle adopts a Siamese network with Circle Loss to learn trajectory embeddings. Furthermore, SiamCircle employs a data augmentation process to enable self-supervised learning and enrich the input data to address the limited access to high-quality data and ground truth. We evaluate the performance of SiamCircle in downstream tasks using trajectory ranking and clustering performance via seven evaluation metrics collectively. Using an ablation study, we explored the impact of different loss functions on the model’s performance. Accordingly, we selected a 2-D convolutional design with Circle Loss as the best-performing model. In a comparative study, we compared our model against three other baselines. We observed up to 19
We find locally exceptional subgroups of nodes in attributed graphs, combining both node attributes and structural information to assess subgroup exceptionality. Subgroups are locally exceptional if their behavior deviates from the behavior of a well-chosen local peer group, as opposed to the more common generic subgroup discovery approach where behavior is compared to the global behavior across the full dataset. This notion of Local Subgroup Discovery had been introduced for traditional flat-table data; to the best of our knowledge, we are the first to incorporate this notion explicitly in graph data. Our approach combines shortest-path distance with Gower's Distance, integrating both network structure and node attributes to rank nodes in relation to a prototype node. Combining this notion of exceptionality with existing LSD techniques, we discover local subgroups in three attributed graph datasets.
Clustering is an unsupervised exploratory task that helps experts understanding the structure of their data. Constraints based on expert knowledge can be introduced, but obtaining them remains challenging, making the explanation of results essential for adjusting parameters and uncovering new insights. We address explainable clustering by modeling the data in two spaces: one for clustering and one for explanation. Our method ECS (Explainability-driven Cluster Selection) aims to produce a high-quality clustering while ensuring interpretability through patterns that cover most instances in a cluster and distinguish them from others. It relies on ensemble clustering and a new constraint programming (CP) model for selecting the clusters and their explanations.
Statistical modeling of multivariate time-series data poses significant challenges due to their high dimensionality and complex inter-variable relationships. Reliable forecasts or anomaly detection on these datasets require capturing such relationships within and between the features. While traditional deep learning architectures are good at capturing temporal non-linear patterns within features, they are less efficient at modeling inter-variable relationships explicitly structured as graphs-a capability where Graph Neural Networks (GNNs) excel. Inspired by the success of GNNs, Graph Deviation Network (GDN) was originally proposed for anomaly detection on industrial multivariate time-series data. After proving its merits through experiments with real-world data, GDN gained significant popularity in the research community, claiming to learn the hidden graph structure in any multivariate time-series data. Various modifications to GDN were proposed over the years, but essentially all of them kept its Graph Structure Learning (GSL) module intact. However, until now, this module has never been rigorously evaluated. This work scrutinizes the contribution of the GSL module. Our experiments reveal that the graph learned by GSL is relatively ineffective, and the key to the overall performance achieved by GDN lies almost entirely in the downstream Graph Attention Network (GAT) module. We hope our findings will garner attention for further development of the GSL module of GDN, whose fidelity can improve the performance of GDN variants.
Linguistic richness is essential for advancing natural language processing (NLP), as dataset characteristics often directly influence model performance. However, traditional metrics such as TypeToken Ratio (TTR), Vocabulary Diversity (VOCD), and Measure of Textual Lexical Diversity (MTLD) do not adequately capture contextual relationships, semantic richness, and structural complexity. In this paper, we introduce an autoencoder-based framework that uses neural network capacity as a proxy for vocabulary richness, diversity, and complexity, enabling a dynamic assessment of the interplay between vocabulary size, sentence structure, and contextual depth. We validate our approach on two distinct datasets: the DIFrauD dataset, which spans multiple domains of deceptive and fraudulent text, and the Project Gutenberg dataset, representing diverse languages, genres, and historical periods. Experimental results highlight the robustness and adaptability of our method, offering practical guidance for dataset curation and NLP model design. By enhancing traditional vocabulary evaluation, our work fosters the development of more context-aware, linguistically adaptive NLP systems.
In applied sciences, an experiment is defined by a parameter setting, and it leads to an associated outcome. For example, when carried out under different conditions, a chemical reaction leads to different yields. Finding a satisfactory outcome is a difficult task when the number of possible settings is too large: it becomes unfeasible to explore all the possibilities due to material and time constraints. It is therefore important to quickly identify a setting that leads to a sufficient enough outcome. Bayesian optimization offers a strategy to efficiently explore parameter settings. However, the latter needs to be improved to identify interesting settings earlier. Our contribution, BOWSA, analyzes the outputs of Bayesian optimization through sensitivity analysis: it quantifies the association between each parameter and the outcome. Then, BOWSA uses this information to generate a score that refines the outputs of the Bayesian optimization. This paper provides a methodological description of BOWSA. It proposes an implementation that uses the existing EDBO, Experimental Design via Bayesian Optimization tool, a practical implementation of Bayesian optimization designed to handle both discrete and categorical inputs, making it particularly well-suited for experimental science applications. An evaluation has been conducted on several datasets derived from well-known functions and two chemical synthesis reactions from the EDBO repository. Comparative results demonstrate that on average, BOWSA identifies productive settings earlier than Bayesian optimization alone.
In Natural Language Processing (NLP), dealing with under-represented topics is challenging, especially in unsupervised tasks where clustering might not adequately capture minority topics. To tackle this challenge, our paper presents a novel unsupervised data augmentation method that integrates Gaussian Mixture Models (GMMs) and Large Language Models (LLMs). Due to their flexibility and robustness, GMMs can detect clusters corresponding to underrepresented areas in the data, while LLMs create synthetic documents to enrich these clusters and improve their representation. Experiments on various imbalanced text datasets demonstrate that our approach preserves clustering performance in all cases and often enhances cluster interpretability, offering a robust and scalable solution for improving data representation in unsupervised NLP tasks.
We report on an application of three multivariate time series classification methods, HIVE-COTE 2.0, MINIROCKET, and MR-PETSC, to gaze and eyelid movement data to classify expertise. Our methods can be used to noninvasively monitor performance and identify experts using low-grade equipment. The test case was Tetris, which is a video game in which players arrange falling blocks to clear horizontal lines with increasing points and difficulty as the game advances. In addition to being able to classify the expert players, we can attribute the patterns within a time series that led to a prediction with MR-PETSC. This allows us to describe eye behavior that is associated with expertise. This method can be used in any performance classification that involves screen-based activity that is accompanied by eye movement recordings, for example with a webcam.
Hyperparameter optimization (HPO) aims to design machine learning algorithms that generalize well to unseen data by repeatedly evaluating hyperparameter configurations using a validation procedure. When the validation performance of these configurations is overly optimistic compared to the performance on an unseen test set, this is referred to as meta-overfitting. We decompose meta-overfitting into two types: (i) selection-based and (ii) adaptive overfitting. Selection-based overfitting occurs when testing many configurations, which increases the chance of finding a configuration that performs well on the validation set by chance but performs suboptimal on the test set. Adaptive overfitting arises from advanced HPO methods, such as Bayesian optimization, which iteratively utilize validation results of earlier configurations to propose new configurations increasingly tailored to the specific validation set. We provide one of the largest empirical studies of meta-overfitting in the context of HPO for the Combined Algorithm Selection and Hyperparameter Optimization (CASH) problem, analyzing random search and Bayesian optimization for 48 classification and 16 regression datasets using holdout validation. We show evidence of adaptive overfitting in Bayesian optimization for 41 classification datasets, and consistent with prior work, we show that multiclass datasets are less affected by this phenomenon. Additionally, we find that optimization procedures for regression datasets are surprisingly resilient to adaptive overfitting. Furthermore, we explore the effect of various design choices in the validation procedure (i.e., 10-fold cross-validation and varying hold-out set sizes) on meta-overfitting.
Sign language (SL) is a visual language used by the Deaf community. Static sign language recognition (SLR) consists of classifying static hand configurations, i.e., signs, present in isolated images. Due to the expertise required for manual annotation, SLR suffers from a data scarcity issue. Recent studies show that contrastive learning is an effective method for addressing this issue by proposing an efficient unsupervised pre-training. Contrastive learning leverages data augmentation techniques applied to entire images (global-global augmentation). However, fine-tuned, contrastive models often rely on irrelevant aspects of those images, like the background, without focusing solely on the regions of interest. Such models are prone to bias that could lead to unreliable predictions. In response, this paper proposes a new local-global data augmentation technique that helps contrastive models focus during the fine-tuning step on regions of interest, i.e., the signer’s hands. This approach (i) improves the accuracy of contrastive learning by up to 15
Little is known about how the quality of a clustering changes when changing the size of the set used to determine the clustering model. We show that, for K-means clustering, the relationship between dataset size and clustering quality can display counterintuitive behavior. Notably, the quality can significantly deteriorate with more data to build the model. More generally, using artificial datasets and data from bioinformatics, we uncover a variety of learning curve behaviors for K-means. Our results clearly illustrate that the training sample size can have a nontrivial influence on the clustering performance. Our findings should appeal to both the clustering practitioner and the clustering researcher concerned with developing basic insights.
Balancing performance and scalability is a major concern when developing robust ML models for diverse, big-data scenarios, such as predicting demand for a number of products across multiple locations. The two mutually opposite approaches are to use a single ML model for maximizing scalability, often at the expense of performance, or to use a specialized model for each specific use case, which is often prohibitive in terms of computational costs. In this paper, we propose to balance those two approaches using our methods of model clustering and grouping. We achieve the performance level of a single use-case model while preserving the global scalability of the solution. In our experiments, we use a publicly available demand forecasting dataset as a use case. We develop and train baseline shallow ML models and DL models for both maximizing performance and scalability. Then, we showcase a desirable balance that can be achieved using our proposed methods, one that outperforms both shallow ML and specific use-case models.
This paper introduces a novel transfer learning adapter, the Bridged Attention Module (BAM), designed to enhance the performance of Spatial-Temporal Graph Convolutional Networks (ST-GCN) in datalimited forecasting scenarios. BAM improves fine-tuning efficiency by jointly capturing spatial and temporal dependencies, optimizing information flow, and significantly reducing the number of trainable parameters while preserving model accuracy. Experimental evaluations demonstrate that the BAM-enhanced ST-GCN consistently achieves competitive accuracy and, in some cases, surpasses traditional fine-tuning methods, even with limited data. The effectiveness of this approach is validated using electric vehicle (EV) charging station occupancy forecasting, highlighting the practical utility of BAM.
For personalized recommendation in graphs, random walks starting from the user's interest node are known as a general-purpose and fast analysis method. Specifically, Personalized PageRank (PPR), which quantifies the importance of each node by the distribution of visited nodes in random walks, determines nodes with high global importance and source proximity. However, it is difficult to balance both influences monotonically. This paper clarifies that the random walks length is effective to monotonically control the balance of both influences on PPR vectors. In particular, we exploit the fact that correlation between PPR and PageRank values monotonically changes depending on PPR parameter that controls the average random walk length. Here, PageRank is a metric that quantifies the global importance by the probability of random walks from all nodes visiting each node. A case study using the movie rating dataset showed that nodes that are considered to be directly related to the source node get high PPR value by shortening the average random walk length. Moreover, statistical evaluation on nine real-world datasets revealed that changing the average random walk length from 1.01 to 100 resulted in a monotonic increase of the cosine similarity between PPR and PageRank vectors from 0.002 to 0.76 at the maximum.
The digitization of data in hospitals has enabled the collection of mixed temporal data for surgical procedures. These datasets describe patients through an event trace and a multivariate time series. Leveraging this digitized data presents a unique opportunity to develop innovative anesthesia training methods, enabling real-time scenario generation driven by machine learning models. This paper aims to develop a model that (i) simulates the evolution of the physiological variables of a digital patient under anesthesia, in response to medical actions, and (ii) predicts the actions performed by the virtual medical team, with the simulator user being the sole human agent executing actions through the simulator interface. The challenge lies in achieving two goals within a single model: modeling asynchronous data streams for time series forecasting and event prediction. A key difficulty in our medical case is the sparsity of the event sequences. We found no models addressing the combined prediction of event-contextualized time series and time-series-contextualized events. To bridge this gap, we explore four deep learning architectures that we have tailored for unsupervised representation learning and joint prediction tasks: (i) a stacked LSTM (Long Short-Term Memory) network, (ii) a Graph Neural Network (GNN), (iii) the Vanilla Transformer, and (iv) its variant, the Informer. We evaluate these models on a real-world anesthesia dataset, focusing on overall predictive performance and computational efficiency. Our proof-of-concept study demonstrates that the Transformer outperforms the three other models which show significant shortcomings in predicting medical events.
Deep learning methods have become increasingly popular for time series nowcasting; however, their effectiveness is limited when making predictions on out-of-distribution data in dynamic streaming environments. This limitation is mainly caused by concept drifts, which occur when the underlying data distribution changes over time – and proves particularly challenging in multi-stream settings, where numerous streams of data exist without necessarily exhibiting the same dynamic behavior. When confronted with concept drift, since many parts of the neural network may still contain useful knowledge about the relevant domain, re-training only on new data poses the risk of catastrophic forgetting. On the other hand, re-training on old and new data can incur significant computational overhead and latency, which is not desirable in a streaming context. Conversely, stream learning allows models to adapt quickly to the new distribution; however, it often fails to achieve efficiency comparable to batch learning. This work aims to combine the strengths of neural networks and streaming regressors for data stream nowcasting in two key ways. First, a streaming regressor utilizes the higher-order representations learned by the neural network, which is set up as a feature extractor and a regressor. Second, the streaming regressor predicts the residual of the neural network. Experiments conducted on challenging real-world dynamic multi-streaming data indicate that this hybrid approach provides better results than the traditional static neural network.
