An increasing number of financial software system relies on Machine learning models to support human decision-makers. Although these models have shown satisfactory performance to support human decision-makers in classifying financial transactions, the maintenance of such ML systems remains a challenge. After deployment in production, the performance of the models tends to degrade over time due to concept drift. Methods have been proposed to detect concept drift and retrain new models upon detection to mitigate the drop in performance. However, little is known about the effectiveness of such methods in an industrial context. In particular, their evaluation fails to consider the delay between the detection of the drift and the deployment of a new model. This delay is inherent to the strict quality assurance and manual validation processes that financial (and other critical) institutions impose on their software systems. To circumvent this limitation, we formalize the problem of retraining ML models against distribution drift in the presence of delay and propose a novel protocol to evaluate drift detectors. We report on an empirical study conducted on the transaction system of our industrial partner, BGL BNP Paribas, and two publicly available datasets: Lending Club Loan Data and Electricity. We release our tool and benchmark on GitHub. ^1 We demonstrate for the first time how ignoring the delays in the evaluation of the drift detectors overestimates their ability to mitigate performance drift, up to 39.86 ^1 Code available at https://github.com/serval-uni-lu/drift-robustness .
Network alignment, which maps the same entities across multiple networks, has gained tremendous interest in recent years. However, most existing alignment methods propose to align static graphs that are merely a single snapshot of real-world networks. These methods fail to model the inherent dynamics of entire networks where the nodes, links, and attributes are bound to change over time, making dynamic network alignment challenging. Moreover, modeling the interaction between two different dynamic graphs comes with additional challenges: (1) catastrophic forgetting while learning the evolution of individual networks, (2) distributional shift on the same dynamic network, and (3) domain differences between the two networks. Hence, to overcome these challenges, we propose RoCoNA, an end-to-end reservoir sampling-based continual learning approach built over streaming Graph Neural Networks (GNNs) that uses a novel shift-induced regularizer to handle distribution drift and domain differences in evolving networks. We empirically show that our method outperforms the existing state-of-the-art static and dynamic alignment methods. We perform case studies on networks with high distributional shifts to strongly validate our claims. Code is available at https://github.com/shruti400/RoCoNA/tree/main
Dynamic Graph Representation Learning methods have achieved significant success. However, real-world data often do not strictly follow the smoothness assumption in time-dependent relationships between dynamic graph snapshots. Existing methods sometimes fail to capture these relationships effectively. To address this issue, we propose the Community Enhanced Transformer for Dynamic Network Embedding (CeDFormer), which leverages transformers to capture temporal structures in dynamic graphs. CeDFormer is designed to adapt to highly variable graph structures and complex temporal dependencies. To mitigate the high computational cost and limited scalability of transformers on large-scale graph data, we introduce an optimization strategy involving parameter sharing within stable communities from a global perspective. This strategy enhances training speed by 30%∼ 35% without compromising model performance. Extensive experiments on real-world datasets demonstrate that CeDFormer outperforms most other methods on the majority of datasets. Code is available at https://github.com/gjqwanttogjq/CeDFormer .
Several challenges arise when applying Machine Learning in data streams. Firstly, data points are continuously generated, and algorithms should continuously learn from each. The solution should also promptly adapt to changes in data distribution (known as concept drifts). Additionally, while addressing concept drifts, preserving the knowledge gained from past data is crucial to avoid the problem of catastrophic forgetting. Finally, one should consider the temporal dependence that data points may exhibit. Three communities address these problems separately: Streaming Machine Learning (SML), Continual Learning (CL), and Time Series Analysis (TSA). In our previous research, we proposed Continuous Progressive Neural Networks (cPNN), a first approach that considers all the challenges together. It bridges SML, CL, and TSA by producing a continuous adaptation of the CL strategy of Progressive Neural Networks. It uses transfer learning to adapt to changes quickly and manages temporal dependence using Long Short-Term Memory. In this work, we present a comprehensive experimental campaign that analyzes the behaviour of SML models and cPNN in the case of complex temporal dependence and various concept drifts on synthetic and real data streams. Results bring statistical evidence that SML models struggle with substantial temporal dependence, while cPNN is a viable solution.
In the rapidly evolving field of Machine Learning applied to data streams, where information arrives continuously and models must adapt in real-time, drift detection has emerged as a critical component for maintaining model accuracy and reliability. While much attention has been given to structured data, the challenge of handling real-time image streams remains comparatively underexplored, despite its vital role in numerous applications. This study addresses this gap by exploring innovative approaches to visualize and explain drift in image streams. We propose a novel method that leverages zero-shot classification capabilities of a pre-trained ResNet-50 architecture to visualize potential changes within the stream. Additionally, we investigate the application of ResNet50 in combination with Uniform Manifold Approximation and Projection. This hybrid approach aims to identify and visualize drift in high-dimensional image data by projecting it onto a lower-dimensional space, facilitating easier interpretation and analysis of evolving patterns over time. Preliminary results on the CLEAR10 dataset, the first benchmark with a natural temporal evolution of visual concepts, showcase potential applications for drift monitoring in image streams. These findings present a promising avenue for further research and development, aiming at solidifying the integration of these methodologies for improved drift detection and overall models performance in dynamic data contexts.
