Backpropagation relies on instantaneous weight transport and global updates, thus questioning its neural plausibility. Continual learning mechanisms that are largely biologically inspired employ backpropagation as the baseline training rule. In this work, the role of learning rules that avoid the weight transport problem in the context of continual learning is examined. We investigate weight estimation approaches that use linear combinations of local and non-local regularization primitives for alignment-based learning. These approaches are coupled with parameter regularization and replay mechanisms to demonstrate robust continual learning capabilities. We show that the layer-wise operations observed in alignment-based learning help to boost performance in complex task-aware and task-free scenarios on multiple image classification datasets. We study the dynamics of representational similarity from the learning rules and provide its mapping to the knowledge preservation capabilities of the models.
Conventional supervised contrastive learning methods excel in optimising encoders for discriminative tasks. In scenarios where only a few labelled samples are available, however, they struggle in eliminating the inductive bias when transferring from source to target classes. This is a byproduct (and inherent limitation) of their underlying optimisation process that involves training a representation to maximise class separation, without directly optimising for within-class cohesion. As a response to this limitation this paper introduces the Silhouette Distance (SD) loss, a new optimisation objective for supervised contrastive representation learning. SD aims to enhance the quality of learned embeddings by emphasising both the cohesion and separation of representation clusters for each class. We test SD extensively across several few-shot learning scenarios|where labelled data is limited|and we compare its performance against supervised contrastive loss and prototypical network loss for various text and image classification tasks. We also test SD in a cross-domain manner, by training a model on one dataset and testing it on another, within the same modality. Our results demonstrate the superior, at worst competitive, performance of the SD loss compared to its baselines. By leveraging pre-trained models and fine-tuning techniques, our study highlights how the SD loss can effectively improve representation learning across different modalities and domains. This initial study showcases the potential of the SD loss as a robust alternative within the few-shot learning setting. Keywords: Limited label access
Imbalanced data classification suffers from a lack of reliable metrics. This runs primarily from the fact that for most real-life (and commonly used benchmark) problems, we do not have information from the user on the actual form of the loss function that should be minimized. Although it is pretty common to have metrics indicating the classification quality within each class, for the end user, the analysis of several such metrics is then required, which in practice causes difficulty in interpreting the usefulness of a given classifier. Hence, many aggregate metrics have been proposed or adopted for the imbalanced data classification problem, but there is still no consensus on which should be used. An additional disadvantage is their ambiguity and systematic bias toward one class. Moreover, their use in analyzing experimental results in recognition of those classification models that perform well for the chosen aggregated metrics is burdened with the abovementioned drawbacks. Hence, the paper proposes a simple approach to analyzing the popular parametric metric F-beta. We point out that it is possible to indicate for a given pool of analyzed classifiers when a given model should be preferred depending on user requirements.
Efficient and stable neural network training is crucial for advancing machine learning applications. This study explores the promising streaming approach as an alternative to traditional epoch-based methods. This paradigm shift involves transforming training data into a continuous stream, prioritizing challenging examples to enhance the learning process. Building upon this approach, we introduce an innovative Base-Values mechanism aimed at further improving the speed and stability of the streaming training process. We apply this framework to the original streaming training algorithms, resulting in algorithms such as Persistent Loss-Based (PLB) and Persistent Entropy-Based algorithm (PEB). We conduct a comprehensive experimental comparison on the EMNIST dataset, analyzing traditional epoch-based methods and the streaming approach, including both original and new methods employing the Base-Values mechanism. The exploration of various hyperparameters, including pre-training length, mini-batch size, and learning rate, provides valuable insights into their impact on the performance and stability of those neural network training methods. The results demonstrate the superior performance of a novel streaming algorithm that incorporates the Base-Values mechanism compared to both a traditional epoch-based method and other methods.
