Nowadays, in the field of modern natural language processing, the transformer-based neural network architectures are the state-of-the-art techniques. One of the main base models is the BERT, which is trained on a large corpus to provide natural language understanding, and the architecture is designed to be fine-tuned to any domain-specific task. The efficient fine-tuning process requires a hyperparameter optimization, and one of the main parameters is the learning rate, which will control the size of the modification of the model parameters during the training. This study will conclude a BERT model fine-tuning process on the selected Web of Science dataset to observe the effect of the learning rate on the model’s performance and training time. The whole experiment was implemented in the Komondor HPC system to utilize the necessary parallel computational resources. The key finding of this paper is that the learning rate can slow the convergence of the model or lead to catastrophic forgetting, while early stopping plays a key role in the training times. The research concludes that the utilization of the HPC resources can effectively scale up the whole hyperparameter search to determine the optimal learning rate range.

The rapid increase in scientific publications leads to the fact that manual study screening in systematic literature reviews (SLRs) is increasingly resource consuming, inefficient, and inconsistent. Classifying studies that clearly report health-related quality-of-life results, such as EQ-5D data, requires a high level of clinical interpretation and poses challenges for human reviewers. This study investigates the use of Google's Gemini and Gemma large language models (LLMs) in automating EQ-5D detection in the PubMed biomedical database based only on published abstracts. A multi-phase framework is proposed that integrates few-shot prompting, weight ensembling aggregation, and a soft stacking meta-classifier. Nine LLMs are evaluated on a dataset of PubMed studies manually labeled by two experts regarding EQ-5D reporting. The weighted ensemble of gemini-2.5-pro, gemma-3-12b, and gemma-3-27b obtained a 0.74 weighted F1score and 0.74 accuracy, exceeding individually attained results. The ensembling of top-performing models improved the balance between precision and recall compared to individual models, while the soft stacking approach provided greater reliability and interpretability. Feature analysis shows that the probability results from the models are important in guiding the final predictions. The findings suggest that an ensemble-based LLM setup is a reliable and scalable approach for automating screening in biomedical research.

Post-training quantization (PTQ) is one of the most practical ways to reduce the serving cost of Large Language Models (LLMs), but activation quantization remains difficult because outlier-dominated channels lead to large quantization errors. This paper investigates whether part of this degradation is caused by over-migration in scaling-based equivalent transformations. We introduce a quantile-robust scaling policy for SmoothRot-style transforms by replacing max-based activation statistics with high quantiles, and we complement it with constrained gradient-based optimization of channel scales. On LLaMA-3.2-1B under W4A4 quantization, quantile-only policy search improves selected-layer error by 11.1

This paper presents a formalized approach to modeling computer vision pipelines designed for non-intelligent, pre-installed camera networks. Using the Heimdall system as a case study, the paper proposes a deterministic framework that ensures traceable performance optimization and environment-specific adaptation for neural object and license plate recognition. The pipeline includes camera input, VPN-based data transfer, an edge image processing module, and a data reception software layer. The proposed formalization provides mathematical representation of parameter groups, such as resolution, confidence thresholds, and OCR preprocessing settings, and defines stable operational ranges to avoid performance collapse. Experimental validation is performed in an edge AI environment using real industrial camera feeds. Results indicate that deterministic parameter control and localized fine-tuning can significantly increase robustness without new hardware installation.

The EQ-5D (EuroQol 5-Dimensions) is a standardized instrument for the evaluation of health-related quality of life. In health economics, systematic literature reviews (SLRs) depend on the correct identification of publications that use the EQ-5D, but manual screening of large volumes of scientific literature is time-consuming, error-prone, and inconsistent. In this study, we investigate fine-tuning of general-purpose (BERT) and domain-specific (SciBERT, BioBERT) pre-trained language models (PLMs), enriched with biomedical entity information extracted through scispaCy models for each statement, to improve EQ-5D detection from abstracts. We conduct nine experimental setups, including combining three scispaCy models with three PLMs, and evaluate their performance at both the sentence and study levels. Furthermore, we explore a Multiple Instance Learning (MIL) approach with attention pooling to aggregate sentence-level information into study-level predictions, where each abstract is represented as a bag of enriched sentences (by scispaCy). The findings indicate consistent improvements in F1-scores (reaching 0.82) and nearly perfect recall at the study-level, significantly exceeding classical bag-of-words baselines and recently reported PLM baselines. These results show that entity enrichment significantly improves domain adaptation and model generalization, enabling more accurate automated screening in systematic reviews.

The EQ-5D is a widely used tool for measuring health-related quality of life (HRQoL) to support clinical, economic, and policy decision making. Manually classifying the growing volume of literature reporting EQ-5D data for systematic literature reviews is a challenging, inefficient, and labor-intensive task. To address this, we propose a comprehensive classification framework utilizing pre-trained language models (PLMs), including BERT, SciBERT, BioBERT, PubMedBERT, and BioLinkBERT, to categorize PubMed records based on whether the article reports EQ-5D data using article metadata (titles, abstracts, and keywords). We examine three learning approaches: supervised learning, semi-supervised learning with pseudo-labeling, and a co-training strategy with and without Large Language Model (LLM) assistance (GPT and Claude) in pseudo-label generation. We introduce a confidence-based ensembling within the co-training framework to improve classification reliability and robustness. This study provides a systematic multi-phase evaluation of supervised, semi-supervised, and co-training paradigms on PubMed records using different input configurations. It investigates model performance starting with 200 labeled samples, expanded through iterative pseudo-labeling of an unlabeled dataset, while benchmarking across models. The results show that the co-training approach achieves the highest performance, with an F1-score of up to 0.85. Performance is reported using multi-seed evaluation with mean +/- standard deviation and 95% confidence intervals. LLM-assisted co-training improves weaker model pairs but may degrade performance for already strong model combinations and reduce the number of high-confidence pseudo-labeled samples due to confidence thresholds. LLMs used within ensemble and co-training approaches provide an effective framework for EQ-5D literature screening under limited labeled data.

Background:Systematic literature reviews (SLRs) are essential for evidence synthesis in health research but remain labor-intensive, especially at the screening stage. Manual review of titles and abstracts requires substantial human effort, while existing automation tools still have limited adoption in health technology assessment. The EQ-5D questionnaire, a widely used patient-reported outcome measure for health-related quality of life, provides data that frequently underpin reimbursement and policy decisions. Objective:This pilot study evaluated whether recent large language models (LLMs) can support the identification of publications reporting EQ-5D data in PubMed records, using only publicly available metadata (title, abstract, and keywords). Methods:A total of 200 publications retrieved through the EuroQol PubMed filter were manually labeled by experts as reporting or not reporting EQ-5D data. The dataset was split into stratified training, validation, and test subsets. Several machine learning approaches were compared, including a Naïve Bayes baseline using bag-of-words features, a decision-tree model based on full-text keyword occurrence, and transformer-based LLMs (Bidirectional Encoder Representations from Transformers [BERT], Biomedical BERT [BioBERT], Scientific BERT [SciBERT], and Biomedical Language Understanding Evaluation BERT [BlueBERT]). Both classifier-only and fine-tuned configurations were tested across multiple learning rates. Model performance was assessed using accuracy, precision, recall, and F1-score. Results:Baseline approaches achieved near-random test performance (accuracy around 0.53). Classifier-only LLMs modestly improved results (accuracy up to 0.64 with SciBERT). Fine-tuned models substantially outperformed these baselines, with BERT and BioBERT achieving the best performance (accuracy=0.70; F1-score=0.68). In screening-oriented evaluation, this configuration achieved 90.0% sensitivity, 40.0% specificity, and 6 false negatives on the held-out test set. The models reproduced human screening tendencies despite the small dataset size, demonstrating the technical feasibility of LLM-assisted article selection. Conclusions:This study provides the first demonstration of LLM-assisted identification of EQ-5D data in biomedical literature. The findings support technical feasibility but do not establish a reliable stand-alone automated screening tool. Although limited by dataset size, the proposed workflow is reproducible and adaptable to other patient-reported outcome measures. Because validation was based on a single small train-validation-test split, the results should be interpreted as preliminary; future work will scale data collection, include statistical testing, and explore semisupervised learning to further reduce manual screening workload.

Automatic text classification is essential for the efficient categorization of scientific content. In this study, we investigate both flat and hierarchical classification on the Web of Science datasets WoS-46985 and its two variants, WoS-11967 and WoS-5736, using three pre-trained language models (PLMs): BERT, SciBERT, and BioBERT. For flat classification, we independently fine-tune the models for domain-level and fine-grained (leaf-level) category prediction, using dynamic learning rate selection and early stopping. We propose a mutual learning framework where pairs of PLMs exchange high-confidence predictions, integrated with an adaptive ensembling strategy for improved robustness. For hierarchical classification, we adopt a two-stage pipeline: first, predicting domain-level labels using an adaptive ensembling of two PLMs, and second, training domain-specific SciBERT models to predict area-level labels conditioned on the predicted domain. SciBERT consistently outperforms BERT and BioBERT across all classification levels. The mutual learning and hierarchical approaches achieve the highest accuracy across all datasets, especially with the SciBERT-BioBERT adaptive ensembling. Our approaches improve upon prior work, demonstrating that integrating specialized knowledge with collaborative and adaptive strategies enhances performance in hierarchical scientific text classification.

Recent advances in large language models (LLMs) have enabled natural language processing (NLP) to achieve notable progress in almost all tasks, such as text classification, language generation, and machine translation. However, the evaluation of these models has become a challenge as they grow and become more complex. Traditional metrics for evaluation, such as accuracy, BLEU scores, and perplexity, struggle to capture detailed aspects such as semantic understanding, coherence, and contextual relevance. In this work, we review various approaches to the evaluation of LLMs and provide a well-structured classification made up of four core categories: classification metrics, generation metrics, translation metrics, semantic and contextual metrics. We present a taxonomy of these evaluation systems, outlining their advantages and disadvantages, followed by a practical guide to choosing the metrics most appropriate for specific task needs. Additionally, we provide benchmark suites such as GLUE and SuperGLUE that offer a standardized framework through which the models’ performance can be evaluated across several tasks in NLP. The objective of this study is to offer a clear taxonomy as a guide for researchers and practitioners in selecting optimal metrics for their applications and to outline future directions for enhancing the evaluation methodologies.

The exponential increase in scientific literature and online information necessitates efficient methods for extracting knowledge from textual data. Natural language processing (NLP) plays a crucial role in addressing this challenge, particularly in text classification tasks. While large language models (LLMs) have achieved remarkable success in NLP, their accuracy can suffer in domain-specific contexts due to specialized vocabulary, unique grammatical structures, and imbalanced data distributions. In this systematic literature review (SLR), we investigate the utilization of pre-trained language models (PLMs) for domain-specific text classification. We systematically review 41 articles published between 2018 and January 2024, adhering to the PRISMA statement (preferred reporting items for systematic reviews and meta-analyses). This review methodology involved rigorous inclusion criteria and a multi-step selection process employing AI-powered tools. We delve into the evolution of text classification techniques and differentiate between traditional and modern approaches. We emphasize transformer-based models and explore the challenges and considerations associated with using LLMs for domain-specific text classification. Furthermore, we categorize existing research based on various PLMs and propose a taxonomy of techniques used in the field. To validate our findings, we conducted a comparative experiment involving BERT, SciBERT, and BioBERT in biomedical sentence classification. Finally, we present a comparative study on the performance of LLMs in text classification tasks across different domains. In addition, we examine recent advancements in PLMs for domain-specific text classification and offer insights into future directions and limitations in this rapidly evolving domain.

The goal of this work was to evaluate a state-of-the-art distributed graph neural network framework in order to gain deeper insights into their operation and to explore opportunities for advancing both graph neural network architectures and distributed training systems. The framework was tested in a community cloud environment using $\mathbf{1}-\mathbf{4}$ virtual machines, varying batch sizes, and different datasets. As a result of the evaluation, we designed a reference architecture that makes it easy to reproduce and apply the framework in future research. The experiments show that increasing the batch size can significantly reduce training time, but at the cost of accuracy. In contrast, adding more virtual machines improves training speed without degrading accuracy; however, scalability strongly depends not only on the model and the infrastructure, but also on the characteristics of the dataset. Sparse graphs scale more effectively, while dense graphs are more challenging. Overall, the experiments successfully reproduced a state-of-the-art setup and demonstrated measurable speedup through distributed training, providing a foundation for future research.

Quantization effectively reduces the serving costs of Large Language Models (LLMs) by speeding up data movement through compressed parameters and enabling faster operations via integer arithmetic. However, activating integer arithmetic requires quantizing both weights and activations, which poses challenges due to the significant outliers in LLMs that increase quantization error. In this work, we investigate these outliers with an emphasis on their effect on layer-wise quantization error, then examine how smoothing and rotation transform the observed values. Our primary contributions include introducing a new metric to measure and visualize quantization difficulty based on channel magnitudes, as well as proposing a hybrid approach that applies channel-wise scaling before rotation, supported by a mathematical formulation of its benefits. The code used in our experiments is available at https://github.com/czakop/smooth-rot-exper.

Signature verification is essential in banking, judicial, and governmental institutions. This study presents an offline signature authentication system based on a lightweight Siamese Neural Network (SNN) with metric learning. Unlike large CNN-based based approaches that rely on large datasets and complex architectures, this method utilizes extensive preprocessing and a compact CNN architecture to reduce computational cost while maintaining competitive accuracy. The images are pre-processed before being fed to the two-headed architecture; the system based on denoised, cropped and centered images achieved accuracy of 80-85% on the CEDAR and BHSig260 datasets, while using approximately 1.93 million parameters, significantly fewer than deep CNN-based approaches like state-of-the-art SigNet (10M+ parameters). The results demonstrate that effective preprocessing and hyperparameter tuning can enable data-efficient signature verification without requiring large-scale datasets.

Efficient text classification is essential for handling the increasing volume of academic publications. This study explores the use of pre-trained language models (PLMs), including BERT, SciBERT, BioBERT, and BlueBERT, fine-tuned on the Web of Science (WoS-46985) dataset for scientific text classification. To enhance performance, we augment the dataset by executing seven targeted queries in the WoS database, retrieving 1,000 articles per category aligned with WoS-46985's main classes. PLMs predict labels for this unlabeled data, and a hard-voting strategy combines predictions for improved accuracy and confidence. Fine-tuning on the expanded dataset with dynamic learning rates and early stopping significantly boosts classification accuracy, especially in specialized domains. Domain-specific models like SciBERT and BioBERT consistently outperform general-purpose models such as BERT. These findings underscore the efficacy of dataset augmentation, inference-driven label prediction, hard-voting, and fine-tuning techniques in creating robust and scalable solutions for automated academic text classification.

Large Language Models (LLMs) have transformed natural language processing, yet their deployment remains challenging due to substantial computational, memory, and energy demands. Post-training quantization has emerged as a key strategy for enabling efficient inference, particularly in resource-constrained settings. This systematic review focuses on weight-activation quantization, with a unique emphasis on the emergent outlier phenomenon in LLM activations. This work evaluates recent techniques that mitigate activation outliers and improve quantization efficiency, distinguishing itself from prior reviews. Using the PRISMA methodology, we examine 52 recent studies to uncover key trends and evaluate the effectiveness of different approaches. By synthesizing insights from these works, this review presents a diverse set of techniques and their implications for activation quantization, laying the groundwork for future research and practical advancements in LLM deployment.

We present SmoothRot, a novel post-training quantization technique to enhance the efficiency of 4-bit quantization in Large Language Models (LLMs). SmoothRot addresses the critical challenge of massive activation outliers, by integrating channel-wise scaling with Hadamard transformations. Our technique effectively transforms extreme outliers into quantization-friendly activations, significantly improving quantization accuracy. Experiments conducted on popular LLMs (LLaMA2 7B, LLaMA3.1 8B, and Mistral 7B) demonstrate that SmoothRot consistently reduces the performance gap between quantized and FP16 models by approximately 10-30% across language generation and zero-shot reasoning tasks, without introducing additional inference latency. Code is available at https://github.com/czakop/smoothrot.

In recent years, deep metric learning and contrastive learning have become essential approaches to represent similarity and difference using neural networks. By comparing pairs or triplets of examples, these methods construct embedding spaces that are the foundation of modern computer vision, natural language processing, and multimodal systems. Yet, while they have enabled breakthroughs in object re-identification and face verification, fundamental challenges remain: efficient negative sampling, robust training stability, and generalization across domains.In this talk, I will provide a brief overview of the evolution of metric and contrastive learning, from early Siamese and triplet architectures to recent developments in loss functions and sampling strategies. Building on these foundations, I will highlight the current research frontier, beyond pairwise comparisons toward self-supervised representation learning.

Deep metric learning (DML) is a deep learning-based approach to similarity measurement. After the popular image-and vision-based deep learning applications showed that deep neural network-based feature extraction improves the performance of several classification tasks, different studies have started to investigate the application of deep networks for similarity evaluation. After high-performing results in facial recognition, DML has been used in other fields, including visual object tracking, multi-camera re-identification, anomaly detection and natural language processing. This paper presents a short, application-oriented survey on novel solutions that apply DML. The key contribution of this work is to address the gap between theoretical and application-oriented surveys using a structured methodology to identify emerging application domains with categorised use cases. The Scopus scientific database was used as the data source for this study, where a keyword-based filter was applied, and the result metadata was analysed.

The popularity of cloud computing services is increasing significantly, which requires energy-aware workload prediction systems to improve resource efficiency and decrease the power consumption of IT infrastructures. Due to the significant amount of historical data that can be collected in these cloud infrastructures, statistical methods and machine learning-based solutions provide promising results for accurate workload predictions. This paper investigated a publicly available Microsoft Azure trace containing a snapshot of millions of virtual machine workloads from one region to create a cloud workload prediction method. We performed several data preparation and aggregation steps to identify the significant information from the time-series trace data. Based on the processed data, the key contribution of this paper is an evaluation of the ARIMA statistical model and a fine-tuned IBM’s TinyTimeMixer machine learning model to create an accurate workload prediction method.

Background Diabetes mellitus (DM) is a major health concern among children with the widespread adoption of advanced technologies. However, concerns are growing about the transparency, replicability, biasedness, and overall validity of artificial intelligence studies in medicine. Objective We aimed to systematically review the reporting quality of machine learning (ML) studies of pediatric DM using the Minimum Information About Clinical Artificial Intelligence Modelling (MI-CLAIM) checklist, a general reporting guideline for medical artificial intelligence studies. Methods We searched the PubMed and Web of Science databases from 2016 to 2020. Studies were included if the use of ML was reported in children with DM aged 2 to 18 years, including studies on complications, screening studies, and in silico samples. In studies following the ML workflow of training, validation, and testing of results, reporting quality was assessed via MI-CLAIM by consensus judgments of independent reviewer pairs. Positive answers to the 17 binary items regarding sufficient reporting were qualitatively summarized and counted as a proxy measure of reporting quality. The synthesis of results included testing the association of reporting quality with publication and data type, participants (human or in silico), research goals, level of code sharing, and the scientific field of publication (medical or engineering), as well as with expert judgments of clinical impact and reproducibility. Results After screening 1043 records, 28 studies were included. The sample size of the training cohort ranged from 5 to 561. Six studies featured only in silico patients. The reporting quality was low, with great variation among the 21 studies assessed using MI-CLAIM. The number of items with sufficient reporting ranged from 4 to 12 (mean 7.43, SD 2.62). The items on research questions and data characterization were reported adequately most often, whereas items on patient characteristics and model examination were reported adequately least often. The representativeness of the training and test cohorts to real-world settings and the adequacy of model performance evaluation were the most difficult to judge. Reporting quality improved over time (r=0.50; P=.02); it was higher than average in prognostic biomarker and risk factor studies (P=.04) and lower in noninvasive hypoglycemia detection studies (P=.006), higher in studies published in medical versus engineering journals (P=.004), and higher in studies sharing any code of the ML pipeline versus not sharing (P=.003). The association between expert judgments and MI-CLAIM ratings was not significant. Conclusions The reporting quality of ML studies in the pediatric population with DM was generally low. Important details for clinicians, such as patient characteristics; comparison with the state-of-the-art solution; and model examination for valid, unbiased, and robust results, were often the weak points of reporting. To assess their clinical utility, the reporting standards of ML studies must evolve, and algorithms for this challenging population must become more transparent and replicable.