Transformer-based architectures have revolutionized Question Answering (QA), yet navigating the trade-off between architectural complexity, computational efficiency, and robust generalization remains a critical challenge. This study presents a rigorous, side-by-side evaluation of four distinct Transformer models—ALBERT, RoBERTa, DeBERTa, and XLNet—on the SQuAD v2.0 benchmark, which demands both precise span extraction and calibration to abstain from unanswerable queries. By employing identical preprocessing pipelines and hyperparameter schedules, we isolate the impact of specific architectural innovations, including dynamic masking, disentangled attention, and permutation language modeling. Experimental results establish RoBERTa as the performance leader (85.90% EM, 88.69% F1), validating the efficacy of robust optimization strategies. Notably, DeBERTa followed closely (85.41% EM, 88.54% F1) but achieved superior performance on unanswerable questions (88.24%), suggesting that its disentangled attention mechanism offers enhanced verification capabilities against distractor entities. XLNet demonstrated competitive potential (78.30% EM) via its autoregressive formulation, while ALBERT confirmed its status as the efficiency benchmark, delivering respectable accuracy (80.70% EM) with a fraction of the parameter footprint. Qualitative error analysis reveals that while lighter models are prone to "hallucinating" answers, advanced architectures primarily fail due to subtle reasoning errors rather than entity mismatch. These findings provide a definitive decision framework for practitioners: prioritizing RoBERTa and DeBERTa for maximum reliability in complex domains, and leveraging ALBERT for scalable, resource-constrained deployments.
Objective pain assessment is challenging due to the inherently subjective and multidimensional nature of pain, compounded by intersubject variability and noise in naturalistic settings. These factors limit the development of reliable digital biomarkers for clinical decision support and remote monitoring. In this work, we propose a hybrid deep learning architecture that performs early fusion of single channel EEG band power streams with peripheral biosignals recorded by a wristband (electrodermal activity, photoplethysmography/BVP, triaxial accelerometry, and skin temperature). The model uses residual 1D CNN encoders to learn EEG representations and a multilayer perceptron to encode wearable features; the two embeddings are fused in a unified layer to produce predictions for four classes under a strict subject independent evaluation protocol. On PhysioPain, the proposed approach achieves 86.98% accuracy and 87.46% weighted F1, surpassing widely used machine learning baselines, while maintaining an average inference latency of 0.1495 ms per sample for real time operation. In addition to widely used machine learning baselines, we benchmark contemporary deep 1D CNN baselines for a fair comparison: the proposed model improves over 1D DenseNet by 4.49 percentage points in accuracy and 4.70 percentage points in weighted F1, and over 1D ResNet by 7.90 percentage points in accuracy and 7.93 percentage points in weighted F1. To our knowledge, this is the first study to perform subject independent multiclass classification of natural pain phenotypes with explicit handling of class imbalance by early fusion of MindWave EEG with Empatica E4 signals, representing a step toward generalizable and clinically useful digital biomarkers of pain.
Writing the Related Work section is a challenging task for inexperienced researchers, requiring significant time, effort, and skill. This study aims to ease this burden by introducing an ICT-based framework that automates the generation of cohesive and well-structured Related Work sections. We propose SciRelate, a novel framework that combines abstractive multidocument summarization and automated keyword-based subsection heading extraction. SciRelate comprises three main modules: (1) document clustering using Latent Dirichlet Allocation (LDA), (2) related Work generation via a fine-tuned Bidirectional and Autoregressive Transformer (BART) model, and (3) Topic keyword extraction using TopicFusion-MP, a hybrid method that integrates LDA, TextRank, and MPNET embeddings. Experiments were conducted using the NudtRwG dataset, and BART was evaluated using 10-fold cross-validation. LDA clustering effectively identified subsection headings aligned with ground truth structures. The BART model achieved high ROUGE scores: 81.53 (ROUGE-1), 76.19 (ROUGE-2), and 79.03 (ROUGE-L). TopicFusion-MP outperformed six baseline keyword extraction methods in four scenarios, reaching a top similarity score of 0.6398. This study presents a comprehensive, automated pipeline for generating Related Work sections, integrating clustering, summarization, and keyword extraction in a unified framework. SciRelate highlights the potential of ICT tools to support scientific writing, particularly for new researchers.
Topic Models is a tool to extract semantics from textual collections. However, in spite of extensive work on topic modeling, including its evaluation using Language Models (LM), there is need for more research to measure, analyze and validate the topics learned. To this end, we present a Neural Topic Model called ABT which leverages semantic vector spaces (derived from language models) to uncover latent semantic representations in text collections. ABT enables one to explore the themes covered by a given textual collection and analyze them as a hierarchical structure of topics. We demonstrate the effectiveness of ABT by analyzing two text corpora: (1) CliCR, a collection of 10,539 real clinical cases; and (2) MovieLens, a dataset consisting of 9,742 movie titles. We have experimented with different language models to investigate the effects of using specific tasks to tune the neural network. The topics that resulted from our study highlighted concepts that are shown to be effectively used as prominent topics to represent the datasets. The results show that each experimented LM focuses on distinct features to produce the topics and may generate topics with different grades of generality and specificity. Our study sheds light on recent findings of the related literature, which state that the topic coherence metric does not align with assessments performed by human evaluators.
Imbalanced class distributions in Internet of Things (IoT) attack datasets limit the ability of machine learning (ML) models to detect minority intrusions and lead to high false-negative rates. This study investigates the effectiveness of the Synthetic Minority Oversampling Technique (SMOTE) in improving multi-class attack detection across seven IoT devices from the TON_IoT dataset. Four ML algorithms —K-Nearest Neighbor, Support Vector Machine, Logistic Regression, and Multi-Layer Perceptron— are evaluated both with and without SMOTE using accuracy, precision, recall, and F1-score. Results show that SMOTE substantially improves minority-class detection, particularly in devices with severe imbalance. On the Garage Door dataset, KNN accuracy increases from 0.952 to 0.982, and MLP from 0.555 to 0.978, while SVM exhibites modest gains. The largest recall improvement is observed for Modbus with KNN, rising from 0.52 to 0.84. Overall, SMOTE enhances IoT intrusion detection without degrading majority-class performance, although its impact varies across algorithms and device characteristics. Future research is planned to explore hybrid resampling and deep learning-based approaches to further improved detection under extreme imbalance.
