Optimizing GEneral Matrix Multiplication (GEMM) on GPU platforms is becoming increasingly critical to meet the growing computational demands of modern deep neural network research. While significant progress has been made in accelerating high-precision GEMM, the optimization of low-bit GEMM remains a challenging open problem. The CUTLASS library provides highly optimized low-bit GEMM templates leveraging Tensor Cores; however, performance varies considerably depending on tile and pipeline configurations across different GPU architectures. In this work, we propose a novel auto-tuning framework for low-bit CUTLASS GEMM, utilizing a neural network model to predict optimal GEMM template parameters for target GPUs. Our model is trained on a synthetic dataset with up to 116 100 unique samples, encompassing diverse matrix sizes across various Ampere GPUs, and is thoroughly evaluated on these hardware platforms. Experimental results show that our method achieves an accuracy of up to 95.11% on the validation dataset. Furthermore, real-time evaluations of low-bit data types on the A100 GPU demonstrate speedups of up to 1.99× for GEMM operations and 1.28× for the linear layer, compared to the default CUTLASS templates.

The rise of Large Language Models (LLMs) has revolutionized our comprehension of intelligence bringing us closer to Artificial Intelligence. Since their introduction, researchers have actively explored the applications of LLMs across diverse fields, significantly elevating capabilities. Cybersecurity, traditionally resistant to data-driven solutions and slow to embrace machine learning, stands out as a domain. This study examines the existing literature, providing a thorough characterization of both defensive and adversarial applications of LLMs within the realm of cybersecurity. Our review not only surveys and categorizes the current landscape but also identifies critical research gaps. By evaluating both offensive and defensive applications, we aim to provide a holistic understanding of the potential risks and opportunities associated with LLM-driven cybersecurity.

Previous studies have shown that demonstrations can significantly help Large Language Models (LLMs ) perform better on the given tasks. However, this so-called In-Context Learning ( ICL ) ability is very sensitive to the presenting context, and often dozens of demonstrations are needed. In this work, we investigate if we can reduce the shot number while still maintaining a competitive performance. We present SeCoKD, a self-Knowledge Distillation ( KD ) training framework that aligns the student model with a heavily prompted variation, thereby increasing the utilization of a single demonstration. We experiment with the SeCoKD across three LLMs and six benchmarks focusing mainly on reasoning tasks. Results show that our method outperforms the base model and Supervised Fine-tuning ( SFT ), especially in zero-shot and one-shot settings by 30 when evaluated on new tasks, which is more robust than Supervised Fine-tuning.

Unstructured data encompasses a plethora of Personal Identifiable information (PII) represented in both standard (e.g. an email address) and unconventional forms (e.g. slang or emojis). Common examples, arise in social media posts but also more recently, in mesh data scenarios where multiple distributed data owners, might share fragments of their data to enable operations such as service compositions. Correctly identifying PII occurrences in such shared data, is crucial in ensuring that the participating entities adhere to the legal and ethical requirements of privacy legislation. In this paper, we present a toolbox of transformer models and comparatively assess the effectiveness and efficiency of such models in discovering PII in unstructured data. To this end, we evaluated state-of-the-art transformer models such as BERT, RoBERTa and XLNet using datasets containing upto 4.19 million tokens. Our results indicate that XLNet generates a lower false negative rate (0.059) than BERT (0.074) and RoBERTa (0.067). Comparative fine-tuning and evaluation among nine transformer models shows that the BERT models achieve 0.92 macro F1-score, a 5

In the context of programming education, so-called auto-graders allow learners to receive automated feedback on their submissions. Because assessing learners' code typically involves executing the learners' untrusted code, this commonly used mechanism poses a significant security risk for these systems. Since auto-graders are mostly employed in the context of large-scale learning environments, such as universities or Massive Open Online Courses (MOOCs), security considerations are especially important. In this paper, we first introduce our auto-grader CodeOcean, which is regularly used in MOOCs with thousands of active learners, and in university contexts. As the execution of untrusted code can entail severe security implications, ensuring that the application contains no security vulnerabilities is essential. Hence, we partnered with a security consultancy to assess our auto-grader system landscape through a professional penetration test. This work presents the findings and countermeasures resulting from the performed security analysis for CodeOcean. We contextualize overarching enhancements for three main categories of threat vectors to auto-grader systems. Implementing these in any auto-grader system can improve the security and prevent learners from manipulating the assessment of their code. We also discuss the potential consequences of hardening an auto-grader, such as a reduced system performance. Therewith, we provide valuable recommendations for educators, researchers, and system designers to improve the security of auto-graders in the future, supporting their usage in even larger settings or in the context of exams.

Novel class discovery (NCD) aims to infer novel categories in an unlabeled dataset by leveraging prior knowledge of a labeled set comprising disjoint but related classes. Given that most existing literature focuses primarily on utilizing supervised knowledge from a labeled set at the methodology level, this paper considers the question: Is supervised knowledge always helpful at different levels of semantic relevance? To proceed, we first establish a novel metric, so-called transfer flow, to measure the semantic similarity between labeled/unlabeled datasets. To show the validity of the proposed metric, we build up a large-scale benchmark with various degrees of semantic similarities between labeled/unlabeled datasets on ImageNet by leveraging its hierarchical class structure. The results based on the proposed benchmark show that the proposed transfer flow is in line with the hierarchical class structure; and that NCD performance is consistent with the semantic similarities (measured by the proposed metric). Next, by using the proposed transfer flow, we conduct various empirical experiments with different levels of semantic similarity, yielding that supervised knowledge may hurt NCD performance. Specifically, using supervised information from a low-similarity labeled set may lead to a suboptimal result as compared to using pure self-supervised knowledge. These results reveal the inadequacy of the existing NCD literature which usually assumes that supervised knowledge is beneficial. Finally, we develop a pseudo-version of the transfer flow as a practical reference to decide if supervised knowledge should be used in NCD. Its effectiveness is supported by our empirical studies, which show that the pseudo transfer flow (with or without supervised knowledge) is consistent with the corresponding accuracy based on various datasets. Code is released at https://github.com/J-L-O/SK-Hurt-NCD

Despite advances in image recognition, recognizing novel categories in unlabeled data remains challenging for machine learning methods, even though humans can perform this task with ease. A recently developed setting to tackle this problem is Generalized Category Discovery (GCD), in which the task is to, given a labeled dataset, classify an unlabeled dataset, where the unlabeled dataset contains both known classes and novel classes that do not appear in the labeled data. Existing GCD methods mostly focus on learning strong image representations, on which they then apply a clustering algorithm such as k-means. Despite obtaining good performance, they do not fully exploit the potential of the learned features due to the simple nature of the clustering mechanism. To address this issue, we make use of the fact that local neighborhoods in self-supervised feature spaces are highly homogeneous. We leverage this observation to develop Guided Cluster Aggregation (GCA), a hierarchical approach that first groups the data into small clusters of high purity, then aggregates them into larger clusters. Experiments show that GCA outperforms semi-supervised k-means in most cases, especially in fine-grained classification tasks. Code available at https://github.com/J-L-O/guidedcluster-aggregation.

Massive Open Online Courses (MOOCs) are a popular form of online education that often attracts a huge and heterogeneous group of learners with diverse interests and backgrounds. However, most MOOCs follow a one-size-fits-all approach, providing a fixed order of learning materials and expecting all learners to follow this recommended path. Thus, they neither motivate nor support their learners in adapting the courses to their individual preferences. In the work at hand, we tackle this issue by introducing and evaluating the concept of flexible learning paths in MOOCs. We, therefore, establish a network of dependencies between course content, omit intermediate deadlines, and thereby rethink the way learners interact with the course. By presenting learners with a non-linear course format, we encourage them to create their individual learning paths based on instructor-defined dependencies and their personal interests. Our evaluation of flexible learning paths within a programming MOOC shows that learners chose many different learning paths. Despite achieving similar results in individual tasks compared to learners using the traditional course structure, they engaged with less course content, resulting in a slight decrease in their overall performance. This may indicate a lack of self-regulatory learning skills, with learners struggling to organise their work without instructor-given deadlines. However, the flexible course format significantly increased the motivation of learners. By introducing and evaluating the concept of flexible learning paths in MOOCs, this work provides valuable insights into the individualisation of online education.

Phishing is still one of the prevalent threats targeting private persons and organizations. Current teaching best practices often advocate cue-based investigation methods. Previous research primarily confronted participants with phishing emails showing such indicators to assess the success of different education measures. Our large-scale mixed-methods study challenges the behavior of 4,729 participants with four phishing emails that lack technical cues. The phishing emails concerned entirely fictitious entities and were directed at participants in their private lives, recruited from the online education platform openHPI. For our analysis, we apply the human-in-the-loop model for interaction with phishing content to investigate participant behavior when their learned best practices for detection fail. The primary indicator of enhanced phishing resiliency observed in our study was awareness of missing context to the supposed entity. Such context is often successfully enhanced by web searches, significantly contributing to decreased phishing susceptibility.

Generalized Class Discovery (GCD) seeks to identify both known and unknown categories within an unlabeled dataset, utilizing the knowledge from a labeled dataset of known classes. Existing research implicitly/explicitly assumes that the frequency of occurrence for each category, whether known or unknown, is approximately the same in the unlabeled data. However, real-world scenarios often exhibit a long-tailed distribution of visual classes, where known or common classes appear more frequently than unknown or rare ones. Addressing this discrepancy, we introduce a new challenge: Imbalanced Generalized Category Discovery (IGCD), which deals with an imbalanced distribution in unlabeled data, favoring known over unknown classes. To tackle this, we propose a novel Optimal Transport-based Expectation Maximization framework for Imbalanced Generalized Category Discovery (Otem-IGCD) by aligning the marginal class prior distribution. Otem-IGCD also incorporates a systematic mechanism for estimating the imbalanced class prior distribution under the GCD setup. Our comprehensive experiments reveal that Otem-IGCD surpasses previous state-of-the-art GCD methods by achieving an improvement of approximately 2 - 4% on CIFAR100 and 15 - 19% on ImageNet-100, indicating its superior effectiveness in solving the Imbalanced GCD problem.

Fine-tuning and inference on Large Language Models like BERT have become increasingly expensive regarding memory cost and computation resources. The recently proposed computation-flexible BERT models facilitate their deployment in varied computational environments. Training such flexible BERT models involves jointly optimizing multiple BERT subnets, which will unavoidably interfere with one another. Besides, the performance of large subnets is limited by the performance gap between the smallest subnet and the supernet, despite efforts to enhance the smaller subnets. In this regard, we propose layer-wise Neural grafting to boost BERT subnets, especially the larger ones. The proposed method improves the average performance of the subnets on six GLUE tasks and boosts the supernets on all GLUE tasks and the SQuAD data set. Based on the boosted subnets, we further build an inference framework enabling practical width- and depth-dynamic inference regarding different inputs by combining width-dynamic gating modules and early exit off-ramps in the depth dimension. Experimental results show that the proposed framework achieves a better dynamic inference range than other methods in terms of trading off performance and computational complexity on four GLUE tasks and SQuAD. In particular, our best-tradeoff inference result outperforms other fixed-size models with similar amount of computations. Compared to BERT-Base, the proposed inference framework yields a 1.3-point improvement in the average GLUE score and a 2.2-point increase in the F1 score on SQuAD, while reducing computations by around 45%.

In the evolving landscape of educational technology, integrating Artificial Intelligence (AI) into educational assessment creation has emerged as a key area of innovation. This paper compares three methods for generating Multiple Choice Questions (MCQs): (1) without the use of generative AI, (2) with unguided usage of ChatGPT, and (3) with a specialized AI-powered micro-app designed specifically for MCQ generation. The micro-app, named the "MCQ Generator", allows users to set preferences such as difficulty level, number of distractors, and inclusion of hints/feedback, tailoring the final prompt based on these selections. Our study involves instructional designers creating MCQs for hypothetical courses, with educators then evaluating the quality of these questions using a rubric-based approach. The results reveal that AI-assisted methods significantly enhance the efficiency and quality of MCQ generation compared to non-AI methods. Notably, the micro-app demonstrates potential advantages over ChatGPT, offering a more user-friendly interface and a lower barrier to entry for educators. These findings suggest that while ChatGPT can enhance the MCQ creation process, AI micro-apps may provide more tailored functionalities that further streamline educators’ workflows. This paper presents empirical evidence on the utility of AI in educational content development. By doing so, it contributes to the broader discourse on the transformative potential of AI in educational assessment, with a particular focus on online education. Additionally, it explores methods to increase AI adoption within the field.

In recent years, there has been a growing interest in using Machine Learning (ML), especially Deep Learning (DL) to solve Network Intrusion Detection (NID) problems. However, the feature distribution shift problem remains a difficulty, because the change in features’ distributions over time negatively impacts the model’s performance. As one promising solution, model pretraining has emerged as a novel training paradigm, which brings robustness against feature distribution shift and has proven to be successful in Computer Vision (CV) and Natural Language Processing (NLP). To verify whether this paradigm is beneficial for NID problem, we propose SwapCon, a ML model in the context of NID, which compresses shift-invariant feature information during the pretraining stage and refines during the finetuning stage. We exemplify the evidence of feature distribution shift using the Kyoto2006+ dataset. We demonstrate how pretraining a model with the proper size can increase robustness against feature distribution shifts by over 8%. Moreover, we show how an adequate numerical embedding strategy also enhances the performance of pretrained models. Further experiments show that the proposed SwapCon model also outperforms eXtreme Gradient Boosting (XGBoost) and K-Nearest Neighbor (KNN) based models by a large margin.

With the increase in the prevalence of Security Information and Event Management Systems (SIEMs) in today's organizations, there is a growing interest in data-driven threat detection. In this research, we formulate malware detection as a large-scale graph mining and inference problem using host-level system events/logs. Our approach is built on two basic principles: guilt-by-association and exempt-by-reputation, with the intuition, that an adversary's resources are limited; hence, reusing infrastructures and techniques is inevitable. We present MalLink, a system that models all host-level process activities as a Heterogeneous Information Network (HIN). The HIN emphasizes shared characteristics of processes/files across the enterprise, e.g., parent/sub-processes, written/read files, loaded libraries, registry entries, and network connections. MalLink then propagates maliciousness from a set of previously known malicious entities to obtain a set of previously unknowns. MalLink was deployed in a real-world setting, next to the SIEM system of a large international enterprise, and evaluated using 8 days (20 TB) of EDR logs collected from all endpoints within the organization. The results demonstrate high detection performance (F1-score of 0.83), particularly when manually investigating the 50 highest scored files with no prior, 37 are found malicious. This demonstrates MalLink's capability to detect previously unknown malicious files.

Cybersecurity education is often perceived as necessary particularly for laypersons, as experts in the field are usually expected to be aware of the risks posed by human-centered attacks such as phishing. In a lab study with 48 participants from IT-related study programs, we studied their phishing investigation behavior using an eye tracker across three email classification sessions. Between the first two sessions, participants received additional training on detecting phishing attacks. The third session, one week later served to measure retention of performance. Exposure to the teaching material particularly showed to decrease investigation time required for the classification. Further, it helped participants focus on the important indicators inside the phishing emails.

Generalized class discovery (GCD) aims to infer known and unknown categories in an unlabeled dataset leveraging prior knowledge of a labeled set comprising known classes. Existing research implicitly/explicitly assumes that the frequency of occurrence for each category, whether known or unknown, is approximately the same in the unlabeled data. However, in nature, we are more likely to encounter known/common classes than unknown/uncommon ones, according to the long-tailed property of visual classes. Therefore, we present a challenging and practical problem, Imbalanced Generalized Category Discovery (ImbaGCD), where the distribution of unlabeled data is imbalanced, with known classes being more frequent than unknown ones. To address these issues, we propose ImbaGCD, A novel optimal transport-based expectation maximization framework that accomplishes generalized category discovery by aligning the marginal class prior distribution. ImbaGCD also incorporates a systematic mechanism for estimating the imbalanced class prior distribution under the GCD setup. Our comprehensive experiments reveal that ImbaGCD surpasses previous state-of-the-art GCD methods by achieving an improvement of approximately 2 - 4% on CIFAR-100 and 15 - 19% on ImageNet-100, indicating its superior effectiveness in solving the Imbalanced GCD problem.

Optimizing General Matrix Multiplication (GEMM) on GPU platforms has become increasingly important due to the scaling demands of modern deep neural network research. While substantial progress has been made in accelerating high-precision GEMM, optimizing lower-bit GEMM remains an open problem. The CUTLASS library offers highly optimized low-bit GEMM based on tensor cores, but performance varies significantly with tile and pipeline settings across different GPUs. We introduce a novel auto-tuning framework for low-bit CUTLASS GEMM that employs a neural network model to predict optimal GEMM template parameters for target GPUs. This model was trained on a synthetic dataset featuring various matrix sizes from different Ampere GPUs and evaluated on these GPUs. In the test dataset, our method achieved an accuracy of up to 92.9%. Real-time evaluations of low-bit data types on the A100 GPU demonstrated speedups of up to 2.03× for GEMM and 1.44× for the linear layer compared to the default templates.

Binary Neural Networks (BNNs) offer a promising avenue toward achieving efficient deep-learning models but are hindered by the inherent challenge of aligning noisy floating-point gradients with binary parameters. To address this, we introduce Diode, a groundbreaking optimizer designed explicitly for BNNs that bridges this gap by utilizing the gradient's sign information in a unique, latent-weight-free approach. By focusing on the gradient sign's lower-order moment estimate for parameter updates, Diode uniformly fine-tunes binary parameters, significantly enhancing model convergence without the dependency on 32-bit latent weights or embedding buffers. This paper showcases Diode's superior performance through comprehensive evaluations on a variety of vision and Natural Language Processing (NLP) tasks. Remarkably, Diode advances the state-of-the-art by increasing BNext-18 Top-1 accuracy on ImageNet ILSVRC2012 by 0.96\% with eightfold fewer training iterations. In the case of ReActNet, Diode not only matches but slightly exceeds previous benchmarks without resorting to complex multi-stage optimization strategies, effectively halving the training duration. Additionally, Diode proves its robust generalization capability on the binary BERT architecture within the GLUE benchmark, outperforming the existing BiT design by 3.3\% without data augmentation and establishing a new SOTA accuracy of 78.8\% with augmentation. The implementation of Diode will be made available to the public at: (insert the link in the final version).

Online education has become an integral part of everyday life. As one form of online education, traditional Massive Open Online Courses mostly rely on video-based learning materials. To enhance accessibility and provide more variety of the learning content, we studied how podcasts can be integrated into online courses. Throughout three studies, we investigated the acceptance and impact of podcasts made available to learners on the online education platform openHPI. Throughout the studies we applied different methodologies, such as a Posttest-Only Control Group study, and a Static-Group Comparison. In the initial two studies, we identified that podcasts can serve as reasonable addition to MOOCs, enabling additional learning just as well as videos, and investigated the optimal podcast design for our learners. In one of our six-week courses on cybersecurity with more than 1,500 learners, alongside the third study, we identified that consuming an additional podcast can increase learning outcomes by up to 7.9%. In this manuscript, we discuss the applied methodologies and provide reasoning behind design decisions concerning, e.g., the podcast structure or presentation to be taken as inspiration for other educators.