Ensuring data ownership and traceability of unauthorised redistribution are central to safeguarding intellectual property in shared data environments. Data fingerprinting addresses these challenges by embedding recipient-specific marks into the data, typically via content modifications. We propose NCorr-FP, a Neighbourhood-based Correlation-preserving Fingerprinting system for structured tabular data that preserves the statistical fidelity of data. The method uses local record similarity and density estimation to guide the insertion of fingerprint bits. The embedding logic is then reversed to extract the fingerprint from a potentially modified dataset. Extensive experiments confirm its effectiveness, fidelity, utility, and robustness. Results show that fingerprints are virtually imperceptible, with minute Hellinger distances and KL divergences, even at high embeddeding rates. The system also maintains high data utility for downstream predictive tasks. The method achieves 100% detection confidence under substantial data deletions and remains robust against adaptive- and collusion attacks. Satisfying all these requirements concurrently on mixed-type datasets highlights the strong applicability of NCorr-FP to real-world data settings.

Cyber threat intelligence (CTI) analysts must answer complex questions over large collections of narrative security reports. Retrieval-augmented generation (RAG) systems help language models access external knowledge, but traditional vector retrieval often struggles with queries that require reasoning over relationships between entities such as threat actors, malware, and vulnerabilities. This limitation arises because relevant evidence is often distributed across multiple text fragments and documents. Knowledge graphs address this challenge by enabling structured multi-hop reasoning through explicit representations of entities and relationships. However, multiple retrieval paradigms, including graph-based, agentic, and hybrid approaches, have emerged with different assumptions and failure modes. It remains unclear how these approaches compare in realistic CTI settings and when graph grounding improves performance. We present a systematic evaluation of four RAG architectures for CTI analysis: standard vector retrieval, graph-based retrieval over a CTI knowledge graph, an agentic variant that repairs failed graph queries, and a hybrid approach combining graph queries with text retrieval. We evaluate these systems on 3,300 CTI question-answer pairs spanning factual lookups, multi-hop relational queries, analyst-style synthesis questions, and unanswerable cases. Results show that graph grounding improves performance on structured factual queries. The hybrid graph-text approach improves answer quality by up to 35 percent on multi-hop questions compared to vector RAG, while maintaining more reliable performance than graph-only systems.

In live systems, where the underlying document corpora evolve frequently, a query executed at two different points in time can yield two different result sets. Although not important in traditional web search settings, domains, such as patent retrieval or systematic literature reviews, rely on the time of execution to obtain the relevant result set. To ensure reproducibility and auditability of their search results researchers in these fields usually rely on Boolean Retrieval. This is because sparse and dense retrieval methods do not satisfy this requirement, as sparse retrieval relies on global term and document statistics, and dense retrieval relies on document embeddings. These values and vectors are subject to change if the document corpora are updated and therefore change the ranking as well. In this paper we present TimIR ( https://timir.ds-ifs.tuwien.ac.at ), a dashboard to explore the evolution of Information Retrieval publications over time, while showcasing a hybrid retrieval system that allows researchers to recreate sparse rankings for historical states of the document corpora. Having search result lists that can be recreated, with a system like TimIR for example, makes it possible to cite such sets of data, especially when the results are used for further down-the-stream research, without the impracticality of additionally storing the search results themselves. Furthermore, it possible to compare rankings for a particular query over time, or to explore the literature available during a specific time period, for example the time when Karen Spärck Jones introduced TF-IDF.

Cyber threat intelligence (CTI) reports are critical for understanding adversarial behaviors but are often unstructured and lack sufficient labeled data, which makes automated extraction of Tactics, Techniques, and Procedures (TTPs) challenging. This paper addresses these issues by introducing TTPFShot, a novel retrieval-based few-shot learning framework that classifies TTPs from CTI texts with minimal labeled data. TTPFShot leverages a vector database to retrieve semantically similar examples from a sentence-based dataset derived from the MITRE ATT CK framework. These examples are used to construct few-shot prompts that guide large language models to accurately map CTI sentences to the appropriate TTP categories. Comprehensive evaluations on both sentence-based and document-based datasets demonstrate that TTPFShot outperforms existing approaches, such as TTPXHunter, by achieving higher precision, recall, and F1 scores. These results underscore the framework’s ability to mitigate data scarcity issues and improve TTP classification accuracy in real-world settings.

The growing demand for open science and data-intensive research highlights the urgent need for robust Research Data Management (RDM) systems in universities. While European institutions such as TU Wien have implemented mature FAIR-compliant infrastructures, Indonesian universities still face challenges of fragmented repositories, limited interoperability, and the absence of institutional policies. This paper presents the development of a FAIR-compliant RDM framework tailored for Institut Teknologi Bandung (ITB), derived from lessons learned at TU Wien. The framework integrates five pillars-Policy & Governance, Infrastructure, Processes & Services, Trust & Quality, and Capacity Building-supported by an implementation roadmap. A prototype system was developed using opensource components (InvenioRDM, DBRepo, JupyterHub) to enable research and business intelligence dashboards. Comparative analysis shows that the proposed framework bridges global best practices with local needs, providing both theoretical contributions to institutional data governance and practical tools for evidence-based decision-making. The outcomes aim to strengthen institutional research transparency, support Indonesia's innovation agenda, and offer a replicable model for other universities.

Model stealing attacks endanger the confidentiality of machine learning models offered as a service. Although these models are kept secret, a malicious party can query a model to label data samples and train their own substitute model, violating intellectual property. While novel attacks in the field are continually being published, their design and evaluations are not standardised, making it challenging to compare prior works and assess progress in the field. This paper is the first to address this gap by providing recommendations for designing and evaluating model stealing attacks. To this end, we study the largest group of attacks that rely on training a substitute model – those attacking image classification models. We propose the first comprehensive threat model and develop a framework for attack comparison. Further, we analyse attack setups from related works to understand which tasks and models have been studied the most. Based on our findings, we present best practices for attack development before, during, and beyond experiments and derive an extensive list of open research questions regarding the evaluation of model stealing attacks. Our findings and recommendations also transfer to other problem domains, hence establishing the first generic evaluation methodology for model stealing attacks.

Keeping up with state-of-the-art in science is increasingly difficult for researchers due to the current pace of publishing. Inspired by previous work, we address this challenge by formulating the task of Narrative Knowledge Delta ( 𝒩𝒦 ) Extraction which focuses on identifying differences between pairs of scientific articles that tackle the same research problem, presented in a narrative form. We create two manually annotated ground truth datasets and one automatically generated dataset of 𝒩𝒦 sentences. Using these datasets, we design and evaluate a 𝒩𝒦 extraction approach from pairs of papers using four LLMs: GPT-4o, GPT-4o-mini, Llama3.1-8b, and Llama3.1-70b. We then apply a scientific fact-checking model to evaluate the LLMs’ 𝒩𝒦 output using manually annotated data as ground truth claims. The results show a general improved performance in few-shot settings when examples from the automatically generated data are incorporated. However, our manual analysis reveals challenges and limitations in creating annotated data for evaluating 𝒩𝒦 extraction by LLMs (Data, prompts, and code are available at: https://github.com/Alaa-Ebshihy/nkd_llm_2024 ).

This paper presents a comprehensive evaluation of multilingual cybersecurity text classification using conventional machine learning (ML) models, sentence-transformer embeddings, and open-source large language models (LLMs). We construct a manually labeled dataset of English and German news articles and benchmark models across zero-shot and fewshot settings while accounting for LLM knowledge cutoffs. Our results show that classic ML models, when combined with highquality embeddings, achieve performance equal to or better than state-of-the-art LLMs. For instance, an Multi-Layer Perceptron (MLP) classifier with multilingual-e5-large embeddings reaches an F1-score of 0.99 in the pre-cutoff setting, matching Qwen2.5-72B's few-shot performance ($F 1=0.99$) post-cutoff. Notably, this level of performance is achieved with over 99% lower computational requirements. Several embedding-based ML pipelines outperform all zero-shot LLMs, highlighting their costefficiency and robustness. These findings challenge the presumed superiority of LLMs and underline the importance of cutoffaware evaluations in practical applications.

Cyber Situational Awareness (CSA) is crucial for understanding and anticipating developments across diverse domains. This paper introduces a novel approach employing advanced Artificial Intelligence (AI) and Natural Language Processing (NLP) techniques to effectively analyze and enrich Cyber Threat Intelligence (CTI) and Open Source Intelligence (OSINT) data. The paper designs an unified CTI and OSINT processing pipeline that integrates named entity recognition (NER), relationship extraction, classification, and summarization, addressing current limitations in CTI analysis. Notably, our evaluation of existing language models revealed significant shortcomings, with general-purpose tokenizers recognizing only 1.62

Machine learning (ML) models were shown to be vulnerable to model stealing attacks, which lead to intellectual property infringement. Among other attack methods, substitute model training is an all-encompassing attack applicable to any machine learning model whose behaviour can be approximated from input-output queries. Whereas previous works mainly focused on improving the performance of substitute models by, e.g. developing a new substitute training method, there have been only limited ablation studies that try to understand the impact the strength of an attacker has on the substitute model's performance. As a result, different authors came to diverse, sometimes contradicting, conclusions. In this work, we exhaustively examine the ambivalent influence of different factors resulting from varying the attacker's capabilities and knowledge on a substitute training attack. Our findings suggest that some of the factors that have been considered important in the past are, in fact, not that influential; instead, we discover new correlations between attack conditions and success rate. In particular, we demonstrate that better-performing target models enable higher-fidelity attacks and explain the intuition behind this phenomenon. Further, we propose to shift the focus from the complexity of target models toward the complexity of their learning tasks. Therefore, for the substitute model, rather than aiming for a higher architecture complexity, we suggest focusing on getting data of higher complexity and an appropriate architecture. Finally, we demonstrate that even in the most limited data-free scenario, there is no need to overcompensate weak knowledge with unrealistic capabilities in the form of millions of queries. Our results often exceed or match the performance of previous attacks that assume a stronger attacker, suggesting that these stronger attacks are likely endangering a model owner's intellectual property to a significantly higher degree than shown until now.

Log data provides crucial insights for tasks like monitoring, root cause analysis, and anomaly detection. Due to the vast volume of logs, automated log parsing is essential to transform semi-structured log messages into structured representations. Recent advances in large language models (LLMs) have introduced the new research field of LLM-based log parsing. Despite promising results, there is no structured overview of the approaches in this relatively new research field with the earliest advances published in late 2023. This work systematically reviews 29 LLM-based log parsing methods. We benchmark seven of them on public datasets and critically assess their comparability and the reproducibility of their reported results. Our findings summarize the advances of this new research field, with insights on how to report results, which data sets, metrics and which terminology to use, and which inconsistencies to avoid, with code and results made publicly available for transparency.

Researchers of all disciplines produce, share, and reuse data as part of everyday research. Most funders require them to manage and document their data using data management plans (DMPs). DMPs are often static documents that researchers create by answering questions in predefined templates at the beginning of the research and, therefore, may become outdated and obsolete as the project progresses. It is essential to keep the DMP up to date at all stages of the research lifecycle since numerous stakeholders and various services participate in data management that depend on information from them. In this paper, we propose a conceptual service architecture that uses machine-actionable data management plans to automate the exchange and synchronization of information between different semi-automated research data management (RDM) services acting on behalf of different stakeholders. To solve the stated problem, we analyze typical use cases in which the DMPs change and formulate requirements based on which we developed the conceptual architecture. We depict the designed architecture through a set of views, namely physical, development, logical, and process, using UML and BPMN representation that describe the processes required to synchronize DMP information among multiple services. We instantiate it by implementing a service that connects a data repository and a DMP tool. Thus, we evaluate to what extent the defined processes help in keeping DMP contents up to date and which criteria must be fulfilled to keep them highly automated. The result of the paper feeds into a larger discussion on streamlining interconnectivity and machine-actionability across planning, tracking, and assessing research phases. It also facilitates consensus building on enhancing the Research Data Alliance’s recommendation for machine-actionable DMPs.

Trusted Research Environments (TREs) enable the analysis of sensitive data under strict security assertions that protect the data with technical, organizational, and legal measures from (accidentally) being leaked outside the facility. While many TREs exist in Europe, little information is available publicly on the architecture and descriptions of their building blocks and their slight technical variations. To highlight on these problems, an overview of the existing, publicly described TREs and a bibliography linking to the system description are provided. Their technical characteristics, especially in commonalities and variations, are analysed, and insight is provided into their data type characteristics and availability. The literature study shows that 47 TREs worldwide provide access to sensitive data, of which two-thirds provide data predominantly via secure remote access. Statistical offices (SOs) make the majority of sensitive data records included in this study available.

Trust is an essential pre-condition for the acceptance of digital infrastructures and services. Transparency has been identified as one mechanism for increasing trustworthiness. Yet, it is difficult to assess to which extent and how exactly different aspects of transparency contribute to trust, or potentially impede it in cases of overwhelming complexity of the information provided. To address these issues, we performed two initial studies to help determining the factors that influence or have impact on trust, focusing on transparency across a range of elements associated with data, data infrastructures and virtual research environments. On one hand, we performed a survey among IT experts in the field of data science focusing on quality aspects in the context of re-using and sharing open source software, assessing issues such as the need for documentation, test cases, and accountability. On the other hand, we complemented this with a set of semi-structured interviews with senior researchers to address specific issues of the degree of transparency achievable with different approaches. They include, for example, the amount of transparency we can achieve with approaches from explainable AI, or the usefulness and limitations of data provenance in determining the suitability of data for reuse and others. Specifically, we consider mechanisms on three levels, i.e. technical, process-oriented as well as social mechanisms. Starting from attributes of trust in the “analogue world”, we aim to understand which of these can be applied in the digital world, how they differ, and what additional mechanisms need to be established, in order to support trust in complex socio-technological processes and their emergent results when the traditional approaches cannot be applied anymore.

Cyber threats are evolving rapidly, making anomaly detection (AD) in system log data increasingly important for detection of known and unknown attacks. The configuration of AD algorithms heavily depends on the data at hand. It often involves a complex feature selection process and the determination of parameters such as thresholds or window sizes. In many cases, configuration requires manual intervention by domain experts, which limits accessibility and effectiveness of AD algorithms. This work introduces a Configuration-Engine (CE), which employs a semi-supervised approach to automate the configuration process or optimize existing configurations. The CE utilizes statistical methods to identify log line properties to recognize meaningful tokens for AD methods to monitor. It categorizes variables by their characteristics and behavior over time, then specifies which log parts a detector should observe, and sets appropriate configuration parameters.The CE was evaluated using four different detectors. Evaluations on different Apache Access and audit datasets containing attack traces showed that the CE achieved an average precision of over 0.94 for Apache and over 0.79 for audit datasets, while maintaining high recall, competing with the performance of expert-crafted configurations. The optimization approach was able to strongly improve the precision of both the CE’s and the experts’ configurations for Apache data in 7 out of 16 cases. Furthermore, the CE’s configurations were significantly dissimilar to each other when generated on audit data, highlighting the importance of automated configuration.

Cyber Situational Awareness (CSA) is an important element in both cyber security and cyber defence to inform processes and activities on strategic, tactical, and operational level. Furthermore, CSA enables informed decision making. The ongoing digitization and interconnection of previously unconnected components and sectors equally affects the civilian and military sector. In defence, this means that the cyber domain is both a separate military domain as well as a cross-domain and connecting element for the other military domains comprising land, air, sea, and space. Therefore, CSA must support perception, comprehension, and projection of events in the cyber space for persons with different roles and expertise. This paper introduces NEWSROOM, a research initiative to improve technologies, methods, and processes specifically related to CSA in cyber defence. For this purpose, NEWSROOM aims to improve methods for attacker behavior classification, cyber threat intelligence (CTI) collection and interaction, secure information access and sharing, as well as human computer interfaces (HCI) and visualizations to provide persons with different roles and expertise with accurate and easy to comprehend mission- and situation-specific CSA. Eventually, NEWSROOM’s core objective is to enable informed and fast decision-making in stressful situations of military operations. The paper outlines the concept of NEWSROOM and explains how its components can be applied in relevant application scenarios.

The rapid advancement of Large Language Models (LLMs) has transformed natural language processing, yet comprehensive evaluation methods are necessary to ensure their reliability, particularly in Retrieval-Augmented Generation (RAG) tasks. This study aims to evaluate and compare the performance of open-source LLMs by introducing a rigorous evaluation framework. We benchmark 20 LLMs using a combination of established metrics such as BLEU, ROUGE, BERTScore, along with and a novel metric, RAGAS. The models were tested across two distinct datasets to assess their text generation quality. Our findings reveal that models like nous-hermes-2-solar-10.7b and mistral-7b-instruct-v0.1 consistently excel in tasks requiring strict instruction adherence and effective use of large contexts, while other models show areas for improvement. This research contributes to the field by offering a comprehensive evaluation framework that aids in selecting the most suitable LLMs for complex RAG applications, with implications for future developments in natural language processing and big data analysis.

Large Generative AI (GAI) models have the unparalleled ability to generate text, images, audio, and other forms of media that are increasingly indistinguishable from human-generated content. As these models often train on publicly available data, including copyrighted materials, art and other creative works, they inadvertently risk violating copyright and misappropriation of intellectual property (IP). Due to the rapid development of generative AI technology and pressing ethical considerations from stakeholders, protective mechanisms and techniques are emerging at a high pace but lack systematisation. In this paper, we study the concerns regarding the intellectual property rights of training data and specifically focus on the properties of generative models that enable misuse leading to potential IP violations. Then we propose a taxonomy that leads to a systematic review of technical solutions for safeguarding the data from intellectual property violations in GAI.

In the era of big data, research has become increasingly data-driven, with vast amounts of information being generated and analyzed to produce new insights and discoveries. This data deluge requires a combination of methods and technologies to store, process, share and preserve research data. With many of the world’s most valuable data being stored in relational databases where it evolves over time as new knowledge is gained and old knowledge invalidated, current repository systems fail to provide researchers with interfaces to conveniently work with this kind of data within their research environments. For this reason, we have developed DBRepo, an institutional data repository for research data in databases (DBRepo) supporting guidelines of the Working Group on Data Citation of the Research Data Alliance. The system has been in use at TU Wien for almost three years now and provides a variety of data science-related interfaces and can be integrated into many workflows and tools. Further, it assists researchers in depositing their datasets by suggesting the table schema (column names, data types, primary key constraints) and it addresses data interoperability issues by suggesting semantic concepts for dataset columns and units of measurements, where applicable. DBRepo is currently in use by six universities globally who use it as data store for hot and cold research data sets. In the paper, we describe their use-cases and provide lessons learned from the various deployments and workflows. Finally, we show how depositing research data into DBRepo increases the data’s visibility.

This paper analyzes the most common problems limiting reproducibility of Information Retrieval research and provides researchers with insights and guidelines to improve the reproducibility of experiments and to allow the verification of obtained results. We conducted a study on 45 reproduction reports off 17 different papers, which have been published at renowned IR conferences. We analyzed the reports qualitatively and quantitatively and looked into the different insights from different groups. Occurring problems are classified into three problem families and 13 categories and afre then analyzed with respect to their influence on the reproduction process as well as on their frequency of appearance over time and per conference. Of these 17 different papers, 14 papers were reproducible to a certain degree without significant differences to the original results, but in many cases not the whole experiment was reproducible due to missing code, information or data. Also, we look at assumptions that were made when reproducing the different papers, as some experiment workflows were incomplete and information was missing. In addition, we propose recommendations to make machine learning research more reproducible and FAIR.