Just-in-time defect prediction identifies defect-inducing commits at commit time to focus quality assurance efforts, but its validity depends on the quality of underlying defect datasets, typically generated using SZZ-based algorithms. Because the correctness of these datasets is discussed frequently, this study sets out to quantify how residual SZZ errors propagate into just-in-time defect prediction evaluation. We construct a manually validated ground-truth sample covering 420 commits across 6 open-source projects and provide a stratified, weighted evaluation protocol for estimating population-level performance. Using this sample, we evaluate PR-SZZ-generated datasets and reassess state-of-the-art model performance. We find that PR-SZZ generates wrong defect labels for approximately 12% of commits, mainly through false positives. When just-in-time defect prediction models learned on these defect datasets are evaluated against manual ground truth rather than PR-SZZ labels, average model recall drops from 0.72 to 0.64 and precision from 0.27 to 0.19, indicating that SZZ-based evaluations can substantially misrepresents real model performance. We also find that previously reported performance improvements gained by feature engineering only partly persist under manual evaluation, suggesting that added model capacity may capture systematic labeling errors rather than genuine defect-inducing behavior, particularly as feature characteristics can correlate with SZZ mechanisms. Overall, our findings highlight the need for reliable ground-truth inference and rigorous evaluation practices.

Decentralized unmanned aerial vehicle (UAV) swarms can accomplish demanding tasks through cooperation due to their inherent robustness and scalability. However, existing testing methods typically involve randomly inducing a failure—an approach that may not degrade system performance as much as a targeted failure would. This leaves a critical gap in understanding the true robustness of a swarm. For example, one UAVs in a group of UAVs transporting cargo could lose power, and the overall swarm performance may degrade, e.g., transportation time may become longer than expected. However, the failure of some UAVs may cause greater performance degradation than the failure of others. In this paper, we propose a systematic methodology for generating test cases that challenge a UAV swarm’s robustness as much as possible within computational constraints. The methodology incorporates techniques from scenario-based testing, including search-based test case generation, and is repeated for a range of functionally relevant swarm sizes. We demonstrate the methodology with a case study that is based on the concept of a UAV cargo delivery mission. We used a genetic algorithm to configure the failure behavior that is injected into the test, specifically which UAV to fail and when the failure occurs, with the goal of maximizing degradation in system performance (measured as an increase in flight time). Our results show that a genetic algorithm outperforms random search when the swarm size is less than or equal to the sum of a swarm controller parameter and a test parameter: the number of neighbors that an individual UAV considers to be in its “neighborhood,” and the number of UAVs that are configured to fail during the test, respectively. For swarm sizes larger than this threshold, the fitness landscape of the search becomes much more constrained, and a genetic algorithm does not provide a substantial benefit compared to random search. To support broader adoption, we also propose a generalized testing process for decentralized swarms that accounts for system functionality, robustness, and scalability. Overall, this work provides both a theoretical framework and empirical results to guide the process of generating challenging test cases for decentralized UAV swarms.

This study examines the changing nature of employee data sharing in AI-driven workplaces. AI tools, such as intelligent assistants embedded in daily routines, necessitate a reevaluation of data sharing paradigms due to their continuous data generation and processing. Despite the vast body of literature on data sharing, the constant nature of data sharing in AI-driven environments is rarely investigated. This gap between the practical relevance of such tools and our limited understanding of the changes they entail highlights the need for an extended understanding of the evolving nature of data sharing and its implications. We emphasize the concept of Continuous Data Sharing, transitioning from static decision-making to recognizing data sharing as an ongoing process characterized by changes in data dynamics, employee participation, and unpredictability. In summary, this work underscores the importance of redefining employee data sharing in the context of digital workplace transformation and increasing AI integration.

Legacy and legacy-like enterprise systems often remain difficult to modernize because critical workflows expose limited programmable interfaces and still require manual GUI interaction. This paper reports a pre-deployment evaluation study motivated by the development of legacy-use, an industry-oriented framework for automating such workflows with multimodal LLM agents. During framework development, domain experts helped identify stateful workflows where successful demos are not sufficient: a failed agent run may still leave persistent invalid changes in business or healthcare records. We therefore evaluate computer-use agents using atomicity: a run should either complete the intended workflow correctly or fail without unintended persistent side effects. We construct a domain-expert-informed benchmark of 28 Windows GUI workflows, each specified with an initial state, goal state, and task-specific validator. We compare expert-crafted prompts with prompts generated from screen recordings of expert golden-path executions. Across six hosted computer-use agents, our results show that useful completion, safe failure, and non-atomic side effects are distinct operational profiles. We conclude that workflow capture, state validators, and atomicity-aware acceptance tests should be first-class requirements for AI-based legacy workflow automation.

Military capability is increasingly determined by software. Yet defence platforms are procured on decade-long timescales, while the software and AI models they carry must evolve in days or hours. This paper calls this mismatch the lifecycle paradox, and argues it is the central problem Software-Defined Defence (SDD) must solve. SDD rests on three dimensions: software and systems engineering (design, procurement, certification), AI engineering (sovereignty and trust of learned components), and connectivity and infrastructure engineering (timely exchange of information among sensors, AI, and operators). The proposed path to resilient SDD starts from civilian technologies, addressed through a continuous, DevOps-style loop: model-based systems engineering and simulation-based testing front-load design and verification; tactical connectivity and low-power edge execution carry that design into contested operation; continuous compliance, assurance, and variability management run as cross-cutting concerns. This loop is sustainable given capabilities already proven in automotive, manufacturing, space, and energy. The next step is validating them under adversarial or defence-certified conditions, with short-, medium-, and long-term paths to closing gaps. Closing the SDD gap while preserving civic benefits is a distributed responsibility: researchers must redirect methods toward adversarial conditions; industry must expose tooling to operational needs; policymakers must shape regulatory instruments; and defence agencies must validate results with operators. Recommendations span three horizons: a short-term baseline of adversarial testing and connectivity pilots; a medium-term pipeline of incremental certification; and a long-term validation closing the loop under operational conditions.

Background: The OpenSSF Scorecard is widely used to assess the security posture of open-source software repositories, with the Maintained metric serving as a key indicator of recent maintenance activities, helping users identify actively maintained projects and potentially abandoned dependencies. However, the metric is inherently retrospective, providing only a short-term snapshot based on the past 90 days of repository activity and offering no insight into the future. This limitation complicates risk assessment for developers and organizations that rely on open-source dependencies. Aims: In this paper, we investigate the feasibility of forecasting future maintenance activities as captured by the OpenSSF Maintained score. Method: Focusing on 3,220 GitHub repositories linked to one of the top 1

Reviewing software changes is a critical activity that helps prevent the introduction of defects, ultimately saving development time and reducing costs. Just-in-time defect prediction has emerged as a promising approach to support this process by estimating the likelihood of defects in newly submitted commits. Effort-aware evaluations were proposed to better manage developers’ limited time and to analyze the applicability of defect prediction approaches. However, current effort-aware approaches neglect the time-dependent nature of software engineering when evaluating the performance and rank commits to find most defective commits with limited effort. This reveals a gap between the evaluation of models in research and their application in practice, where a timely decision for every single commit is needed. To assess the impact of effort-aware evaluations on the applicability of defect prediction approaches, we quantify their limitations and provide insights into the implications for future research. Our findings show that effort-aware metrics can overestimate the proportion of defects identified within a limited inspection budget by up to 55

Open Source Software (OSS) libraries form the backbone of modern software systems, yet their long-term sustainability often depends on maintainers being reachable for support, coordination, and security reporting. In this paper, we empirically analyze the availability of contact information - specifically e-mail addresses - across 686,034 Python libraries on the Python Package Index (PyPI) and their associated GitHub repositories. We examine how and where maintainers provide this information, assess its validity, and explore coverage across individual libraries and their dependency chains. Our findings show that 81.6

High‑quality test datasets are essential for evaluating the reliability of Deep Neural Networks (DNNs). Traditional test adequacy metrics lack empirical evidence that datasets evaluated using them can also expose diverse DNN faults. Mutation testing provides a fault-oriented perspective on dataset quality by assessing its ability to expose injected faults, as measured by the mutation score (MS). However, its high computational cost limits its practical adoption, motivating the search for computationally efficient proxies.In this paper, we first systematically evaluate the correlation between MS and existing test adequacy criteria using 233 mutants spanning 15 DNN fault categories across four well-established computer vision benchmarks. Our results show that structural adequacy metrics exhibit no meaningful correlation with MS. Non‑structural metrics, such as Likelihood and Distance‑based Surprise Coverage, show only weak positive correlations, indicating their limited value as MS proxies. We then compare covariance-aware latent-space deviation, represented by Mahalanobis Distance-based Surprise Coverage (MDSC), with direct latent-space dispersion, operationalized through Latent Space Class Dispersion (LSCD). Although both metrics achieve comparable computational efficiency, they differ in how they characterize class distributions and in their relationship with MS. MDSC achieves a mean Pearson correlation of 0.505 and a mean Spearman rank correlation of 0.329 with MS. In comparison, LSCD achieves a mean Pearson correlation of 0.844 and a mean Spearman correlation of 0.507, indicating that direct latent-space dispersion exhibits a stronger relationship with MS under the studied settings. These empirical insights indicate that while both methods offer scalable approximations to mutation testing, LSCD shows a stronger correlation with the mutation score.

Large language models (LLMs) perform strongly on general-purpose code generation, yet their applicability to enterprise domain-specific languages (DSLs) remains underexplored, especially for repository-scale change generation spanning multiple files and folder structures from a single natural-language (NL) instruction. We report an industrial case study at BMW that adapts code-oriented LLMs to generate and modify project-root DSL artifacts for an Xtext-based DSL that drives downstream Java/TypeScript code generation. We develop an end-to-end pipeline for dataset construction, multi-file task representation, model adaptation, and evaluation. We encode DSL folder hierarchies as structured, path-preserving JSON, allowing single-response generation at repository scale and learning cross-file dependencies. We evaluate two instruction-tuned code LLMs (Qwen2.5-Coder and DeepSeek-Coder, 7B) under three configurations: baseline prompting, one-shot in-context learning, and parameter-efficient fine-tuning (QLoRA). Beyond standard similarity metrics, we introduce task-specific measures that assess edit correctness and repository structural fidelity. Fine-tuning yields the most significant gains across models and metrics, achieving high exact-match accuracy, substantial edit similarity, and structural fidelity of 1.00 on our held-out set for multi-file outputs. At the same time, one-shot in-context learning provides smaller but consistent improvements over baseline prompting. We further validate practical utility via an expert developer survey and an execution-based check using the existing code generator.

Assessing the robustness of safety-critical deep learning (DL) systems is of utmost importance, as these systems can cause harm when deployed in the real world. Metamorphic testing (MT) is one commonly used method to evaluate the robustness of DL systems, as it does not require expensive labelled ground truth data. This paper tackles two challenges: (1) One challenge in regulated domains such as the automotive industry is to provide a traceable argumentation of why a certain metamorphic relation (MR) was chosen. We adopt the idea of defect-based testing to MT and argue that an MR is traceable if it can be linked to a defect hypothesis. We demonstrate how to assess the robustness of safety-critical DL systems using the example of LiDAR object detectors. To this end, we create three new MRs for the LiDAR domain and identify five MR that can be reused by adapting them from related domains. Our experiments on the nuScenes dataset with three different object detectors produce 3.9 million test verdicts, of which 0.7 million are test failures. This shows that our defect-based MR effectively uncover failures. (2) A second challenge resulting from executing numerous metamorphic test cases is that MT can lead to the generation of an impractically high number of failures. We show how to prioritize the most critical failures, such as failures that occur close to the ego vehicle. By prioritizing, we reduced the observed 685,000 failures to 5397 safety-critical failures corresponding to a 127-fold reduction.

Agile methods have shaped the development of enterprise software systems during the last two decades. However, many modern cyber-physical systems (CPS) are still developed in as yet waterfall-like processes. The consequence is that CPS development misses out on such advantages of agile methods as handling changing requirements providing fast updates, or dealing with fast feedback on product quality. This is especially problematic today, when the software in CPS systems is more networked than ever, requiring updates to keep pace in an ever evolving network-connected technical environment, as well as patching too often software-induced cyber-security vulnerabilities. In sum, modern CPS must be developed so as to meet the need for updates at intervals of rapidly accelerating frequency. In this paper, we discuss the lack of systematic cross-model consistency management as one of the reasons why established agile methods are not used in CPS development. We present a road map that leads to systematic consistency management, laying the foundations of novel agile methods in CPS development. We discuss solutions in the context of model-driven automotive systems engineering. This domain especially can serve as a litmus test of agility in CPS development, because automotive systems engineering stands to benefit substantially from agile methods to address such pressing issues as strong assurance of dependability and configurability while also offering the flexibility of software over-the-air updates.

Flaky tests produce non-deterministic outcomes, passing or failing inconsistently when executed under conditions that developers intend to maintain constant. Within continuous integration (CI) environments, these tests disrupt development workflows by causing pipeline failures unrelated to intended changes. Modern CI systems commonly rerun failed tests within the same job to identify flaky failures. Existing research predominantly analyzes data from this detection mechanism, overlooking undetected flaky failures that cause the most severe disruptions. We analyzed 8.8 billion test executions from four industry-scale projects over two-month periods. All apparently deterministic failures were systematically validated through reruns in fresh environments within the projects’ original CI infrastructure to identify undetected flaky failures. We found that, across all projects, undetected flaky failures account for 9.8%–16.3% of failed pipeline runs. These flaky failure rates spike temporarily, triggered primarily by code changes and test reordering. While failures concentrate heavily in small test case subsets, the most disruptive test cases with undetected flaky failures remain largely invisible to tools relying on current detection mechanisms. Additionally, test environments significantly influence flakiness, with up to 3× variation in flake rates between them. These findings reveal fundamental limitations in current detection strategies and highlight the need for comprehensive approaches to flaky test management. Our publicly available dataset of 154,000 flaky test cases with 1.7 million flaky failures provides an empirical foundation for developing improved strategies for industrial CI environments.

Software systems rely heavily on open source software (OSS) libraries, which offer benefits but also pose risks. When vulnerabilities arise, the OSS community may struggle to address them due to inactivity or lack of resources. Research highlights the link between OSS maintenance and financial support. To sustain the OSS ecosystem, maintainers should register on donation platforms and link these profiles on their project pages, enabling financial support from users and industry stakeholders. However, a detailed study on donation platform usage in OSS is missing. This study analyzes the adoption of donation platforms in the PyPI ecosystem. For each PyPI library, we retrieve assigned URLs, dependencies, and, when available, owner type and GitHub donation links. Using PageRank, we analyze different subsets of libraries from both a library and dependency chain perspective. Our findings reveal that donation platform links are often omitted from PyPI project pages and instead listed on GitHub repositories. GitHub Sponsors is the dominant platform, though many PyPI-listed links are outdated, emphasizing the need for automated link verification. Adoption rates vary significantly across libraries and dependency chains: while individual PyPI libraries show low adoption, those used as dependencies have much higher usage. This suggests that many dependencies actively seek financial support, benefiting developers relying on PyPI libraries.

Autonomous vehicles rely on deep learning (DL) models like object detectors and traffic sign classifiers. Assessing the robustness of these safety-critical components requires good test cases that are both realistic, lying in the distribution of the real-world data, and cost-effective in revealing potential failures. Unlike previous methods that use adversarial attacks on the pixel space, our approach identifies latent space-aware test cases using a conditional variational autoencoder (CVAE) through three steps: (1) Train a CVAE on the dataset. (2) Generate test cases by computing adversarial examples in the CVAE’s latent space. (3) Cluster challenging test cases based on their latent representations. The resulting clusters characterize regions that reveal potential defects in the DL model, which require further analysis. Our results show that our approach is capable of generating failing test cases for all classes of the MNIST and GTSRB datasets in a purely data-driven way, surpassing the baseline of random latent space sampling by up to 75 times. Finally, we validate our approach by detecting previously introduced faults in a faulty DL model. We suggest complementing expert-driven testing methods with our purely data-driven approach to uncover defects experts otherwise might miss. To strengthen transparency and facilitate replication, we provide a replication package and digital appendix to make our code, models, visualizations, and results publicly available.

Existing research has demonstrated promising results when applying large language models (LLMs) to detect security vulnerabilities in source code. However, these studies have been exclusively evaluated on benchmarks from open-source systems, using publicly known vulnerabilities that are likely part of the LLMs’ training data. This raises concerns that reported performance metrics may be inflated due to data contamination, providing a misleading view of the models’ actual capabilities.In this paper, we quantify this effect with a case study that evaluates five frontier LLMs on two carefully curated datasets: CWE-Bench-Java (an open-source dataset) and TS-Vuls (based on a closed-source commercial codebase). To provide a second angle, we also split CWE-Bench-Java by CVE record date to explore temporal contamination based on LLM knowledge cutoff dates.Our results reveal that the average F1 score dropped by approximately 20 percentage points when comparing the open-source to the closed-source dataset. Additionally, the precision drops from 56% to 34% on average, which is statistically significant (p < 0.05) for four of five models. This declining trend is consistent across all tested LLMs and metrics. In contrast, the results for the temporal split on the open-source data are inconclusive, suggesting that using a knowledge cutoff may reduce but does not ensure the elimination of contamination effects.Although our study is based on a single closed-source system and thus not generalizable, these findings provide the first empirical evidence that evaluating LLM-based vulnerability detection on open-source benchmarks may lead to overly optimistic results. This motivates the inclusion of closed-source datasets in future LLM evaluations.

Ensuring confidentiality during multi-model consistency checking is a critical challenge in collaborative systems engineering. However, it is not yet clear how to assess and compare current and future solutions for multi-model consistency checking with regarding to confidentiality. Therefore, this paper introduces a formalized system model for confidentialitypreserving consistency checking. A formalization of the confidentiality preservation capability of such a system model is proposed to assess whether a given consistency checking system prevents unauthorized information leakage under specific assumptions. Based on these definitions and formalizations, we present an assessment method where an abstract system model is derived from either an implemented or conceptualized consistency checking system, and our confidentiality formalization is applied to assess the system’s guarantees for model data confidentiality. Our approach provides system and software engineers with a structured method to assess the confidentiality preservation capability regarding model data in their multi-model consistency checking systems, helping them to identify potential weaknesses and guiding improvements to enhance confidentiality where needed. To demonstrate the applicability of our framework, we apply it to an existing prototype of a partly confidentialitypreserving consistency checking system, as well as two improved versions, assessing their confidentiality preservation strengths and areas for improvement.