Code review is essential for maintaining software quality but remains time-consuming and cognitively demanding, especially in industrial environments. While language models (LMs) have shown promise in automating code review tasks, most prior work targets dominant programming languages such as Java and Python, leaving industrially relevant languages like C# underexplored. This study presents an empirical evaluation of monolingual C#-specific fine-tuning of three architecturally diverse open-source LMs – a review-specialized pretrained model, a code-pretrained large language model, and a general-purpose instruction-tuned model – across three core automated code review tasks: Code Change Quality Estimation, Review Comment Generation, and Code Refinement. The review-specialized model was fine-tuned across all three tasks, while the two larger models were fine-tuned exclusively for comment generation due to computational constraints, using a dataset combining public benchmarks with proprietary industrial repositories. We systematically examined the effects of programming-language scope (C#-only vs. multilingual) and natural-language composition (English-only vs. English–German), using both automated metrics and expert-based human evaluations. Our results show that the benefits of monolingual fine-tuning are task-dependent: C#-specific adaptation improves review-worthiness detection and enhances lexical alignment and human-perceived quality in comment generation, yet it reduces performance in code refinement, where broader multilingual pretraining proves more effective. English-only finetuning consistently outperforms bilingual configurations in comment quality. Across all tasks, fine-tuned LMs offer substantial efficiency gains over human reviewers, but human judgment remains superior for semantically complex and context-sensitive assessments, suggesting that LMs are best deployed as assistive tools within hybrid review workflows.
TLQD is an unmanned aerial vehicle (UAV) test case generation tool based on quality–diversity algorithms. It employs a two-level structural strategy with behavior-guided search, exploring test cases at different granularities in both global and local regions of the search space. In addition, TLQD includes a feasibility repair operator to handle constraint-violating test cases. This short paper reports the implementation of TLQD in the UAV Testing Tool Competition at ICST 2026.
Individual fairness testing evaluates a machine learning (ML) classifier by estimating its individual fairness ratio (IFr) and has become an important approach for assessing ML classifiers. However, existing studies on ML fairness testing focus exclusively on the Fairness Through Awareness (FTA) setting, where protected attributes are explicitly used. In this study, we introduce the first framework for individual fairness testing under Fairness Through Unawareness (FTU), where protected attributes are not explicitly available. The proposed framework, called IFT ${ }_{\text {unaware }}$, evaluates a classifier by estimating its IFr in the FTU setting. It extends individual fairness testing approaches developed for FTA by incorporating techniques, including protected attribute inference with confidence control, a generalized notion of individual similarity, and statistically guaranteed termination criteria. Our experiments show that IFT $_{\text {UNAWARE }}$ can compute IFr estimates with statistical guarantees and demonstrate the effectiveness of both the generalized similarity notion and confidence-based protected attribute inference. These results highlight both the feasibility and the limitations of assessing individual fairness under FTU and provide a foundation for future methodological advances.
We present an exploratory study on using Large Language Models (LLMs) to generate Alloy formal specifications from both requirements documentation and production source code, and to derive executable test cases from those specifications. We evaluate on two real open-source Python libraries: Flipper, a feature flag management system, and Cerberus, a data validation library. In both cases, the LLM produced workable Alloy specifications and executable tests without any manual correction. For Flipper, our pipeline uncovered a genuine bug that the existing test suite had missed: the library silently accepts duplicate flag names, directly contradicting its documented uniqueness requirement. A direct LLM baseline–generating tests from the same README but skipping the Alloy step–achieved 68 For Cerberus, the code-derived specification captured an implicit abstraction over sized types that the documentation-derived spec omitted, producing two additional tests. Across both libraries, code-based specifications showed lower variance in test generation (mean SD = 2.15) than documentation-based ones (mean SD = 5.0), though whether this generalises remains an open question. Index Terms–formal specifications, Alloy, large language models, automated testing, specification drift, software validation.
Collaborative fuzzing is an approach that addresses the limitations of a single fuzzer by combining the complementary strengths of different fuzzers. However, existing collaborative fuzzing has primarily focused on improving general exploration performance, such as increasing code coverage and maximizing overall bug detection. As a result, it does not sufficiently capture target-oriented requirements, such as rapidly reaching specific target sites or reproducing vulnerabilities. When extending collaborative fuzzing to directed fuzzing scenarios, additional challenges arise. Different fuzzers employ different distance metrics and seed prioritization criteria, making it difficult to compare their performance under a unified standard. Moreover, the effectiveness of each fuzzer may vary depending on the current exploration state, making it challenging to determine which fuzzer should be prioritized during execution. To address these limitations, this dissertation proposes a framework that preserves the strengths of collaborative fuzzing while supporting target-oriented objectives. Specifically, it introduces a common evaluation criterion to enable consistent comparison of fuzzer performance, along with an adaptive collaboration mechanism that dynamically selects the most suitable fuzzer based on the current exploration state. The ultimate goal is to improve both exploration efficiency and vulnerability reproduction performance.
The emergence of Autonomous Vehicles (AVs) has spurred research into testing the resilience of their perception systems, i.e., ensuring that they are not susceptible to critical misjudgements. It is important that these systems are tested not only with respect to other vehicles on the road, but also with respect to objects placed on the roadside. Trash bins, billboards, and greenery are examples of such objects, typically positioned according to guidelines developed for the human visual system, which may not align perfectly with the needs of AVs. Existing tests, however, usually focus on adversarial objects with conspicuous shapes or patches, which are ultimately unrealistic due to their unnatural appearance and reliance on white-box knowledge. In this work, we introduce a black-box attack on AV perception systems that creates realistic adversarial scenarios (i.e., satisfying road design guidelines) by manipulating the positions of common roadside objects and without resorting to “unnatural” adversarial patches. In particular, we propose TrashFuzz, a fuzzing algorithm that finds scenarios in which the placement of these objects leads to substantial AV misperceptions—such as mistaking a traffic light’s colour—with the overall goal of causing traffic-law violations. To ensure realism, these scenarios must satisfy several rules encoding regulatory guidelines governing the placement of objects on public streets. We implemented and evaluated these attacks on the Apollo autonomous driving system, finding that TrashFuzz induced violations of 15 out of 24 traffic laws.
Test flakiness is a prevalent condition within modern build pipelines. To quantify the flakiness, test engineers consider the rate of change of a test’s outcomes. However, this only shows one dimension of the problem: the quantity of test flakiness at a particular point in time. It does not tell us the severity of the flakiness, nor the outcome diversity, which is useful information for root cause analysis. In this paper, we propose 3 new encodingbased scores to measure the quantity, severity, and outcome diversity involved in a given instance of test flakiness. Our results show that the performance of the flattened score is comparable to that of pre-existing single-dimensional scores, but analysing them together provides more actionable insights, identifying clusters of tests with similar flakiness behaviour.
This paper reports on the ICST 2026 edition of the tool competition on regression testing of self-driving cars (SDCs). The competition aims to foster research on test prioritization for simulation-based SDC testing, a rapidly growing and practically relevant domain, by providing a common platform for the submission and evaluation of testing tools. To facilitate participation, the competition supplies an advanced infrastructure together with representative case studies, enabling participants to develop and assess test generation, selection, and prioritization approaches for SDCs. In its second edition, the competition features four submitted tools, evaluated using regression metrics for test prioritization (in the previous edition, a selection strategy was the target) and compared with baseline approaches, namely a random strategy, and the tools included in the SBFT edition of the same challenge. The paper presents an overview of the competition, including its motivation, framework, evaluation process, participating tools, and key findings.
Flaky tests in web applications frequently arise from nondeterministic interactions between asynchronous events and dynamic DOM updates. Traditional debugging techniques, including fault localization, struggle to diagnose such failures due to the complex, event-driven, and continuously evolving nature of modern web interfaces. This paper introduces DoeFL, a novel approach that combines structured analysis of DOM event sequences with large language model (LLM) reasoning to identify the root causes of web flakiness. DoeFL models temporal causal relationships between DOM events and flaky test failures using Lamport logical clocks. It then uses LLMs to capture semantic inconsistencies and intent mismatches often missed by purely structural analyses. We evaluate DoeFL on 122 real-world flaky test cases collected from 47 web projects. Using only its Lamport-based structural component, DoeFL achieves $47.5 \%$ Top-1 and $82.0 \%$ Top-5 localization accuracy. When augmented with LLM-based reasoning, performance further improves, achieving Top-1 accuracy between $65.6 \%$ and $77.9 \%$ and Top-5 accuracy between 89.3% and 95.1%, depending on the selected LLM.
Recent advances in blackbox test generation (fuzzing) allow combining syntax specifications (grammars) with semantic constraints over grammar elements. These constraints not only help in producing valid inputs (“ $\langle$ checksum $\rangle$ should be a Luhn checksum over $\langle$ number $\rangle$ ”) but also allow specifying testing goals (“ payload should be as short as possible”). Such constraints give testers unprecedented control over the generated inputs. However, constraint-based fuzzers like ISLa or Fandango have only been limited to input features and blackbox fuzzing. This paper shows how to extend this concept with execution constraints relating to the program under test. We introduce a set of primitives that check for specific execution features such as coverage of specific locations, memory usage, execution time, and more. Testers can then use these to specify additional testing goals: “I want an input that is (1) valid, (2) as short as possible, and (3) results in a maximum of code coverage.” We implement our approach on top of the Fandango fuzzer and find that its evolutionary algorithm efficiently finds solutions that satisfy input and execution constraints. We demonstrate that combining input and execution constraints enables us to specify and implement approaches such as directed, performance, and coverage-guided fuzzing, effectively covering the entire range of testing goals from whitebox to blackbox fuzzing in a unified framework.
We present a method for requirement conformance falsification for cyber-physical systems developed in a continuous integration setting, where software is validated repeatedly as new system revisions become available. Falsification is a validation approach that combines adaptive test generation, execution, and monitoring driven by formally specified requirements. Existing falsification methods treat each system revision as an independent validation task and therefore do not leverage information collected in prior tasks. In this article, we formalize the continuous falsification problem and propose the CF-OGAN algorithm with the ability to transfer knowledge across successive system revisions. CF-OGAN reuses neural network weights, adds small adaptation layers for new revisions, and gradually reduces random exploration as the model learns from individual revisions. We evaluate CF-OGAN on 80 validation tasks, comprising 16 correctness requirements over five revisions of different benchmark problems from the ARCH-COMP 2024 competition. We compare CF-OGAN with a non-transfer baseline and random search to quantify gains from transfer and model learning. Overall, the falsification rate increases from 43.2% with random search and 60.7% with the non-transfer baseline to 76.0% with CFOGAN. The mean number of executions is reduced from 647 (random search) and 496 (non-transfer baseline) to 325 with CF-OGAN. These results suggest that cross-revision knowledge transfer can substantially improve both effectiveness and efficiency in continuous requirement falsification pipelines for cyber-physical systems.
Conventionally, unit tests exercise their unit under test on predetermined input to validate its behavior. The advent of property-based testing frameworks has led to unit tests that exercise the unit under test on randomly-generated inputs, for which the unit’s behavior has to satisfy developer-specified invariant properties. The promise of increased coverage and stronger validation may lead developers to evolve existing conventional unit tests into property-based ones. This paper reports on the results of an empirical study of 257 property-based unit tests (PBT) from 64 open-source repositories that have evolved from a conventional unit test. The study examines each PBT-introducing commit for change patterns, and for the type of features of property-based testing frameworks that are adopted. Next, it investigates the impact of PBT adoption by running test cases to compute changes in code coverage and test failures. Finally, the study investigates the evolution of the newly-introduced PBT by comparing its first to its most recent version. The study’s findings include that 37 out of 257 evolutions required no changes to the body of the test. When changes are required, these are most likely to target setup code followed by oracle code. Over half of the studied PBTs use off-the-shelf input generators, yet see an improvement in code coverage by an average of 4 statements. Additionally, for 5 test cases, the PBT found a counterexample where the original unit test passed. Finally, 80% of the evolved test cases are still present in the current version of the project.
Contract assertions, such as preconditions, postconditions, and invariants, play a central role in software verification, test generation, and debugging, but remain difficult to produce manually. Approaches based on dynamic analysis, such as Daikon, can infer such assertions from execution traces, yet their accuracy strongly depends on the quality of the test suites used for dynamic analysis, and on the space of candidate specifications that the technique is able to handle. Extending the language of candidate specifications in expressiveness allows these approaches to infer relevant properties that would otherwise be missed, but at the cost of increasing the number of candidate specifications to consider, and the corresponding cost of dynamic analysis. This amplifies the problem of generating false positives, invalid specifications that the tests used for dynamic analysis fail to identify, as well as the generation of irrelevant properties, properties that hold in general for the software under analysis, but do not constitute useful information. Our research aims at improving the accuracy of dynamic specification inference by incorporating LLMs into the inference process and subsequent refinement stages. A first result of our approach studies LLMs as a source of executable counterexamples for candidate assertions that pass dynamic analysis. Evaluated on 43 Java methods and three state-of-the-art LLMs, the approach discards up to $11.68 \%$ of invalid assertions inferred by SpecFuzzer and improves precision by up to $7 \%$, without reducing recall. Building on this result, our work investigates broader evidence-guided refinement strategies for improving dynamic specification inference, combining counterexample-guided and mutation-based mechanisms to improve precision, recall, and behavioral coverage.
A broken mainline in large-scale, fast-paced monolithic repositories disrupts thousands of developers and greatly affects overall productivity. Merge pipelines can help to protect the main branch from such breakages by validating Pull Requests (PRs) on speculative integration states (i.e., exactly the state resulting from merging the PR). When applied at scale, however, merge pipelines become bottlenecks as PR submission rates outpace CI capacity. Naive optimizations rely heavily on parallelization and, while effective, remain insufficient for fully resolving these bottlenecks. More sophisticated approaches have demonstrated the capability for further improvements; however, they require deep system integration, which can prevent widespread adoption. We present PREDIBATCH, a lightweight approach that uses machine learning to group low-risk PRs into batches, which reduces the required number of speculative integration states. predibatch relies solely on language-agnostic features from PR metadata and CI history, making it applicable to many CI systems. Our simulation-based evaluation on real-world data from BMW’s merge pipeline setup shows that PREDIBATCH significantly reduces average turnaround times by 8.3% and saves 13.3 % of compute resources compared to the current setup.
Software Reliability Engineering provides strategies for monitoring applications and detecting defects even in production environments. The Tricorder methodology addresses this challenge by generating resource-consumption profiles at runtime of a system under test and applying the DAMICORE unsupervised clustering algorithm to detect anomalies between a reference version and a new version. Although Tricorder has been validated in previous studies across multiple application domains, its practical use has been limited to ad hoc Python scripts, which require considerable manual effort to adapt to different scenarios. This paper presents TriAD (Tricorder-based Anomaly Detection), a reliability tool that encapsulates the Tricorder methodology within a web application with persistent storage, developed in accordance with the RefTEST reference architecture. An initial experiment using the JavaSort application demonstrates that TriAD achieves an $85 \%$ anomaly detection rate while correctly classifying $75 \%$ of non-defective runs. In addition, the tool reduces execution time by $7.5 \%$ for monitoring and $40 \%$ for analysis compared to the original script-based implementation. A demo video of the TriAD tool is available at: https://youtu.be/OOGH4Y-2db0.
Automated reproduction of software bugs from natural-language reports remains a challenging problem, particularly for mobile applications where reproduction often requires complex user interface interactions. Recent advances in large language models (LLMs) have sparked interest in leveraging generative models for Android bug reproduction. However, most existing approaches evaluate success primarily based on whether a failure is triggered during execution, providing limited insight into the semantic fidelity of generated reproduction sequences. In this paper, we present a reflective empirical study assessing the capability of an LLM to reproduce Android application bugs directly from natural-language reports using the AndroR2 dataset of 90 manually validated bug reports and ground-truth UIAUTOMATOR scripts. We evaluate two complementary tasks: failure-type classification and executable reproduction script generation. To assess reproduction fidelity, we introduce critical-step coverage, an intent-level metric that measures whether generated scripts capture the essential user actions required to trigger a failure. Our results reveal an important asymmetry: while classification performance varies significantly across defect categories (70% overall accuracy), reproduction scripts achieve high intent coverage (mean $89.5 \%$). These findings suggest that outcome-based success metrics alone are insufficient to characterize automated bug reproduction quality and motivate the need for intent-aware evaluation frameworks for LLM-based testing.
Code obfuscation is widely used by malware to evade analysis and detection. Modern malware often applies multiple obfuscation techniques simultaneously, making it necessary to identify the set of techniques used in a binary. This task can be naturally formulated as a multi-label classification problem. Existing approaches commonly rely on opcode frequency statistics or handcrafted features, but they ignore the sequential context of instructions and therefore have limited ability to capture structural patterns introduced by obfuscation. In this paper, we propose a Transformer-based method for detecting code obfuscation techniques from opcode sequences. We construct a dataset by applying representative O-LLVM obfuscation techniques—Bogus Control Flow (BCF), Control Flow Flattening (FLA), and Instruction Substitution (SUB)—to programs from the GNU Coreutils suite. Opcode sequences are extracted through disassembly, segmented with a sliding-window strategy, and processed by a DistilBERT encoder in a multi-label classification framework. Experimental results show that the proposed approach learns characteristic patterns of different obfuscation techniques and achieves strong performance even in multi-obfuscation settings. These findings suggest that contextual modeling of opcode sequences with Transformer architectures can effectively complement traditional statistical approaches to obfuscation detection.
Generative Large Language Models (LLMs) are increasingly used in non-generative software maintenance tasks, such as fault localization (FL). Success in FL tasks depends on a model’s ability to reason about program semantics that are beyond surface-level syntactic and lexical features. However, widely used LLM benchmarks primarily evaluate code generation, which differs fundamentally from program semantic reasoning. Meanwhile, traditional fault localization benchmarks like Defect4J and BugsInPy are either not scalable or obsolete because their datasets have become part of LLM training data, leading to biased results. This paper presents the first largescale empirical investigation into the robustness of LLMs’ fault localizability. Inspired by mutation testing, we develop an end-to-end evaluation framework that addresses several limitations in current LLM evaluation, e.g., data contamination, scalability, automation, and extensibility. Given real-world seed programs with specifications, we inject unseen faults and ask LLMs to localize them. We filter out underspecified programs, where correct fault localization is inherently ambiguous. For each program an LLM localizes successfully, we apply semantic-preserving mutations (SPMs) and rerun localization to assess the LLM’s robustness and whether the LLM’s reasoning relies on syntactic cues rather than semantics. We evaluate $\mathbf{1 0}$ state-of-the-art LLMs on 750,013 fault-localization tasks sourced from over 1300 Java and Python programs. We observe that SPMs cause an LLM to fail to localize the same fault it correctly localized earlier in 78% of cases, and that LLMs’ reasoning on the code found earlier in the context is noticeably better. These results suggest that LLMs’ codereasoning is tied to code features irrelevant to semantics. We also identify code patterns that are challenging for LLMs to reason about. To the best of our knowledge, no prior work has evaluated the robustness of LLMs’ code reasoning in fault localization at this scale. Overall, our findings motivate fundamental advances in how LLMs represent, interpret, and prioritize code semantics to reason more deeply about program logic.
Recently, ChatGPT has been the most influential Artificial intelligence (AI) tool due to its systematic way of responding to any query that the user prompts. In this paper, we use Large Language Model (LLM) based tool for good seed generation. Our main objective is to perform fuzzing based on the seeds generated by LLM. We developed a SLS-Fuzz (Self Learned Seeder for efficient Fuzzing) tool to perform efficient fuzzing. The SLS component is the main contribution, which incrementally creates good seeds by adapting the LLM features of learning the runs. We compare the results with the Random seed generation model. The performance of these test cases corresponding to the LLM-generated seeds over the randomly generated seeds is evaluated in terms of their bug detection capabilities.
