Fuzzing is a testing technique that generates a large number of inputs to cause program crashes. As software development accelerates and projects scale, the demand for fuzz testing in software assurance has increased. Performing comprehensive fuzz testing on all functions has become increasingly challenging and resource-intensive. Current methods for determining when to stop fuzz testing activities rely on metrics such as function coverage, potential vulnerability function coverage or crash count. However, these metrics fail to account for the scale of the functions under test. For example, function coverage may lead to excessive testing on non-critical functions, while vulnerability function coverage can result in premature termination if the estimated number of vulnerability functions is too low. This paper introduces a novel fuzzing termination tool, F-800, an agent based on function clustering. F-800 first establishes a relationship between function clusters and vulnerability distribution, then refines clusters using functional summaries. It subsequently monitors code coverage of each cluster to determine whether each function cluster has been sufficiently tested. The fuzzing campaign terminates once neither the coverage of function clusters nor the number of crash occurrences within specific clusters continues to increase. Our experiments on eight function libraries demonstrate that F-800 significantly improves testing efficiency, reducing fuzzing time by 1.4–7.2 hours (5-30%) across different configurations while maintaining minimal bug loss (averaging 0.25 bugs), outperforming existing approaches such as potential-vulnerability or function-coverage-based methods.

Diffusion models, as generative models, achieve remarkable results across diverse applications. Despite their excellent performance, diffusion models are vulnerable to backdoor attacks, as recent studies show. Although a few methods attempt to mitigate backdoors, they require adjusting the entire model, incurring high costs and risking degradation in generative performance. To address this, we propose NEUROPATCH, a framework that patches only a minimal subset of backdoor-related erroneous neurons without modifying the neural net work itself, thereby significantly reducing intervention costs and preserving model performance. Specifically, NEUROPATCH first approximates the distribution of clean samples to detect backdoor-infected inputs, then ranks neurons based on the activation deviations induced by these abnormal samples. Finally, it patches the most error-inducing neurons by attaching lightweight corrective controllers to adjust their outputs, without changing the model parameters. We evaluate NEUROPATCH on 168 diffusion models across two datasets and three main stream backdoor attacks. Extensive results demonstrate both the effectiveness and efficiency of our approach: NEUROPATCH achieves, on average, up to 24 & times; faster repair speed compared to state-of-the-art methods, while slightly improving backdoor mitigation performance. Ablation studies further confirm that ranking neurons by their error-inducing scores is a key factor driving the effectiveness of our framework. Our code is available at: https://github.com/Mospic/NeuroPatch.

With rapid development of Large Language Models (LLMs), they are increasingly being adopted as knowledge-based intelligent systems to support decision-making across domains such as education, healthcare, business, and government. Alongside these strides, LLMs have also raised concerns about their security and reliability. One such concern is Input-Conflicting Hallucination (ICH), a type of hallucination that conflicts with user input. Since the data annotation in NLP tasks is expensive and labor-intensive, existing ICH attack methods have adopted Metamorphic Testing (MT) to bypass the oracle problem, i.e. check the ICHs without the ground truths. However, these attacks are all black-box methods utilizing a limited set of Metamorphic Relations (MRs) and can be easily defended against by decoder-only LLMs. In response, we propose HalluTrigger, a simple yet effective grey-box method that constructs six semantic-guided MRs to generate attack cases, and propose a diversity-guided MP prioritization method to promote the trigger of ICHs in target LLMs. We evaluate HalluTrigger on four NLP datasets and three popularly used target LLMs. As a result, the designed MRs achieve higher hallucination trigger rates than existing state-of-the-art baselines, and the diversity-guided MP prioritization can efficiently trigger ICHs with less time.

Reliable pose estimation is critical for intelligent systems, including autonomous vehicles, unmanned aerial vehicles, and virtual reality applications. While visual-inertial odometry (VIO) has made significant advancements in estimating pose, its performance can still be affected by sensor noise, environmental variations, and calibration errors. To evaluate the performance of VIO, existing testing methods rely on real-world datasets or manually degraded data, where sensor measurements or images are artificially modified according to predefined, fixed perturbation patterns. These approaches are costly, require time-consuming annotation, and rely on ad hoc modifications that limit scenario diversity and hinder the exploration of challenging cases. In this paper, we design and implement OdoTest, the automated testing framework for VIO systems. OdoTest equips sensor-specific transformation operators, including IMU perturbations, camera degradations, and sensor calibration errors, that systematically generate various and realistic test scenarios. Moreover, OdoTest adopts an odometry fitness-guided testing strategy to prioritize scenario generation, improving testing efficiency. By leveraging odometry-specific metamorphic relations (MRs), OdoTest can automatically detect odometry errors without requiring manual ground-truth labels for test scenarios. We evaluate OdoTest on twelve state-of-the-art VIO systems to answer three main research questions: 1) the effectiveness of transformation operators in revealing odometry errors; 2) the ability of OdoTest to generate error-triggering scenarios; and 3) whether retraining on these scenarios can improve odometry performance. Results show that OdoTest’s transformation operators effectively expose odometry errors, while its fitness-guided scenario generation efficiently creates challenging test cases. Moreover, retraining with these scenarios significantly enhances both estimation accuracy and system robustness, which demonstrates the usefulness of OdoTest.

Embodied Foundation Models (EFMs) integrate multimodal understanding, future-state reasoning, and executable robot actions. Yet their safety alignment for human-injury prevention remains underexplored, primarily because real-world data of robots harming humans or creating hazardous household situations cannot be safely or ethically collected. To address this challenge, we propose a safety-critical data construction pipeline for human-injury prevention in EFMs.Starting from real DROID observations, our construction pipeline proceeds through scene understanding, hazard-aware image editing, temporal prompt generation, and single-pass rollout synthesis. The temporal prompts specify the expected scene evolution, while Wan2.7 synthesizes realistic robotic rollouts from the edited hazardous states in a single pass. Using this pipeline, we construct ROBOSHACKLES, a 10,000-clip robotic video dataset derived from real DROID observations, spanning two direct-harm and four indirect-harm categories. To ensure dataset quality, we assess task completion and visual quality with automatic metrics, and evaluate six representative EFMs under a refusal-based safety criterion. Results show that all evaluated models produce unsafe actions in the tested safety-critical scenarios, yielding a 100

The rapid advancement of quantum computing has led to the development of various quantum libraries, empowering compilation, simulation, and hardware backend interfaces. However, ensuring the correctness of these libraries remains a fundamental challenge due to the lack of mature testing methodologies. The state-of-the-art tools often rely on domain-specific configurations and expert knowledge, which limits their accessibility and scalability in practice. Furthermore, although these tools demonstrate strong performance, they adopt measurement-based for output validation in testing, which makes them produce false positive reports. To alleviate these limitations, we propose QSPE, a practical approach that follows the differential testing principle and extends the existing approach, SPE, for quantum libraries. QSPE is fully automated, requiring no pre-set configurations or domain expertise, and can effectively generate a large set of diverse program variants that comprehensively explore the quantum compilation space. To mitigate the possible false positive reports, we propose statevector-based validation as an alternative to measurement-based validation. In our experiments, the QSPE approach demonstrates remarkable effectiveness in generating 22,770 program variants across multiple quantum computing platforms. By avoiding α-equivalence at the quantum and classical program wise, QSPE can reduce redundant generation and save more than 90% of execution cost. Finally, the statevector-based validation method assists QSPE to reduce false alarms and effectively detects 708 miscompilations across multiple quantum libraries. Notably, 81 of the discovered bugs have been officially approved and acknowledged by the Qiskit development team, demonstrating the practical impact of our approach.

As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white and grey-box techniques are often inapplicable to closed-source models, while standard "black-box" text metrics fail to capture the unique fragility of code, where syntactic variation does not always imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Unlike prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior, calculating the Von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness (Spearman's correlation up to -0.98), significantly outperforming lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.

Neural networks (NNs) have rapidly advanced, demonstrating exceptional performance across various fields, leading to a surge in open-source NN projects. The complexity and rapid growth of these projects pose significant challenges for maintenance within the open-source community. Given that NN architecture code is the core asset of NN projects, understanding its reuse in the open-source community is essential for effective maintenance, such as reducing redundancy and identifying potential intellectual property violations. While prior studies have examined code reuse in open-source projects, they have two key limitations: They do not specifically address NN structure code, and they rely on manually selected small-scale datasets that do not sufficiently represent the broader open-source ecosystem. To address these limitations, this study explores reuse patterns in a large-scale dataset of 20,000 open-source projects on GitHub, focusing specifically on NN structure code. Specially, we categorize NN structure reuse into three types: (1) exact reuse with no changes; (2) shallow reuse with minor adjustments like renaming variables or adjusting parameters; and (3) conceptual reuse with significant modifications, while retaining the same layer sequence. We then propose a detection framework, NNReuse, to identify these reuse types and conduct an empirical evaluation of their prevalence and characteristics. As a practical application, we also assess potential license conflicts based on NNReuse. Extensive experiments show that 55.6% of projects and 54.17% of NN structures exhibit structural similarities that are consistent with potential NN structure reuse in open-source projects. Among these, exact reuse is particularly common and introduces significant redundancy, with an estimated storage optimization potential of up to 34.49%. Reuse primarily occurs at a high level, with 43.3% involving the reuse of overall network architecture. Additionally, in projects with license protection, as much as 64.3% may present potential license conflicts, highlighting the importance of strengthened license compliance and proactive IP risk mitigation in the open-source community.

Autonomous Driving Systems (ADS), as the core of next-generation intelligent transportation, require rigorous safety verification to achieve large-scale application. However, traditional real-vehicle testing faces bottlenecks such as high costs and limited scenario coverage. For this reason, simulation testing has become the mainstream solution for ADS testing due to its advantages of low cost, high efficiency, and comprehensive scenario coverage. This paper systematically organizes the theoretical methodologies and related tools for simulation-based ADS testing. It first elaborates on the two core architectures of modular and end-to-end architectures, as well as the differences between module-level and system-level testing; then details the three simulation testing levels, and compares and analyzes the characteristics of mainstream simulation platforms; further explores scenario-based testing methods, including search-based testing for identifying edge risks, metamorphic testing for solving the problem of verification without ground truth, and formal methods for ensuring system compliance; finally constructs a multi-dimensional safety evaluation index system covering collision and conflict, trajectory and lane-keeping, comfort and smoothness, and scenario coverage. This paper aims to provide a structured knowledge system for ADS simulation testing in the current field and to offer research directions for subsequent technological iteration and commercial implementation.

In the realm of deep learning, a variety of neuron coverage criteria for Deep Neural Networks (DNNs) have been devised to effectively assess the quality of test suites and facilitate the generation of test inputs. Recently proposed coverage criteria, incorporating representation distribution and causal relationships, have infused fresh vitality into this field. However, the focus of previous works is primarily on Convolutional Neural Networks for computer vision, leading to a research gap in exploring coverage testing for language models. Concurrently, with the rise of large language models, transformer-based language models have become increasingly dominant, and numerous ones have sprouted. Therefore, the effectiveness of coverage criteria in transformer-based language tasks, especially with the introduction of novel criteria, remains an unresolved open problem. To tackle it, this study examines these concerns by evaluating a wide range of criteria, including four well-established ones and two state-of-the-art criteria, across three types of transformer-based models: encoder-only, decoder-only, and encoder–decoder models. Building on previous research, we conduct a comprehensive evaluation across three key areas: regarding test suite properties, (1) Error-revealing capability, i.e., sensitivity to adversarial examples; (2) Diversity, i.e., distribution diversity and sample fairness (category diversity); and regarding test suite generation, (3) Input generation guidance, i.e., the ability to guide the generation of more valuable samples. The experimental results demonstrate that the impact of coverage criteria is multifaceted. For the error-revealing capability of test suites, the additional coverage for erroneous samples over noise samples is only 0.32%. In terms of distribution diversity and sample fairness, 26 and 30 cases, respectively, out of 33 configurations are effectively evaluated. Additionally, incorporating neuron-wise coverage guidance during test suite generation slightly increases the production of adversarial samples by 4.56%. In conclusion, while current coverage criteria can act as an antidote for assessing simple diversity, they remain largely a placebo for the core task of revealing adversarial errors, particularly when relying on individual criterion. Consequently, their practical application requires carefully evaluating the tradeoff between computational overhead and potential benefits given the massive scale of Transformers. However, this low cost-effectiveness ultimately highlights the urgent need to explore and develop more robust and efficient criteria designed specifically for Transformer-based models.

Although Large Language Models (LLMs) excel at code generation, recent research reveals that they exhibit an insufficient grasp of core programming concepts, such as data flow and control flow. This limitation undermines their robustness when encountering variations in these concepts in practice; however, effective solutions that explicitly target this gap remain limited. To address this challenge, we propose ProCURE, a concept-aware consistency learning framework designed to enhance LLMs’ understanding of programming concepts. Specifically, ProCURE first performs automated concept-oriented code augmentation to construct a concept-aligned dataset covering representative programming concepts. It then conducts concept-aware fine-tuning, encouraging the model to capture fine-grained concept variations and learn appropriate generation behaviors under such variations via a novel concept-sensitive consistency loss. To quantify programming concept understanding, we introduce the Concept Consistency Score (CCScore), defined as the proportion of correct generations preserved under concept variations. A higher CCScore indicates a more profound understanding of programming concepts. We evaluate ProCURE on four open-source LLMs across three widely used code generation benchmarks. Experimental results show that ProCURE improves CCScore by an average of 17.9 points, demonstrating its effectiveness in addressing the programming concept understanding gap.

Large language models (LLMs) have recently shown impressive results on diverse code-related tasks, benefiting from large-scale training and instruction tuning. However, studies reveal that their grasp of fundamental programming concepts, such as data flow and control flow, remains shallow, leading to fragile performance when code requires deeper reasoning. This limitation restricts the practical adoption of LLMs in real-world software development. To address this issue, this work introduces a counterfactual code augmentation framework combined with concept-aware tuning, designed to guide LLMs toward stronger conceptual understanding. Comprehensive evaluation across multiple models and benchmarks demonstrates the effectiveness of the proposed approach.

The rapid growth of the decentralized finance (DeFi) ecosystem has given rise to flash loan, a type of uncollateralized loan service that enables users to easily borrow substantial amounts of funds. However, this has prompted attackers to conduct malicious arbitrage within DeFi protocols, known as notorious flash loan attacks, resulting in significant asset losses. Existing works primarily focus on investigating price oracle manipulation, a common tactic in flash loan attacks, but lack a comprehensive understanding regarding the entire process of flash loan attacks and the diverse range of attack methods. In this paper, we empirically study 155 real-world flash loan attack incidents, representing the largest-scale study to date. We first categorize these incidents into five types based on their root causes and compile statistics on their distribution, then elucidate the vulnerable code and finance mechanisms exploited in each category. Subsequently, we identify the symptoms of codebased vulnerabilities and summarize the abstract attack models for the entire process. Finally, we evaluate the effectiveness of state-of-the-art off-chain tools in detecting code-based vulnerabilities within their scope of capabilities. We find that Slither performs the best in detecting 22 % of temporal reentrancy vulnerabilities, and DeFiTainter has a 52% false negative rate in detecting price oracle manipulation, mainly attributed to three limitations.

Autonomous Driving Systems (ADS) are considered safety-critical, as even a minor fault may lead to catastrophic consequences. To evaluate their reliability and robustness under failure conditions, Fault Injection (FI) techniques have been widely adopted. Most existing FI methods employ data-driven approaches, such as surrogate modeling and reinforcement learning, to generate test cases. While these techniques have shown promise, they often incur substantial costs in terms of data collection and training time. Moreover, their performance is highly sensitive to the quality and quantity of training data, which can limit their applicability in diverse or unseen scenarios. In this paper, we propose LOFT, an efficient multi-objective search-based FI testing framework that leverages Large Language Models (LLMs) to identify diverse and realistic critical faults. To accommodate the structured and non-linguistic nature of raw simulation data, LOFT adopts a two-stage LLM-based fault injection pipeline. In the first stage, an LLM converts singleframe simulation data into natural language descriptions and suggests appropriate fault types. In the second stage, a separate LLM examines the broader scenario context to determine the optimal time window for fault injection. The outputs from the two LLMs are then used to initialize and guide a multi-objective search procedure aiming at discovering a diverse set of critical faults. We implement LOFT and evaluate on an ADS provided by our industrial partner. Experimental results show that, compared with two baseline approaches, LOFT detects over $90 \%$ more critical faults and identifies an average of 2.2 additional fault types within an equivalent number of simulations.

Fuzzing a library requires experts to understand the library usage well and craft high-quality fuzz drivers, which is tricky and tedious. Therefore, many techniques have been proposed to automatically generate fuzz drivers. However, they fail to generate rational fuzz drivers due to the lack of adherence to proper library usage conventions, such as ensuring a resource is closed after being opened. To make things worse, existing library fuzzing techniques unconditionally execute each driver, resulting in numerous irrational drivers that waste computational resources while contributing little coverage and generating false positive bug reports. To tackle these challenges, we propose a novel automatic library fuzzing technique, Scheduzz, an LLM-based library fuzzing technique. It leverages LLMs to understand rational usage of libraries and extract API combination constraints. To optimize computational resource utilization, a dual scheduling framework is implemented to efficiently manage API combinations and fuzz drivers. The framework models driver generation and the corresponding fuzzing campaign as an online optimization problem. Within the scheduling loop, multiple API combinations are selected to generate fuzz drivers, while simultaneously, various optimized fuzz drivers are scheduled for execution or suspension. We implemented Scheduzz and evaluated it in 33 real-world libraries. Compared to baseline approaches, Scheduzz significantly reduces computational overhead and outperforms UTopia on 16 out of 21 libraries. It achieves 1.62x, 1.50x, and 1.89x higher overall coverage than the state-of-the-art techniques CKGFuzzer, Promptfuzz, and the handcrafted project OSS-Fuzz, respectively. In addition, Scheduzz discovered 33 previously unknown bugs in these well-tested libraries, 3 of which have been assigned CVEs.

Network protocol is of paramount importance as it serves as the bridge for secure communication within and between networks, and protocol security is vital for ensuring the integrity, confidentiality, and availability of data transmission. Nonetheless, protocol testing is a challenging task. we propose LLMFuzz, a novel LLM-based network protocol testing approach, and extend ProFuzzBench by integrating three advanced fuzzers, AFLNet Legion, ChatAFL, and LLMFuzz-along with three protocols (HTTP, MQTT, and Modbus). Experimental results demonstrate that our newly proposed and incorporated LLMFuzz outperforms the baseline fuzzer AFLNet and state-of-the-art fuzzer ChatAFL across multiple network protocols and evaluation metrics.

As a relatively new programming language, Rust has gained significant popularity in recent years due to its safety features during compilation. However, Rust developers often face challenges stemming from its strict compilation checks due to the steep learning curve of safety rules. To make matters worse, the lack of training data and the unique semantics of Rust lead to poor performance in learning-based automated program repair techniques. To address these challenges, we propose a novel error injection approach to generate a balanced training dataset and leverage the Mid-level Intermediate Representation (MIR) as enhanced features for Rust’s unique compilation error repair. Using these innovations, we fine-tuned a new code model, LLaRRA: L arge L anguage a nd R ust R epair A ssistant. Experimental results demonstrate that LLaRRA significantly outperforms state-of-the-art models in terms of Pass@K and Acc@K.

Relational DBMS implementations are expected to adhere to SQL standards. However, there are currently no tools available that can automatically verify this conformance. The main reasons are twofold. First, the SQL standard specification, documented in natural language, tends to be ambiguous and is not directly executable. Second, it is difficult to generate test queries that thoroughly cover all aspects, e.g., keywords and parameters, defined in the SQL specification. In this work, we introduce the first method for semantic conformance testing of RDBMSs. Our contributions are threefold. Firstly, we formally define the denotational semantics of SQL and implement them in Prolog, creating an executable reference RDBMS for differential testing against existing RDBMSs. Secondly, we propose three coverage criteria based on these formal semantics, along with a coverage-guided query generation algorithm that effectively generates queries achieving high semantic coverage. Lastly, we apply our approach to six widely-used and thoroughly tested RDBMSs, e.g., MySQL, PostgreSQL and OceanBase, uncovering 19 bugs and 13 inconsistencies, all of which are confirmed by RDBMS developers.

Neural networks have become integral to many fields due to their exceptional performance. The open-source community has witnessed a rapid influx of neural network (NN) repositories with fast-paced iterations, making it crucial for practitioners to analyze their evolution to guide development and stay ahead of trends. While extensive research has explored traditional software evolution using Software Bill of Materials (SBOMs), these are ill-suited for NN software, which relies on pre-defined modules and pre-trained models (PTMs) with distinct component structures and reuse patterns. Conceptual AI Bills of Materials (AIBOMs) also lack practical implementations for large-scale evolutionary analysis. To fill this gap, we introduce the Neural Network Bill of Material (NNBOM), a comprehensive dataset construct tailored for NN software. We create a large-scale NNBOM database from 55,997 curated PyTorch GitHub repositories, cataloging their TPLs, PTMs, and modules. Leveraging this database, we conduct a comprehensive empirical study of neural network software evolution across software scale, component reuse, and inter-domain dependency, providing maintainers and developers with a holistic view of its long-term trends. Building on these findings, we develop two prototype applications, Multi repository Evolution Analyzer and Single repository Component Assessor and Recommender, to demonstrate the practical value of our analysis.

Fuzzing is a testing technique that generates a large number of inputs to cause program crashes. As large language models grow, so do the programs developed with their assistance, leading to an exponential increase in code complexity and function counts. Performing comprehensive fuzz testing on all functions has become increasingly challenging and resource-intensive. Current methods for determining when to stop fuzz testing activities rely on metrics such as function coverage, vulnerability function coverage or crash count. However, these metrics fail to account for the scale of the functions under test. For example, function coverage may lead to excessive testing on non-critical functions, while vulnerability function coverage can result in premature termination if the estimated number of vulnerability functions is too low. This paper introduces a novel fuzzing testing termination criterion based on function clustering. We compare our criterion with three existing methods.Fisrt, by leveraging langurage model for function encoding and a multi-metric fusion algorithm for determining the number of clusters, we establish a relationship between function clustering and vulnerability distribution. Second, our experiments on eight function libraries demonstrate that the proposed termination criterion significantly improves testing efficiency, reducing fuzzing time by 1.4–7.2 hours (5–30%) across different configurations while maintaining minimal bug loss (averaging 0.25 bugs), outperforming existing criteria like vulnerability function coverage-based approaches.