Debugging is a central yet cognitively demanding part of software development, requiring problem-solving skills, expertise, and appropriate tools. Prior work has documented a variety of debugging strategies—such as hypothesis testing, simplification, and forward or backward reasoning—as well as individual factors that influence their use. However, we still lack an integrated understanding of how expert developers select and adapt these strategies in response to changing contextual factors during real-world challenging debugging. In this paper, we conducted a three-phase study: we first synthesized prior work on challenging debugging contexts, then used a short survey of 35 web developers to surface up-to-date examples of difficult defects, and finally conducted semi-structured interviews with 16 expert web developers to understand how they select and adapt strategies in these contexts. We identified a taxonomy of static and dynamic contextual factors that shape strategy selection. Dynamic factors can evolve during debugging and prompt strategy transitions, whereas static factors provide relatively stable constraints on developers’ choices. We also present a descriptive state-transition model showing how experts adapt strategies as contextual conditions like clarity, reproducibility, and constraints evolve during a debugging scenario. Our findings highlight the need for more contextual design of debugging tools as well as educational decision-making frameworks for choosing effective strategies.
Flaky tests produce non-deterministic outcomes, passing or failing inconsistently when executed under conditions that developers intend to maintain constant. Within continuous integration (CI) environments, these tests disrupt development workflows by causing pipeline failures unrelated to intended changes. Modern CI systems commonly rerun failed tests within the same job to identify flaky failures. Existing research predominantly analyzes data from this detection mechanism, overlooking undetected flaky failures that cause the most severe disruptions. We analyzed 8.8 billion test executions from four industry-scale projects over two-month periods. All apparently deterministic failures were systematically validated through reruns in fresh environments within the projects’ original CI infrastructure to identify undetected flaky failures. We found that, across all projects, undetected flaky failures account for 9.8%–16.3% of failed pipeline runs. These flaky failure rates spike temporarily, triggered primarily by code changes and test reordering. While failures concentrate heavily in small test case subsets, the most disruptive test cases with undetected flaky failures remain largely invisible to tools relying on current detection mechanisms. Additionally, test environments significantly influence flakiness, with up to 3× variation in flake rates between them. These findings reveal fundamental limitations in current detection strategies and highlight the need for comprehensive approaches to flaky test management. Our publicly available dataset of 154,000 flaky test cases with 1.7 million flaky failures provides an empirical foundation for developing improved strategies for industrial CI environments.
User feedback prioritization provides crucial guidance for software engineering teams to formulate effective evolutionary strategies and enhance maintenance efficiency. However, existing approaches typically treat extracted feedback information as scattered and isolated features, making it difficult for models to bridge the semantic gap between informal user expressions and software functionalities, thereby failing to accurately assess the functional criticality of the feedback. Furthermore, these static methods struggle to capture the dynamic interdependencies among continuous feedback streams—whether newly emerging issues conflict with current update plans, ultimately leading to inaccurate prioritization results. To address these challenges, this study proposes UWC-UFP, a novel framework that introduces the Knowledge Tracing (KT) architecture into the user feedback prioritization task for the first time. The framework constructs a Software Function Concept Tree (SFCT) to capture hierarchical associations between functional modules and employs an enhanced Subtree Autoencoder (STAE) to learn semantically disentangled and structure-aware software function concept representations. Furthermore, a User Feedback State Tracing (UFST) model dynamically updates feedback states to simulate interactive influences among feedback items, enabling the continuous correction of feedback priorities through iterative comparison. Experiments on five real-world datasets demonstrate that UWC-UFP achieves consistent improvements over baseline methods in both ranking accuracy and robustness, especially under imbalanced data scenarios. The model achieves an average F1-score improvement of 0.35 and exhibits strong interpretability by revealing user feedback influence patterns across software functions. These results highlight the potential of integrating Knowledge Tracing and software function concept learning for adaptive, interpretable, and robust user feedback prioritization in software maintenance.
Publish/subscribe protocols are widely adopted in IoT and distributed systems. Brokers mediate message delivery among multiple participants, acting as convergence points for complex multi-party dependencies and protocol states. As a result, flaws in broker implementations can compromise the security and stability of entire systems. Although fuzzing has proven effective for bug discovery, most existing fuzzers adopt a two-party model, which limits their ability to explore multi-party communication logic. Moreover, existing fuzzers typically focus on either memory corruption or logic errors without considering whether a broker implementation is specification-compliant. In this paper, we present PSFuzzer, a multi-party black-box fuzzer designed for pub/sub protocols. PSFuzzer employs a multiparty fuzzing framework with two concurrent senders to systematically explore multi-party interactions. We propose a unified intermediate representation for multi-party message transmission and introduce an LLM-driven dependency extraction approach to automatically extract multi-party dependency rules for various protocols. Guided by these rules, PSFuzzer coordinates concurrent message sending and combines random exploration with a message-priority scheduling strategy to improve fuzzing efficiency. To detect protocol non-compliance bugs, PSFuzzer leverages differential testing and an LLM-based analyzer to automatically validate and localize specification violations. We implemented PSFuzzer and evaluated it on 14 brokers across three categories of pub/sub protocols. PSFuzzer discovered 157 bugs, including 29 memory bugs and 128 non-compliance bugs, with 26 CVEs assigned. The comparison with state-of-the-art fuzzers indicates that PSFuzzer outperforms them in both code coverage and bug finding capabilities.
Modern software systems are increasingly complex, and static analysis tools are commonly used to identify potentially vulnerable code by issuing warnings. However, these warnings often require manual inspection to confirm whether the reported issues are real, making the process time-consuming and error-prone. Directed fuzzing has emerged as a powerful automated technique to validate the warnings. However, applying it to the entire project in response to each warning is computationally infeasible, often requiring days of execution to achieve only incremental improvements in code coverage. We present SNIPTEST, an execution-based warning triage framework that generates and fuzzes compiled code slices centered around static-analysis warnings. Rather than proving exploitability in the full program, SNIPTEST provides evidence about how a warning behaves under progressively expanded sliced execution contexts. It employs a layer-by-layer slicing strategy, incrementally expanding context around the target location to validate potential vulnerabilities with increasing precision. We evaluate SNIPTEST on a benchmark of 97 true vulnerabilities and 97 false alarms across three real-world projects. SNIPTEST produces Possible True Positive evidence for 53 of 97 confirmed vulnerabilities (54.6
The proliferation of open-source software (OSS) has made software supply chains prime targets for attacks like Package Confusion, where adversaries publish malicious packages with names deceptively similar to legitimate ones. To protect against such attacks and safeguard the use of OSS, multiple confusion detection methods have been proposed. However, existing methods are limited to single-signal retrieval strategies (relying solely on lexical or semantic metrics), struggle with high False Positive Rates (FPR), and are vulnerable to adversarial evasion. Critically, as content-agnostic approaches, they fundamentally fail to distinguish benign packages with high naming similarity from malicious, code-dissimilar impersonations, leading to persistent high FPR. To address these limitations, we introduce ConfuScope, a novel multi-agents framework for package confusion detection. Specifically, it first discovers potential confusion targets using fine-tuned word embedding models with hybrid similarity search. After that, it subsequently evaluates risk via a fused machine learning model that uniquely combines: (1) a multi-dimensional metadata group and (2) a novel package content analysis group, to reduce the FPR and mitigate the impact of adversarial evasion. To assess the effectiveness of ConfuScope, we evaluate it on challenging ConfuDB and NeupaneDB datasets. Our results demonstrate that ConfuScope significantly outperforms state-of-the-art baselines, ConfuGuard and Typomind, improving the weighted F1-score by 11%–75% while reducing the FPR by 17%–22%, and effectively discovering the legitimate target packages.
In collaborative software development, high-quality commit messages facilitate developers to quickly understand the content and purpose of code changes (i.e., the two important parts of high-quality commit messages: “what” and “why”) in the form of natural language. Nonetheless, most commit messages lack important information because writing high-quality commit messages is quite time-consuming and laborious. To alleviate the above problems, researchers have tried to explore and promote automatic commit message generation (CMG) techniques. However, we observe that existing CMG techniques still face several challenges. Firstly, according to the widely accepted definition of high-quality commit messages, the datasets used by current methods contain a considerable amount of low-quality data, which limits the effectiveness of model learning and hinders the generation of high-quality commit messages. In addition, deploying large models can be difficult in low-resource scenarios or under computational constraints. However, the performance of small-scale models for CMG has not been fully exploited. To address these issues, we employ cuttingedge large language models (LLMs) to automatically identify high-quality commit messages, building a robust dataset for training. On this basis, we propose a novel training framework IEEF (Information Extraction, Enhancement, Fine-tuning) that effectively improves the quality of generated commit messages by enhancing the model’s ability to understand and reason about two critical pieces of information mentioned above. To verify the effectiveness of IEEF, we conduct a comprehensive experimental evaluation using multiple advanced CMG models. The results reveal that our framework consistently improves the quality of generated commit messages across nine models, especially for small-scale models. Meanwhile, this framework effectively narrows the performance gap caused by model size differences, offering a viable solution for utilizing small-scale models in resource-constrained environments.
Termination analysis is crucial for identifying programs that may run indefinitely, thereby preventing issues such as denial-of-service conditions. Despite its undecidable nature, both formal methods and recent machine learning approaches have been developed to address this challenge. However, existing machine learning methods face limitations: they either rely on expensive execution traces or utilize static representations that inadequately capture non-termination patterns in loop structures. To overcome these limitations, we propose SAFTerm, a Structure-aware Framework for Termination analysis that integrates structural and semantic program information. Our approach introduces a novel program representation named CPG-loop, which augments Abstract Syntax Trees with Control Flow Graph information specific to loop structures. SAFTerm employs a dual-module architecture featuring a semantic-aware module using Graph Neural Networks to extract semantic features, and a loop-aware module using Convolutional Neural Networks to learn non-termination behaviors from loop patterns. To evaluate the effectiveness of SAFTerm, we conducted experiments on both verification competition (SV-COMP and TermComp) and LLM-generated (HumanEval and MBPP) benchmarks, covering diverse programs, including those involving arrays, bitwise operations and pointers. Compared with state-of-the-art neural network-based methods, our framework achieves average performance gains of up to 22.68% in mAP and 17.89% in AUC. Notably, SAFTerm effectively handles challenging instances that remain unsolvable by formal verification tools, achieving 86.74% in mAP and 71.93% in AUC on these difficult instances.
Prompt engineering has emerged as a key practice to guide Large Language Models (LLMs) in code generation tasks, under the assumption that better prompts yield better outputs. Yet little is known about which characteristics of prompts actually drive variations in output quality. In this paper, we take a first step toward addressing this gap by investigating how measurable properties of prompts relate to the quality of generated code. We conducted an empirical study of real-world, single-turn interactions between developers and ChatGPT collected from GITHUB. Prompts were characterized along readability and structural dimensions, while output quality was measured as the conceptual consistency to committed code, perceived usefulness, and correctness. Regression analysis shows that readability metrics are strong predictors of usefulness and correctness, whereas structural features such as the number of sentences and task type significantly affect conceptual consistency. Overall, our findings provide evidence that prompt quality is multi-dimensional and that measurable readability-related and structural prompt properties can serve as indicators of the correctness, usefulness, and conceptual consistency of LLM-generated code in software engineering.
Large language model (LLM) agents have emerged as a transformative software paradigm, enabling autonomous task execution through natural language reasoning, multi-agent collaboration, and tool integration. However, their non-deterministic decision-making and complex interaction patterns introduce unprecedented challenges for failure diagnosis and system improvement. Unlike traditional software, where failures often manifest as code defects, agent failures are embedded in lengthy, language-heavy execution trajectories that include natural language reasoning, obscuring root causes and complicating repair. This survey provides a systematic review of the burgeoning field of trajectory analysis for failure attribution and system enhancement in LLM agents. We collect 55 papers published between early 2025 and April 2026, spanning software engineering, artificial intelligence, and human-computer interaction. We organize the literature along five key dimensions: failure taxonomy, failure attribution, system enhancement and optimization, trajectory monitoring and analysis tools, as well as datasets and benchmarks. Our analysis reveals that while the field has progressed rapidly from simple LLM prompting to causal inference, fine-tuned tracer models, and dynamic intervention, the step-level attribution accuracy remains limited, and benchmark diversity is still a bottleneck. We identify four complementary taxonomic perspectives, four methodological paradigms for attribution, and three families of enhancement strategies. We also critically assess existing benchmarks, highlighting a recent shift toward full observability and unrecoverable failure annotation. This survey serves as both a structured overview for current research and a strategic guide for advancing robust, efficient, and self-improving agent ecosystems. We further call upon the software engineering community to engage more deeply with this emerging class of software, bringing expertise in fault localization, program repair, and empirical methods to bear on the unique challenges posed by LLM agents.
Code summarization supports program comprehension and defect detection, and developers increasingly use both human-centered and AI-based interventions when performing this task. We present a controlled human-subjects study (N = 47) examining the effects of physical exercise and AI-generated code summaries with varying correctness on developers’ summarization and bug detection performance. Participants summarized GitHub code under different intervention conditions, with outcomes evaluated along multiple dimensions, including accuracy, completeness, conciseness, readability, defect detection, and response time. We analyzed the data using mixed-effects models to account for repeated measures across participants and code artifacts. Surprisingly, under the exercise intervention studied, we did not observe consistent benefits for summarization quality. AI assistance is generally useful for code summarization, but incorrect AI assistance substantially degrades bug detection performance. These findings provide empirical evidence on the nuanced benefits and risks of human- and tool-based interventions in code summarization and bug detection.
Test automation has made significant progress over the past few decades, yet achieving high test coverage remains a long-standing challenge. To overcome the limitations of existing methods in reaching certain complex or hard-to-trigger code branches, directed input generation (DIG) has received growing attention. With the rise of Large Language Models (LLMs), recent studies have explored their potential and demonstrated their effectiveness for this task. However, these efforts have largely focused on standalone functions. Whether LLMs can effectively generate directed inputs for real-world, repository-level code remains an open question. To bridge this gap, we construct the first benchmark for repository-level DIG named RepoDIGBench and conduct a comprehensive evaluation of existing techniques on it. Our results reveal a significant performance drop: the state-of-the-art technique succeeds on only 26.4% of RepoDIGBench samples, compared to its 74.4% success rate on the standalone function dataset. Our analysis of the failure cases reveals its two core limitations, i.e., limited reasoning ability and lack of contextual awareness, in complex repository-level scenarios. To address these limitations, we investigate a set of targeted strategies and develop an enhanced LLM-based approach, named RepoDIG. RepoDIG improves the state-of-the-art DIG technique in pass@5 by 2.2× using the same base LLM, and by 3.6× with a stronger model. Furthermore, we investigate the effectiveness of RepoDIG on downstream tasks by using it to enhance the code coverage of the state-of-the-art test generator CoverUp. By targeting the lines missed by CoverUp, RepoDIG raises the line coverage in target functions from 80.0% to 87.6% and branch coverage from 70.9% to 81.5%.
Microservice systems are widely adopted in modern cloud-native architectures due to their flexible modular design. However, complex inter-service dependencies and cascading fault propagation make root cause analysis (RCA) highly challenging. Although recent automated RCA methods have improved fault diagnosis from different perspectives, existing approaches still face difficulty in jointly achieving temporal causal validity, structural plausibility, and propagation-level interpretability. To address this limitation, we propose TCRCA, a topology-guided multi-modal causal inference framework designed for interpretable root cause localization in microservices. TCRCA constructs a unified multi-modal service graph that integrates logs, metrics, and traces, enabling more efficient fault propagation modeling. It introduces a multi-scale Granger causal mechanism, combined with service topology priors, to enhance causal reasoning and fault analysis. Experimental evaluations on three public benchmarks show that TCRCA significantly outperforms state-of-the-art methods, improving Top-1 root cause localization accuracy by 17.2%, reducing the number of false alerts by 69.6%, and accurately reconstructing over 92% of fault propagation paths. These results demonstrate the practical advantages of TCRCA in real-world applications.
Recent statements about the impressive code capabilities of large language models (LLMs) are usually supported by evaluating on open-access benchmarks. Considering the vast size and wide-ranging sources of LLMs’ training code data, it could explicitly or implicitly include code data of test set, leading to LLMs being more susceptible to data contamination on code generation benchmarks. However, due to the opacity of training code data, massive code reuse, the black-box access of models, and the rapid growth of synthetic code data, detecting and mitigating data contamination for LLMs on code generation benchmarks faces significant challenges. In this paper, we propose CDD, which stands for Contamination Detection via output Distribution for LLMs in code generation. CDD necessitates only the sampled code texts to detect data contamination, by identifying the peakedness of LLM’s output distribution. To mitigate the impact of data contamination in code evaluation, we present TED: Trustworthy Evaluation via output Distribution, based on the correction of LLM’s output distribution. To facilitate this study, we introduce two benchmarks, i.e., DETCON and COMIEVAL, for data contamination detection and contamination mitigation evaluation in code-related tasks. Extensive experimental results show that CDD achieves the average relative improvements of 21.8%-30.2% over other contamination detection approaches in terms of Accuracy, F1 Score, and AUC metrics, and can effectively detect implicit contamination. TED substantially mitigates performance improvements up to 66.9% attributed to data contamination across various contamination setups. In real-world applications, we reveal that certain LLMs exhibit a high potential to suffer from data contamination on HumanEval benchmark. Moreover, we also introduce a new evaluation metric MGI, Memorization Generalization Index, to assess the generalizability of LLM’s evaluation results on the benchmark. MGI is applied to wellknown open-source code LLMs, offering a novel dimension for evaluating model performance on the code generation benchmark.
Software engineering relies on abundant observational data, but empirical studies without controlled experiments are often limited to finding statistical correlations instead of detecting deeper, more reliable causal relationships. This limitation hinders our understanding of cause-and-effect dynamics in software development. This paper explores how retrospective cohort studies, a well-established method in other (scientific and empirical) disciplines, can be adapted to strengthen causal inference in mining software repositories (MSR) research. We systematically examine how retrospective cohort studies (RCS) are conducted in fields like epidemiology and extract key methods used to conduct RCS, which we transfer to the MSR context. To demonstrate the potential of this approach, we provide concrete examples of application to a real-world MSR study. Our findings suggest that RCS can enhance empirical software engineering research by enabling more rigorous causal reasoning in order to bridge the gap between correlation and causation in observational software data analysis for MSR studies.
Synthetic vulnerabilities have gained increasing attention in recent years for addressing the class imbalance issue in deep vulnerability detector training datasets, thereby boosting detector training. Despite their promise, existing work generally employs all available synthetic vulnerabilities indiscriminately, leaving a fundamental question under-explored: are all synthetic vulnerabilities beneficial for detector training? To bridge this gap, we empirically demonstrate a non-monotonic relationship between detector performance and the quantity of synthetic vulnerabilities, which we attribute to the limited diversity of synthetic samples, both internally and relative to the original training data. Motivated by these observations, we propose DIVUSE, a novel method for selecting synthetic vulnerabilities to better train deep vulnerability detectors. Specifically, DIVUSE evaluates synthetic vulnerabilities using intra-dataset and interdataset similarity to select a maximally diverse subset for detector training. Our extensive experiments on four deep vulnerability detectors and three synthetic vulnerability datasets demonstrate that DIVUSE outperforms seven baseline approaches across various settings.
Code generation has largely improved development efficiency in the era of large language models (LLMs). With the ability to follow instructions, current LLMs can be prompted to generate and optimize code solutions given detailed descriptions in natural language. Many research efforts are devoted to improving the correctness of LLM-generated code by proposing different benchmarks. Despite the focus on correctness, the time efficiency of LLM-generated code solutions is underexplored. Current correctness benchmarks are not suitable for time efficiency evaluation since their test cases cannot well distinguish the time efficiency of different code solutions. Besides, the current execution time measurement is not stable and comprehensive, threatening the validity of the time efficiency evaluation. To address the challenges in the time efficiency evaluation, we propose COFFE, a benchmark for evaluating the time efficiency of LLM-generated code solutions in code generation and code optimization. COFFE contains 398 and 358 problems for functionlevel and file-level code generation, respectively. To improve the distinguishability, we design a novel stressful test case generation approach with contracts and two new formats of test cases to improve the accuracy of generation. To improve the robustness, we also design a test case input scale mutation method to generate test cases with different input scales. For the time evaluation metric, we propose efficienct@k based on CPU instruction count to ensure a stable and solid comparison between different solutions. We evaluate 19 popular LLMs and 7 LLM-based code optimization methods on COFFE and identify 10 findings. Based on the findings, we draw some implications for LLM researchers and software practitioners to facilitate future research and usage of LLMs in code generation and optimization.
The rapid evolution of blockchain technology has attracted global users to operate nodes and participate in consensus processes. However, heterogeneous networks and hardware conditions may induce chaotic behavior in nodes, leading to critical faults such as packet loss or data corruption. These faults compromise system robustness, causing node crashes or desynchronization. Existing chaos engineering methods for blockchain systems suffer from inefficient fault injection and lack feedback mechanisms to iteratively guide fault selection. To address these limitations, we propose ChaosChain, a framework for efficient and effective chaos engineering in blockchain systems to expose robustness vulnerabilities. For efficiency, we design a parallel fault coordination mechanism that injects multiple fault types into arbitrary nodes simultaneously while enforcing blockchain consensus security constraints.We construct a global timeline of all executed faults and validate security constraints before each injection. To improve effectiveness, we introduce next fault guidance, a feedback-driven approach that selects optimal faults to trigger complex node state transitions.We model each node’s state as an interference-resistant state machine that captures state transitions during faults without cross-node interference. We evaluate ChaosChain across four implementations in Filecoin and Ethereum, uncovering six new robustness issues. Compared to state-of-the-art tools, our framework injects 4× more faults concurrently.
Android provides a wide range of system settings that allow users to control the runtime behaviors of apps, such as screen rotation and UI display. However, setting-related bugs occur when developers do not fully align their apps with the extensive range of system settings that users can define. These bugs can commonly affect apps’ UI, causing setting-related UI display (SUD) bugs that negatively impact user experience. While existing research has explored automated detection of SUD bugs, these approaches often suffer from false negatives. This limitation stems from an incomplete understanding of how app components should adapt UI elements to diverse system settings. To address this gap, we conducted an empirical study to identify common patterns of unexpected setting adaptations that result in SUD bugs. These patterns then served as the test oracle for our proposed automated tool, SUDFinder. To ensure the test coverage, SUDFinder injects a test activity to visually render the XML configuration files of each UI page. We evaluated SUDFinder on 29 popular, open-source apps on F-Droid and found that it effectively identifies 98 previously unknown SUD bugs, achieving a precision of 0.76. So far, 67 have been confirmed and 37 have been fixed by the app developers.
