Unit testing is crucial for detecting bugs in individual program units but consumes time and effort. Recently, large language models (LLMs) have demonstrated remarkable capabilities in generating unit test cases. However, several problems limit their ability to generate high-quality unit test cases: (1) compilation and runtime errors caused by the hallucination of LLMs; (2) lack of testing and coverage feedback information restricting the increase of code coverage; and (3) the repetitive suppression problem causing invalid LLM-based repair and generation attempts. To address these limitations, we propose TestART , a novel unit test generation method. TestART improves LLM-based unit testing via co-evolution of automated generation and repair iteration, representing a significant advancement in automated unit test generation. TestART leverages the template-based repair strategy to effectively fix bugs in LLM-generated test cases for the first time. Meanwhile, TestART extracts coverage information from successful test cases and uses it as coverage-guided testing feedback. It also incorporates positive prompt injection to prevent repetition suppression, thereby enhancing the sufficiency of the final test case. This synergy between generation and repair elevates the correctness and sufficiency of the produced test cases significantly beyond previous methods. Through comparative experiments, TestART demonstrates an 18% improvement in pass rate and a 20% enhancement in coverage across three types of datasets compared to baselines. Additionally, it achieves better coverage rates than EvoSuite with only half the number of test cases. These results demonstrate TestART's superior ability to produce high-quality unit test cases by harnessing the power of LLMs while overcoming their inherent flaws.
Understanding the semantics of program code is a fundamental problem in software engineering, posing continuous challenges for testing and analysis, as well as for AI agents. We introduce neural program modeling : an approach that learns the relationship between input features and execution features (such as traces or coverage) from pairs of generated system inputs and monitored execution data. The resulting program-specific model can predict system inputs that trigger desired program behaviors—in seconds, without executing the program. It can also predict program traces and coverage for a given input, again without running the program. Once trained, such a model acts as a proxy for the program under test, supporting a range of tasks: (1) Predicting inputs for specific execution features, guiding test generators and developers; (2) Replacing the original program during search-based testing or fuzzing, reducing costly or risky executions; (3) Predicting which program parts a test input will execute, guiding test selection ; (4) Predicting which input will trigger a specific behavior, serving as an input filter to prevent undesired outcomes. To our knowledge, this is the first work that trains reversible, program-specific neural machine translation models to both predict execution behavior from inputs and synthesize inputs to achieve the requested behavior in real-world programs. In our evaluation of real-world programs, including Bottle and the Google re2 library, we show that neural program modeling can predict execution features for unseen inputs, achieving similarity ratios ranging from 91.57% to 98.59%. We can also predict program inputs that trigger specified behaviors with accuracy and recall up to 100% after validation and refinement. All predictions can be automatically validated and refined, either by retraining the model on new input-trace pairs or by using a backpropagation loop to adjust its parameters. Model training and inference are resource-efficient and can be performed on commodity hardware.
Neural network repair aims to fix the ‘bugs’ of neural networks by modifying the model's architecture or parameters. However, due to the data-driven nature of neural networks, it is difficult to explain the relationship between internal neurons and erroneous behaviors, making further repair challenging. While several work exists to identify responsible neurons based on gradient or causality analysis, their effectiveness heavily rely on the quality of available ‘bugged’ data and multiple heuristics. Consequently, achieving precise localization and targeted repair remains a significant challenge, particularly in data-scarce scenarios. In this work, we address the issue utilizing the power of formal verification. Specifically, we propose VeRe , a verification-guided repair framework that leverages linear relaxation to precisely and efficiently estimate the repair significance of neurons. Based on the linear bounds, VeRe synthesizes ideal intervals that provide sound guarantees for correct behaviors, thereby facilitating surgical and targeted adjustments of neuron parameters to rectify faulty behaviors. We evaluated VeRe on various repair tasks, and results show that VeRe consistently achieves superior repair success rates and high generalization with negligible impact on original performance. Notably, VeRe exhibits a significantly lower dependency on repair data than existing baselines, effectively removing faulty behaviors even in data-restricted scenarios.
Neural code models are increasingly embedded in software development workflows, but their susceptibility to backdoor attacks presents a significant security risk. Existing studies have mainly examined injection-based attacks, which insert anomalous patterns into code and can often be weakened by standard sanitization and screening. This focus may create a false sense of security regarding backdoor attacks. In this paper, we introduce Semantically-Equivalent Transformation (SET)-based backdoor attacks, a new class of attacks that use semantics-preserving low-prevalence code transformations to generate stealthy triggers. We further propose Stylistic Pattern Backdoor Triggers (SPBT), a framework for constructing and prioritizing such triggers. Our experiments across five tasks and six languages, using models including CodeBERT, CodeT5, and StarCoder, show that SET-based attacks achieve attack success rates comparable to injection-based counterparts while preserving model utility. More importantly, SET-based attacks proved difficult for the evaluated automated defenses and human inspectors to detect, showing substantially lower detectability than injection-based counterparts. We evaluate normalization-based countermeasures and find they offer only partial mitigation. These results motivate further investigation into scalable defenses tailored to SET-based attacks.
The number of recorded vulnerabilities in software continues to rise, despite attempts to improve secure coding practice. Much software security research focuses on software developers’ abilities, tools and motivations. However, software security budgets and priorities ultimately derive from an organisation's senior management. There is little research on senior management's understanding of software security. We interviewed 21 professional developers, managers and consultants to assess perceptions of how management priorities shape software security in organisations. We found that management priorities are important in providing the resources needed for secure software development. Software-security understanding can be lacking in senior management, which, along with budget pressures, can affect its prioritisation. Developers can be self-motivated to code securely despite lack of management support. Disturbingly, some senior managers will actively subvert software security. Acquisitions, divestitures, downsizing, and funding issues may affect an organisation's software security stance. We make a number of recommendations for policymakers, the C-suite and other stakeholders, and suggest directions for further research.
Early detection of performance regressions during software development enables precise attribution of performance changes to specific code revisions and significantly reduces debugging effort and development costs. Regression detection requires stable measurements, since run-to-run noise can obscure important performance changes. Achieving stable measurements is particularly challenging for short-running benchmarks commonly used in CI/CD pipelines. We propose two benchmarking configurations that guarantee measurement stability. The first is a general-purpose configuration, aimed at highly stable measurements, enabling regression detection and reliable performance comparisons across single-threaded benchmarks and different versions of the same multi-threaded benchmark. The second targets multi-threaded benchmarks requiring cross-benchmark performance comparisons. These configurations control multiple factors, grouped into three classes: hardware optimizations, system settings, and runtime options. We evaluate these configurations, together with additional variants that isolate each individual factor and quantify their impact, for a total of \(29\) configurations. We evaluate across five environments, including idle and busy UMA/NUMA servers and a personal laptop, using benchmarks implemented in managed and unmanaged languages. Experimental results show that the proposed configurations are stable: the general-purpose configuration achieves an average relative median absolute deviation (RMAD) below 0.55% across all workloads and environments, while the multi-threaded configuration achieves an average RMAD below 2%.
Failures recovery is essential for forensic analysis and bug diagnosis, yet remains challenging for resource-constrained IoT devices in deployment. These devices lack facilities to record incorrect program states, and advanced hardware tracing mechanisms are typically unavailable. Production environments further restrict their already limited resources. We present Reverb , a lightweight ex-vivo IoT device failure recovery system tailored for edge-centric IoT deployments. Reverb leverages the fact that most external inputs pass through a centralized edge, enabling edge-side input recording without modifying device runtime behavior. For device-local and non-deterministic events not observable at the edge devices, we use recorded external input-guided program analysis to deduce missing local inputs. By combining recorded and inferred data, the cloud can faithfully replay complete device executions on demand. We implemented a prototype and evaluated it in both smart home and industrial scenarios. Reverb introduces no runtime overhead on devices and incurs minimal long-term storage cost. Across 19 failing executions on 13 real-world devices, it successfully reconstructed traces with high consistency to real hardware. The recovered traces further enabled automated bug diagnosis to identify root causes.
The prevailing approach in software engineering for code classification tasks is to rely on large-scale code language models such as StarCoder and CodeLlama. While these models have demonstrated strong performance, they come with the price of intensive computations that raise energy concerns and limit their deployment in resource-constrained environments. In this work, we propose ‘CodeLite’, a relatively lightweight framework that integrates moderately sized code models, such as CodeBERT, with traditional frequency-based techniques, achieving better performance and computational efficiency compared to larger models. Our approach comprises multiple stages, starting with the enhancement of encoder-based code models, where we leverage their intermediate-layer information to capture richer lexical and syntactic signals beyond the default [CLS] token. In parallel, we employ a regex-enhanced TF-IDF component tailored for source code to capture complementary frequency-based patterns. These representations are then combined with the learned code embeddings through suitable fusion techniques for final decision-making. Evaluated across multiple downstream tasks, including programming language identification, authorship attribution, and AI plagiarism, CodeLite consistently outperforms strong baseline models, achieving up to 4.2% absolute accuracy improvement over other approaches and 11.8% incremental gain over ablation components while incurring significantly lower training costs. We further conduct several statistical tests and ablation studies to validate the contribution of each component in the proposed framework. Overall, our results demonstrate that a careful combination of lightweight models along with task-aware feature enhancements can serve as a practical and efficient alternative to heavyweight language models for various software engineering tasks.
Trajectory prediction models have become increasingly vital in various safety-critical domains, such as healthcare, maritime surveillance, and intelligent transportation systems. These applications utilize machine learning models to predict the operational state of a moving agent based on its motion trajectory. Despite their widespread deployment in these domains, testing such models remains a significant challenge due to the high cost of manual labeling, further exacerbated by the large scale and domain-specific characteristics of trajectory datasets. Test input prioritization has emerged as a promising solution to address the labeling cost issue, aiming to identify potentially misclassified inputs early to accelerate debugging and improve testing efficiency. The current state‑of‑the‑art test prioritization technique that can be applied to trajectory prediction tasks is MLPrior. However, MLPrior has the following limitations when applied to trajectory prediction scenarios: 1) MLPrior relies on rich input features, which are limited in trajectory data. Trajectory inputs typically contain only basic spatiotemporal coordinates (e.g., latitude, longitude, timestamps), offering insufficient attribute richness. 2) MLPrior inherits the limitations of learning‑based approaches. When the training data are imbalanced, their effectiveness can be significantly reduced. To overcome these limitations, we propose TrajectoryTest, a trajectory-specific test prioritization strategy that integrates trajectory-related information with MLPrior-derived representations to rank test inputs. Moreover, TrajectoryTest employs an adaptive strategy that switches between learning-based and uncertainty-based ranking depending on the model’s prediction error ratio. We conduct a comprehensive empirical study that confirms the limitations of MLPrior in trajectory prediction scenarios and demonstrates that TrajectoryTest outperforms all existing test input prioritization techniques, including the state-of-the-art approach MLPrior, multiple confidence-based methods, and the baseline random selection. The experimental results show that TrajectoryTest achieves improvements ranging from 7.03% to 9.56% over MLPrior (the state-of-the-art method) and confidence-based approaches on natural datasets, and from 6.71% to 9.94% on noisy datasets.
Over the past decade, modern code review (MCR) has been established as a cornerstone of software quality assurance and a vital channel for knowledge transfer within development teams. However, the manual inspection of increasingly complex systems remains a cognitively demanding and resource-intensive activity, often leading to significant workflow bottlenecks. This article presents a comprehensive roadmap for the evolution of MCR, consolidating over a decade of research (2013–2025) into a unified taxonomy comprising improvement techniques, which focus on the technical optimization and automation of downstream review tasks, and understanding studies, which investigate the underlying socio-technical mechanisms and empirical phenomena of the review process. By diagnosing the current landscape through a strategic SWOT analysis, we examine the transformative impact of generative AI and identify critical gaps between burgeoning AI capabilities and industrial realities. We envision a future where MCR evolves from a human-driven task into a symbiotic partnership between developers and intelligent systems. Our roadmap charts this course by proposing three pivotal paradigm shifts, Context-Aware Proactivity, Value-Driven Evaluation, and Human-Centric Symbiosis, aiming to guide researchers and practitioners in transforming MCR into an intelligent, inclusive, and strategic asset for the AI-driven future.
Technical debt (TD) describes the additional costs that emerge when developers have opted for a quick and easy solution to a problem, rather than a more effective and well-designed, but time-consuming approach. Self-Admitted Technical Debts (SATDs) are a specific type of technical debts that developers intentionally document and acknowledge, typically via textual comments. While these comments are a useful tool for identifying TD, most of the existing approaches focus on capturing tokens associated with various categories of TD, neglecting the rich information embedded within the source code. Recent research has focused on detecting SATDs by analyzing comments, and there has been little work dealing with TD contained in the source code. In this study, through the analysis of comments and their source code from 974 Java projects, we curated the first ever dataset of TD identified by code comments, coupled with its code. We found that including the classified code significantly improves the accuracy in predicting various types of technical debt. We believe that our dataset will catalyze future work in the domain, inspiring various research related to the recognition of technical debt; The proposed classifiers may serve as baselines for studies on the detection of TD.
Blockchain smart contracts automate agreements, replacing third-party intermediaries with immutable and enforceable algorithms. Despite their potential, the complexity of blockchain often undermines human trust and their wide adoption. To address this challenge, we systematise existing knowledge on three critical human-centric qualities: transparency, accountability and understandability. Although these qualities are frequently invoked as pillars of trust in decentralised systems, they remain ambiguously defined and inconsistently operationalised across the literature and platforms. To establish a clearer and more coherent understanding of these qualities in the context of public blockchains and smart contracts, we introduce a structured model that organises existing knowledge into five levels: output, algorithm, external data, process, and application, acquiring insights from literature. This structured approach provides a comprehensive understanding by revealing gaps and areas of consensus, aiding researchers in identifying areas requiring further investigation. To address unclear or conflicting interpretations, we incorporated insights from experienced blockchain developers. We also conduct a comparative analysis to examine whether the characteristics of public blockchains align with standard definitions of transparency, accountability, and understandability. Our findings show that smart contracts support visibility and traceability at low system levels but lack clarity, answerability, and ease of understanding at higher organisational and user interaction levels. These gaps highlight the need for improved communication practices, clearer governance disclosure, and more human-centred design approaches.
Functional programming (FP) promotes immutability and declarative thinking, offering benefits for reliability, maintainability, and scalability. Despite these advantages, FP remains less widely adopted than imperative programming (IP) languages such as C++. Prior studies have explored this gap by analyzing programmers’ self-reported experiences and performance, but cognitive-level insights for novices are limited. To address this, we conducted an eye-tracking study with 49 novice programmers to compare how they comprehend FP (Racket) and IP (C++) during tasks. We designed six paired tasks that covered similar concepts in both languages and employed a mixed-methods approach, including interviews and surveys, to understand their learning experiences. Participants showed fewer fixations and shorter per-element fixation times in Racket than in C++, yet they also frequently revisited parts of the FP code. This combination of lower per-element measures and higher regression rates suggests different cognitive processing patterns in Racket rather than lower effort. Prior C++ experience did not aid in understanding Racket, but gender differences emerged: women spent more time on Racket without improved outcomes. Participants reported lower satisfaction with FP, citing limited exposure, lack of support, and unclear real-world relevance. Our findings offer insights to inform tools, pedagogy, and strategies for making Racket and related FP instruction more approachable in similar educational settings.
Compiler testing is crucial, as compilers serve as foundational infrastructure in software development. Effective compiler testing necessitates not only the generation of diverse test programs but also the systematic specification of compilation options. While existing research predominantly emphasizes test program diversification, it largely overlooks the strategic selection of compilation options required for comprehensive testing. Compilers typically offer a wide range of fine-grained compilation options that allow for precise control over the compilation process, resulting in a vast combination space. Exhaustive enumeration of all option combinations is computationally infeasible, while the stochastic generation of conflictfree, semantically meaningful options presents significant methodological challenges. To address these limitations, we propose OptFuzz, an innovative compiler testing framework that harnesses the generative capabilities of Large Language Models (LLMs) alongside the effectiveness of historical bug-triggering test programs for a comprehensive exploration of the compilation space. OptFuzz leverages LLMs to extract historical bug-triggering test programs from diverse bug reports, which are demonstrated to be effective in uncovering compiler bugs. Therefore, it overcomes the limitations of existing regex-based extraction techniques. Subsequently, OptFuzz employs a code abstraction extraction method based on intermediate representation (IR) to tackle the constraints of LLM context length limitations. Finally, the extracted IR is fed into LLM to generate effective compilation options for compiler testing. Through extensive experiments on GCC and LLVM, OptFuzz demonstrated superior bug detection capability compared to the random compilation space exploration method and other existing technologies. Notably, OptFuzz discovered 64 new bugs in GCC and LLVM, with 53 confirmed or fixed, highlighting our method’s practical utility. The experimental outcomes also indicate that the IR-based analysis substantially decreases overhead and improves bug detection compared to direct utilization of source code.
With the rapid development of software engineering technologies, multilingual programming, i.e., developing a program using multiple programming languages (PLs) collaboratively, has become increasingly popular. This practice is often adopted to reuse existing code, to leverage advantages of specific PLs, to satisfy various software quality needs, and to enhance software development efficiency. However, it also introduces cross-language bugs (CLBs), bugs caused by interactions between multiple PLs. Most existing bug detection tools and techniques are primarily designed for a single PL and are less effective at detecting CLBs. Given the significant advancements of Pre-trained Language Models (PLMs) in the code domain, this work investigates the performance of pre-trained Code Language Models (CodeLMs) in CLB prediction tasks. To this end, we designed and developed a cross-language code identification tool (CLCFinder), and used it to collect and construct a CLB dataset, which involves combinations of three PLs (i.e., Python-C/C++, Java-C/C++, and Python-Java) with nine types of interaction mechanisms. We fine-tuned 13 popular CodeLMs in different sizes on this dataset, in which 80% of the data was used for fine-tuning, 10% for validation, and 10% for testing. We then evaluated their performance in CLB detection. Additionally, we compared the fine-tuning performance of five top-performing models on current state-of-the-art (SOTA) single-language bug datasets to assess their application in CLB detection. Furthermore, we evaluated factors affecting the performance of fine-tuned CodeLMs in CLB detection, including the size of the fine-tuning dataset and the length of token sequence. Finally, we explored the impact of code comments on the performance of fine-tuned CodeLMs. The experimental results are that: First, all 13 CodeLMs exhibited varying degrees of performance improvement after fine-tuning. In particular, after fine-tuning, the UniXcoder-base model achieved the best performance, with an F1 score of 0.7407. Notably, within the scope of our experimental setup, small CodeLMs tended to achieve better performance than large CodeLMs, while large CodeLMs showed limited improvement. Second, the fine-tuned CodeLMs on single-language bug datasets performed poorly in CLB detection, further demonstrating the significant feature differences between CLBs and single-language bugs. Third, enlarging the fine-tuning dataset size significantly enhanced the model performance in CLB detection, but increasing token sequence length did not necessarily improve the model performance. Finally, regarding the impact of code comments, the performance varied significantly across different CodeLMs. Specifically, some CodeLMs’ performance was improved when fine-tuned with code comments, while others showed degraded performance.
Background : Effort-aware metrics (EAMs) have the merit of including software analysis effort in the evaluation of software defect prediction (SDP) models. Yet, almost all EAMs concentrate only on the effectiveness of SDP models, i.e., how well they identify actually defective modules. Instead, EAMs ignore the efficiency of SDP models, i.e., their ability to not classify non-defective modules as defective. Efficiency is important because modules incorrectly classified as defective will undergo useless analysis, thus causing effort waste, and also undermining the confidence of developers in the usefulness of SDP models. Aim : We aim to represent the performance of SDP models along several dimensions, while taking into consideration the effort needed for module analysis. Specifically, we look for indicators that account for both effectiveness and efficiency. Method : We extend Receiver Operating Characteristic (ROC) curves to explicitly represent analysis effort. To this end, we add iso-effort lines to the ROC space and iso-effort points to ROC curves. Iso-effort curves and points show where each point of the ROC space (corresponding to a classifier) is positioned with respect to analysis effort. We also define effort-aware metrics based on the area under the ROC curve. Results : We show that the proposed effort-aware ROC curves support systematic and complete evaluation of SDP models’ performance. Specifically, we show that a single effort-aware ROC curve provides the same information as multiple traditional EAMs. In addition, effort-aware ROC curves provide evaluation-oriented information in a coherent and visually intuitive way. We also show the practical application of the proposed techniques to well-known software defectiveness datasets. Conclusions : The proposed technique can effectively support both researchers and developers in comparing and selecting SDP models.
Web accessibility is essential for inclusive digital experiences, yet the accessibility of LLM-generated code remains underexplored. This paper presents an empirical study comparing the accessibility of web code generated by GPT-4o, Qwen2.5-Coder-32B-Instruct-AWQ, and Gemini-3-Flash against human-written code. Results show that LLMs often produce more accessible code, especially for basic features like color contrast and alternative text, but struggle with complex issues such as ARIA attributes. We also assess advanced prompting strategies (Zero-Shot, Few-Shot, Self-Criticism), finding they offer some gains but are limited. To address these gaps, we introduce FeedA11y , a feedback-driven ReAct-based approach that demonstrates the potential of incorporating accessibility evaluation results into the code generation process. Our work highlights the promise of LLMs for accessible code generation and emphasizes the need for feedback-based techniques to address persistent challenges. We provide the source code and datasets that were used in our experiments in the companion website [15].
Java frameworks such as Spring simplify the development process and enhance productivity, but they also introduce new security risks, particularly in handling flexible development paradigms like entity parameter passing and configuration, which increase the potential for injection attacks. However, existing vulnerability detection tools often focus on code slicing or taint propagation analysis, and struggle with framework complexity, such as their inability to parse data flows in configuration files or correctly handle entity parameter passing paths, leading to many potential vulnerabilities going undetected, while also struggling to address developer-defined sanitizers, resulting in a higher false positive rate. To address these challenges, we developed iDetector , a static analysis tool specifically designed to detect injection vulnerabilities in Java frameworks. First, iDetector constructs a comprehensive code graph using a Code Property Graph (CPG), enabling it to flexibly handle scenarios that require dynamic structural modifications, such as XML configuration injection and dynamic method call resolution. Next, iDetector employs LLM-assisted sanitizer analysis techniques to reduce false positives caused by developer-defined sanitizers substantially. In a comprehensive evaluation of real-world Java projects and existing state-of-the-art tools, iDetector demonstrates superior precision and recall. It has uncovered 172 previously unknown vulnerabilities, of which 51 have been assigned CVE identifiers.
Unit testing is essential for early defect detection and software reliability. Existing automated test generation tools, including both search-based and Large Language Model (LLM)-centric approaches, still face significant challenges, including difficulties in producing compilable and executable tests for complex projects, limited coverage with insufficient exploration of control flows and edge cases, and a lack of systematic feedback mechanisms that integrate global search, semantic reasoning, and adaptive refinement. To address these limitations, we present LegaTest , an automated unit test generation framework that synergizes LLM with Genetic Algorithms (GA) through a coordinated Generation–Repair–Optimization process. LegaTest begins with a dual-prompt strategy to generate structurally sound and semantically meaningful test seeds, which are then iteratively refined using execution feedback and coverage guidance. The framework integrates LLM-driven insights with GA operators such as greedy selection, LLM-guided crossover, and semantic-driven mutation. Multi-strategy test repair with hierarchical criteria progressively resolves syntactic and semantic errors, improving both correctness and robustness. On the Defects4J benchmark as well as four Java projects adopted from the experimental setup of ChatUniTest, LegaTest achieves 66.95% line, 54.37% branch, and 74.16% method coverage with 93.3% generation success and 18.7% semantically meaningful assertions, outperforming existing tools by up to 34.7% in coverage, 59.6% in generation success, and 16.5% in assertions, while maintaining concise test suites.
