Bugs are essential in software engineering; many research studies in the past decades have been proposed to detect, localize, and repair bugs in software systems. Effectiveness evaluation of such techniques requires complex bugs, i.e., those that are hard to detect through testing and hard to repair through debugging. From the classic software engineering point of view, a hard-to-repair bug differs from the correct code in multiple locations, making it hard to localize and repair. Hard-to-detect bugs, on the other hand, manifest themselves under specific test inputs and reachability conditions. These two objectives, i.e., generating hard-to-detect and hard-to-repair bugs, are mostly aligned; a bug generation technique can change multiple statements to be covered only under a specific set of inputs. However, these two objectives conflict in the learning-based techniques: A bug should have a similar code representation to the correct code in the training data to challenge a bug prediction model to distinguish them. The hard-to-repair bug definition remains the same but with a caveat: the more a bug differs from the original code (at multiple locations), the more distant their representations are and easier to detect. This demands new techniques to generate bugs to complement existing bug datasets to challenge learning-based bug prediction and repair techniques. We propose BUGFARM to transform arbitrary code into multiple hard-to-detect and hard-to-repair bugs. BUGFARM mutates code in multiple locations (hard-to-repair) but leverages attention analysis to only change the least attended locations by the underlying model (hard-to-detect). Our comprehensive evaluation of 435k+ bugs from over 1.9M mutants generated by BUGFARM and two alternative approaches demonstrates our superiority in generating bugs that are hard to detect by learning-based bug prediction approaches (up to 40.53% higher False Negative Rate and 10.76%, 5.2%, 28.93%, and 20.53% lower Accuracy, Precision, Recall, and F1 score) and hard to repair by state-of-the-art learning-based program repair technique (28% repair success rate compared to 36% and 49% of LEAM and mu BERT bugs). BUGFARM is efficient, i.e., it takes nine seconds to mutate a code with no training overhead.
The ubiquitous use of third-party libraries in software development has enabled developers to quickly add new functionality to their client software. Unfortunately, library usage also carries a cost in terms of software maintenance: library upgrades may include breaking changes, in which client expectations about library behaviour are no longer met in new library versions. Behavioural breaking changes can be particularly insidious, and in their full generality, could require sophisticated program analysis techniques to (approximately) detect.In this work, we present our UnCheckGuard tool, which detects a class of behavioural breaking changes—those related to exceptions thrown by Java libraries. UnCheckGuard analyzes both sides of the library/client duet. On the library side, UnCheckGuard creates a list of new exceptions that may be thrown by methods in a library’s public API, including by its transitive callees. On the client side, UnCheckGuard identifies client methods that call library methods with new exceptions. To reduce false positives, UnCheckGuard additionally filters out new exceptions that cannot be triggered by particular clients, using taint analysis. It therefore can be used by client developers as a tool to screen library updates for relevant incompatibilities.We have evaluated UnCheckGuard on 302 libraries and 352 library-client pairs drawn from the DUETS collection and found 120 libraries with newly-added exceptions, as well as 1708 callsites to library methods which, when upgraded to the latest version, may introduce a behavioural breaking change in the client due to a newly added unchecked exception. These findings highlight the practical value of UnCheckGuard in identifying exception-related incompatibilities introduced by library upgrades.
Serverless computing significantly alters software development by abstracting infrastructure management and enabling rapid, modular, event-driven deployments. Despite its benefits, the distinct characteristics of serverless functions, such as ephemeral execution and fine-grained scalability, pose unique security challenges, particularly in open-source platforms like OpenFaaS. Existing approaches typically address isolated phases of the DevSecOps lifecycle, lacking an integrated and comprehensive security strategy. To bridge this gap, we propose FaaSGuard, a unified DevSecOps pipeline explicitly designed for open-source serverless environments. FaaSGuard systematically embeds lightweight, fail-closed security checks into every stage of the development lifecycle—planning, coding, building, deployment, and monitoring—effectively addressing threats such as injection attacks, hard-coded secrets, and resource exhaustion. We validate our approach empirically through a case study involving 20 real-world serverless functions from public GitHub repositories. Results indicate that FaaSGuard effectively detects and prevents critical vulnerabilities, demonstrating high precision (95%) and recall (91%) without significant disruption to established CI/CD practices.
Large language models (LLMs) have demonstrated impressive capabilities in code generation, achieving high scores on benchmarks such as HumanEval and MBPP. However, these benchmarks primarily assess functional correctness and neglect broader dimensions of code quality, including security, reliability, readability, and maintainability. In this work, we systematically evaluate the ability of LLMs to generate high-quality code across multiple dimensions using the PythonSecurityEval benchmark. We introduce an iterative static analysis-driven prompting algorithm that leverages Bandit and Pylint to identify and resolve code quality issues. Our experiments with GPT-4o show substantial improvements: security issues reduced from >40% to 13%, readability violations from >80% to 11%, and reliability warnings from >50% to 11% within ten iterations. These results demonstrate that LLMs, when guided by static analysis feedback, can significantly enhance code quality beyond functional correctness.
Background: Software systems powered by large language models are becoming a routine part of everyday technologies, supporting applications across a wide range of domains. In software engineering, many studies have focused on how LLMs support tasks such as code generation, debugging, and documentation. However, there has been limited focus on how full systems that integrate LLMs are tested during development. Aims: This study explores how LLM-powered systems are tested in the context of real-world application development. Method: We conducted an exploratory case study using 99 individual reports written by students who built and deployed LLM-powered applications as part of a university course. Each report was independently analyzed using thematic analysis, supported by a structured coding process. Results: Testing strategies combined manual and automated methods to evaluate both system logic and model behavior. Common practices included exploratory testing, unit testing, and prompt iteration. Reported challenges included integration failures, unpredictable outputs, prompt sensitivity, hallucinations, and uncertainty about correctness. Conclusions: Testing LLM-powered systems required adaptations to traditional verification methods, blending source-level reasoning with behavior-aware evaluations. These findings provide evidence on the practical context of testing generative components in software systems.
Modern code review (MCR) is recognized as an effective software quality assurance practice that is broadly adopted by open-source and commercial software projects. MCR is most effective when developers follow best practices, as it improves code quality, enhances knowledge transfer, increases team awareness and shares code ownership. However, prior work highlights that poor code review practices are common and often manifest in the form of low review participation and engagement, shallow review, and toxic communications. To address these issues, we introduce GitRev, a novel approach that applies gamification mechanisms to boost developer motivation and engagement. GitRev is built on top of a Large Language Model (LLM), used as a points-based reward system that leverages the code change context, and code review activities. We implement GitRev as a GitHub app with a web browser extension that consists of a client-side web browser extension that gamifies the GitHub user interface, and a server-side composed of a Node.js server for authentication and data management. To evaluate GitRev, we conduct a controlled experiment with 86 graduate and undergraduate students. Results indicate the promising potential of our approach for improving the code review process and developers' engagement. GitRev is publicly available at https://anonymous.4open.science/r/GitRev-0B74
In today's software development landscape, the use of third-party libraries is near-ubiquitous; leveraging third-party libraries can significantly accelerate development, allowing teams to implement complex functionalities without reinventing the wheel. However, one significant cost of reusing code is security vulnerabilities. Vulnerabilities in third-party libraries have allowed attackers to breach databases, conduct identity theft, steal sensitive user data, and launch mass phishing campaigns. Notorious examples of vulnerabilities in libraries from the past few years include log4shell, solarwinds, event-stream, lodash, and equifax. Existing software composition analysis (SCA) tools track the propagation of vulnerabilities from libraries through dependencies to downstream clients and alert those clients. Due to their design, many existing tools are highly imprecise-they create alerts for clients even when the flagged vulnerabilities are not exploitable. Library developers occasionally release new versions of their software with refactorings that improve modularity. In this work, we explore the impacts of modularity improvements on vulnerability detection. In addition to generally improving the nonfunctional properties of the code, refactoring also has several security-related beneficial side effects: (1) it improves the precision of existing (fast and stable) SCAs; and (2) it protects from vulnerabilities that are exploitable when the vulnerable code is present and not even reachable, as in gadget chain attacks. Our primary contribution is thus to quantify, using a novel simulation-based counterfactual vulnerability analysis, two main ways that improved modularity can boost security. We propose a modularization method using a DAG partitioning algorithm, and statically measure properties of systems that we (synthetically) modularize. In our experiments, we find that modularization can improve precision of Software Composition Analysis (SCA) tools to 71%, up from 35%. Furthermore, migrating to modularized libraries results in 78% of clients no longer being vulnerable to attacks referencing inactive dependencies. We further verify that the results of our modularization reflect the structures that are already implicit in the projects (but for which no modularity boundaries are enforced).
Refactoring is a non-trivial maintenance activity. Developers spend time and effort refactoring code to remove structural problems, i.e., code smells. Recent studies indicated that developers often apply composite refactoring (composite, for short), i.e., two or more interrelated refactorings. However, prior studies revealed that only 10% of composite refactorings are considered complete, i.e., those fully removing code smells. Many incomplete refactorings can even replace or introduce smells, requiring additional effort for their removal later in the project. Moreover, existing refactoring recommendations are not well-detailed and do not alert developers about these possible side effects. To address these gaps, we conducted a large-scale study involving more than 250k refactorings from 42 software projects, including both open-source and closed-source projects. Our goal is to investigate how the most common complete composites are combined and their side effects in the practice. Our results reveal that the current recommendation to apply Extract Method(s) with fine-grained refactoring types needs refinements. We found that certain fine-grained refactorings like Change Variable Types and Change Return Types can introduce up to 45% of Brain Methods when combined with Extract Method(s). Moreover, Ex-tract Method(s) and Move Method(s), a common recommendation to remove Feature Envy, may inadvertently introduce about 30% of Lazy Classes and approximately 70% of Data Classes. Despite these potential side effects, existing refactoring catalogs and tools' recommenders do not alert developers about these side effects. Finally, we consolidate our findings into a catalog to provide clear guidance for developers and researchers on effectively applying composite refactorings to fully remove code smells.
Docker plays a crucial role in providing uniform software development. Many Docker development projects deliver multiple images in order to support various users who need different base images, versions, and architectures. To do so, the projects need to develop different contents of Dockerfiles for each support. For example, if developers provide their product on different Linux OSs, Dockerfiles need to contain package installing commands with an appropriate package manager for each Linux OS. To reduce the development tasks, many projects often develop their own tool to generate multiple Docker images automatically (hereafter, Dockerfile Preprocessors). However, it is still not clear how the projects adopt Dockerfile Preprocessors and what the benefits are. This study explores the characteristics of projects using Dockerfile Preprocessors, the timing, impact, and purpose, and the maintenance effort of using Dockerfile Preprocessors. Our empirical results show that (i) there is “Container build” pattern that does not generate multiple Dockerfiles; (ii) Projects using DPPs have more tags, supported Docker images, and architecture supports than projects without DPPs; (iii) 66% of projects develop DPPs in the middle of development; (iv) the common reasons for adopting DPP is to reduce the effort of creating Dock-erfiles, and to ease updating versions/variations/architectures; (v) the adoption of DPPs does not increase releasing activities.
Searching for code aims to return code snippets that correspond to specified queries. Improving the accuracy of matching between heterogeneous natural language query inputs and highly structured program language source code is a fundamental issue in code search. The semantics expressed by code statements are not only related to themselves but also to the context in which they exist. Control Flow Graph (CFG) contains the sequence, branching and looping structure relationships of execution of program statements, which are important contextual information for understanding code functionality semantics. Additionally, CFG statements themselves not only contain textual features but also code syntax structure features. However, existing methods have failed to effectively utilize the characteristics of CFG, resulting in inadequate search accuracy. Concerning this issue, our paper constructs a node-enhanced Control Flow Graph(node-enhanced CFG) by setting text and syntax structure attributes for CFG nodes. We also propose a Code Search method based on Node-enhanced CFG embedding, called NCFG-CS. To fully extract features of the node-enhanced CFG, we employ an asymptotic fusion strategy. We first fuse the text features and syntax structure features of code statements, and then merge the features of the entire graph of node-enhanced CFG. In experiments comparing with existing advanced multimodal methods on public datasets, NCFG-CS improves MRR by at least 5%. Ablation experiments indicate that syntax structure features of statements contribute more to code search. Additionally, to verify the performance of NCFG-CS in real search scenarios and the generalization of experiments, we test our model using code-query datasets provided by Gu et al. The experiment once again verified the effectiveness of N CFG-CS, while also demonstrating its good generalization ability.
With the growing popularity of mobile apps, their complexity and resource demands have increased as end users seek more sophisticated features. This rise in complexity often leads to performance issues, which have become a significant concern for developers and are now seen as critical for main-taining competitiveness in the market. In this paper, we conduct an empirical study aimed at analyzing performance issues and their causes in Android apps. To achieve this, we examine 385 Stack Overflow (SO) Q&A posts out of a total of 6059 posts. We qualitatively analyzed and formulated two taxonomies: one for performance issues and another for their causes (first of its kind). Our study identifies key categories of performance-related issues, including UI/UX & Graphics, Development/Coding Practices, Performance Optimization, and Data/Database Han-dling. Similarly, the main causes of performance issues include Inefficient User Interface Design, Poor Resource Management, and Lack of Development/Coding Practices, among others. These comprehensive taxonomies serve as a checklist or guide for developers to help them identify, know the causes (diagnosing), and resolve performance issues in their app development.
Robustness is critical for ensuring that software functions correctly under adverse conditions. Exception-handling mechanisms in programming languages enable developers to deal with these adverse conditions. However, implementing exception-related code can present significant challenges to developers. We investigated exception-related code contributions across Java projects in the Apache ecosystem. We analyzed exception-related pull requests (exception-PRs), which were detected using a validated heuristic. We produced a comprehensive dataset of 988 exception-PRs. We observed no statistically significant differences in complexity metrics between exception-PRs and non-exception-PRs. We also found no significant differences in developers' behavior metrics, indicating consistent engagement regardless of whether the pull request addressed exception-related code. A manual analysis revealed that most exception-PRs focused on system improvements rather than bug fixes, suggesting proactive efforts to enhance software robustness. Moreover, the most frequently addressed aspects of exceptional code in these exception-PRs were: (i) the external representation of adverse situations to end-users (more than 40% of the PRs) and (ii) the implementation of effective error-handling actions (nearly 35% of the PRs) to promote program recoverability. Interestingly, a significant proportion of exception-PRs simultaneously addressed multiple aspects. By understanding the nature and characteristics of exception-PRs, we expect to better support developers in managing erroneous conditions and improving software robustness.
To identify the source of runtime errors, techniques such as “debug-by-execution” are widely used, e.g., steppers. Although this technique can be useful for non-runtime errors, it has been less extensively explored because of the inherent difficulties in stepping through programs with such errors. In this new ideas and emerging results paper, we focus on debugging type errors in statically typed languages. We explore “transformational debugging,” a method that transforms erroneous code to provide users with an alternative perspective on the source code, thereby facilitating the debugging process. The proposed approach comprises three phases: In the first phase, we use the error messages from the compiler's type inferencer to “freeze” parts of the program. In the second phase, the type inferencer is used to reduce the number of “frozen” parts. In the third phase, a partial evaluator evaluates the partially “frozen” program. During partial evaluation, the frozen parts are treated as dynamic and therefore remain as code without being evaluated, whereas the other parts are treated as static and are evaluated. This study proposes the generic algorithms for our debugging method and two heuristic strategies for the algorithms. We obtain promising results using a prototype for a subset of OCaml. We explore a novel avenue for debugging: Transformational debugging and partial evaluation as helpful debugging tools.
GitHub Actions (GHA) are a way to automate CI/CD workflows within the GitHub platform. The deep integration of GHA into GitHub enables to automate a wide range of social and technical activities. In this study, we investigate workflow smells, i.e., characteristics in the workflow that possibly indicate a deeper problem. Through a mining study, we first expose a list of frequent change patterns in the workflows of 83 GitHub projects. We then manually analyze these frequent change patterns to understand the negative effects that the frequent changes try to remove. To validate the list of 22 potential workflow smells that we thus obtain, we carry out a contribution study with 32 projects on GitHub through pull requests that contain a fix to the candidate smell. By qualitatively analyzing the maintainers' comments in 32 pull requests, we settle on 7 confirmed GHA workflow smells.
Writing effective unit tests is often tedious, difficult, and time-consuming. Test recommendation techniques facilitate this process by recommending existing manually written tests from other similar systems for developers to reuse. However, developers still have to put non-trivial effort into modifying the recommended tests. For example, they have to understand vari-ous code elements in the recommended tests to accurately replace them with semantically similar code elements from the target system. In this paper, we propose JTESTMIGRATOR, a technique to automatically migrate unit tests between semantically similar methods across applications. Given a source and a target method with similar functionality across applications, where the source method has some unit tests, JTESTMIGRATOR migrates unit tests by transforming the test code. JTESTMIGRATOR uses semantic similarity and type compatibility of code elements in the source and target systems to transform test code. We implement JTESTMIGRATOR to migrate JUnit tests and evaluate it on 104 tests for 42 methods across 5 popular libraries. JTESTMIGRATOR successfully migrates 76 (73%) of the tests for 32 (76%) of the methods across the 5 libraries. 10 (13%) of the successfully migrated tests increase the code coverage of 4 target methods.
Software engineers employ a variety of approaches to ensure the quality of software systems, including software testing, modern code review, automated static analysis, build automation, and continuous integration. Previous research has shown that software engineers lack situational awareness about the quality assurance (QA) practices in their projects. We propose software quality assurance analytics to help software engineers become aware of their QA usage, and reflect upon it. We have developed a prototype quality assurance analytics dashboard coined RepoInsights that provides both a global overview and a comparative aspect of the different QA practices. Through a qualitative study involving 14 participants who have completed an assignment using our RepoInsights dashboard, we elicit their perspective on quality assurance analytics. We observe that the dashboard has enabled the software engineers to reflect on the QA practices of software projects. Additionally, we distill a set of recommendations for future quality assurance analytics.
Code readability strongly influences code compre-hension and, to some degree, code quality. Unreadable code makes software maintenance more challenging and is prone to more bugs. To improve the readability, using good identifier names is crucial. Existing studies on automatic identifier re-naming have not considered aspects such as the code context. Additionally, prior research has done little to address the typical challenges inherent in the identifier renaming task. In this paper, we propose a new approach for renaming identifiers in source code by fine-tuning a transformer model. Through the use of perplexity as an evaluation metric, our results demonstrate a significant decrease in the perplexity values for the fine-tuned approach compared to the baseline, reducing them from 363 to 36. To further validate our method, we conduct a developers' survey to gauge the suitability of the generated identifiers, comparing original identifiers with identifiers generated with our approach as well as two state-of-the-art large language models, GPT-4 Turbo and Gemini Pro. Our approach generates better identifier names than the original names and exhibits competitive performance with state-of-the-art commercial large language models. The proposed method carries significant implications for software developers, tool vendors, and researchers. Software developers may use our proposed approach to generate better variable names, increasing the clarity and readability of the software. Researchers in the field may use and build upon the proposed approach for variable renaming.
As the use of microservices continues to grow and become a foundational approach to architecting software solutions, ensuring the security of microservices is paramount. Docker images have emerged as the predominant solution to containerize microservices-and thus, Docker images are becoming a large attack surface. Thus, reducing vulnerabilities in Docker images will reduce microservice cyberattacks. A common way to find vulnerabilities in Docker images employs static analysis tools like Trivy and Grype. However, these tools frequently generate disparate vulnerability reports when analyzing the same Docker image, thus causing uncertainty in tool selection. We collected 927 Docker images, analyzed them with Trivy and Grype, and compared the vulnerabilities reported in each image. Among the 865 images found to have vulnerabilities, Trivy and Grype disagreed on both the number of vulnerabilities and the vulnerability IDs found therein. Since both tools interface with external vulnerability databases, some discrepancies can be attributed to how the tools interface with these external resources. The external vulnerability databases partially overlap and frequently contradict one another, thereby creating challenges for static analysis tool developers and end users alike. This New Ideas and Emerging Results (NIER) study contains new and critical information that practitioners need for selecting and using static analysis tools-given that increases in the use of Docker technologies means increases in the size of the attack surfaces.
A source code difference (diff) indicates changes made by comparing new and old source codes, and it can be utilized in code reviews to help developers understand the changes made to the code. Although many diff generation methods have been proposed, existing automatic methods may generate nonoptimal diffs, hindering reviewers from understanding the changes. In this paper, we propose an interactive approach to optimize diffs. Users can provide feedback for the points of a diff that should not be matched but are or parts that should be matched but are not. The edit graph is updated based on this feedback, enabling users to obtain a more optimal diff. We simulated our proposed method by applying a search algorithm to empirically assess the number of feedback instances required and the amount of diff optimization resulting from the feedback to investigate the potential of this approach. The results of 23 GitHub projects confirm that 92% of nonoptimal diffs can be addressed with less than four feedback actions in the ideal case.
Code clones are code snippets that are identical or similar to other snippets within the same or different files. They are often created through copy-and-paste practices and modified during development and maintenance activities. Since a pair of code clones, known as a clone pair, has a possible logical coupling between them, it is expected that changes to each snippet are made simultaneously (co-changed) and consistently. There is extensive research on code clones, including studies related to the co-change of clones; however, detailed analysis of commit logs for code clone pairs has been limited. In this paper, we investigate the commit logs of code snippets from clone pairs, using the git-log command to extract changes to cloned code snippets. We analyzed 45 repositories owned by the Apache Software Foundation on GitHub and addressed three research questions regarding commit frequency, co-change ratio, and commit patterns. Our findings indicate that (1) on average, clone snippets are changed infrequently, typically only two or three times throughout their lifetime, (2) the ratio of co-changes is about half of all clone changes, with 10-20% of co-changed commits being concerning (potentially inconsistent), and (3) 35-65% of all clone pairs being classified as concerning clone pairs (potentially inconsistent clone pairs). These results suggest the need for a consistent management system through the commit timeline of clones.
