Bugs are essential in software engineering; many research studies in the past decades have been proposed to detect, localize, and repair bugs in software systems. Effectiveness evaluation of such techniques requires complex bugs, i.e., those that are hard to detect through testing and hard to repair through debugging. From the classic software engineering point of view, a hard-to-repair bug differs from the correct code in multiple locations, making it hard to localize and repair. Hard-to-detect bugs, on the other hand, manifest themselves under specific test inputs and reachability conditions. These two objectives, i.e., generating hard-to-detect and hard-to-repair bugs, are mostly aligned; a bug generation technique can change multiple statements to be covered only under a specific set of inputs. However, these two objectives conflict in the learning-based techniques: A bug should have a similar code representation to the correct code in the training data to challenge a bug prediction model to distinguish them. The hard-to-repair bug definition remains the same but with a caveat: the more a bug differs from the original code (at multiple locations), the more distant their representations are and easier to detect. This demands new techniques to generate bugs to complement existing bug datasets to challenge learning-based bug prediction and repair techniques. We propose BUGFARM to transform arbitrary code into multiple hard-to-detect and hard-to-repair bugs. BUGFARM mutates code in multiple locations (hard-to-repair) but leverages attention analysis to only change the least attended locations by the underlying model (hard-to-detect). Our comprehensive evaluation of 435k+ bugs from over 1.9M mutants generated by BUGFARM and two alternative approaches demonstrates our superiority in generating bugs that are hard to detect by learning-based bug prediction approaches (up to 40.53% higher False Negative Rate and 10.76%, 5.2%, 28.93%, and 20.53% lower Accuracy, Precision, Recall, and F1 score) and hard to repair by state-of-the-art learning-based program repair technique (28% repair success rate compared to 36% and 49% of LEAM and mu BERT bugs). BUGFARM is efficient, i.e., it takes nine seconds to mutate a code with no training overhead.
The ubiquitous use of third-party libraries in software development has enabled developers to quickly add new functionality to their client software. Unfortunately, library usage also carries a cost in terms of software maintenance: library upgrades may include breaking changes, in which client expectations about library behaviour are no longer met in new library versions. Behavioural breaking changes can be particularly insidious, and in their full generality, could require sophisticated program analysis techniques to (approximately) detect.In this work, we present our UnCheckGuard tool, which detects a class of behavioural breaking changes—those related to exceptions thrown by Java libraries. UnCheckGuard analyzes both sides of the library/client duet. On the library side, UnCheckGuard creates a list of new exceptions that may be thrown by methods in a library’s public API, including by its transitive callees. On the client side, UnCheckGuard identifies client methods that call library methods with new exceptions. To reduce false positives, UnCheckGuard additionally filters out new exceptions that cannot be triggered by particular clients, using taint analysis. It therefore can be used by client developers as a tool to screen library updates for relevant incompatibilities.We have evaluated UnCheckGuard on 302 libraries and 352 library-client pairs drawn from the DUETS collection and found 120 libraries with newly-added exceptions, as well as 1708 callsites to library methods which, when upgraded to the latest version, may introduce a behavioural breaking change in the client due to a newly added unchecked exception. These findings highlight the practical value of UnCheckGuard in identifying exception-related incompatibilities introduced by library upgrades.
Serverless computing significantly alters software development by abstracting infrastructure management and enabling rapid, modular, event-driven deployments. Despite its benefits, the distinct characteristics of serverless functions, such as ephemeral execution and fine-grained scalability, pose unique security challenges, particularly in open-source platforms like OpenFaaS. Existing approaches typically address isolated phases of the DevSecOps lifecycle, lacking an integrated and comprehensive security strategy. To bridge this gap, we propose FaaSGuard, a unified DevSecOps pipeline explicitly designed for open-source serverless environments. FaaSGuard systematically embeds lightweight, fail-closed security checks into every stage of the development lifecycle—planning, coding, building, deployment, and monitoring—effectively addressing threats such as injection attacks, hard-coded secrets, and resource exhaustion. We validate our approach empirically through a case study involving 20 real-world serverless functions from public GitHub repositories. Results indicate that FaaSGuard effectively detects and prevents critical vulnerabilities, demonstrating high precision (95%) and recall (91%) without significant disruption to established CI/CD practices.
Large language models (LLMs) have demonstrated impressive capabilities in code generation, achieving high scores on benchmarks such as HumanEval and MBPP. However, these benchmarks primarily assess functional correctness and neglect broader dimensions of code quality, including security, reliability, readability, and maintainability. In this work, we systematically evaluate the ability of LLMs to generate high-quality code across multiple dimensions using the PythonSecurityEval benchmark. We introduce an iterative static analysis-driven prompting algorithm that leverages Bandit and Pylint to identify and resolve code quality issues. Our experiments with GPT-4o show substantial improvements: security issues reduced from >40% to 13%, readability violations from >80% to 11%, and reliability warnings from >50% to 11% within ten iterations. These results demonstrate that LLMs, when guided by static analysis feedback, can significantly enhance code quality beyond functional correctness.
Background: Software systems powered by large language models are becoming a routine part of everyday technologies, supporting applications across a wide range of domains. In software engineering, many studies have focused on how LLMs support tasks such as code generation, debugging, and documentation. However, there has been limited focus on how full systems that integrate LLMs are tested during development. Aims: This study explores how LLM-powered systems are tested in the context of real-world application development. Method: We conducted an exploratory case study using 99 individual reports written by students who built and deployed LLM-powered applications as part of a university course. Each report was independently analyzed using thematic analysis, supported by a structured coding process. Results: Testing strategies combined manual and automated methods to evaluate both system logic and model behavior. Common practices included exploratory testing, unit testing, and prompt iteration. Reported challenges included integration failures, unpredictable outputs, prompt sensitivity, hallucinations, and uncertainty about correctness. Conclusions: Testing LLM-powered systems required adaptations to traditional verification methods, blending source-level reasoning with behavior-aware evaluations. These findings provide evidence on the practical context of testing generative components in software systems.
