Many popular Python libraries use C-extensions for performance-critical operations allowing users to combine the best of the two worlds: The simplicity and versatility of Python and the performance of C. A drawback of this approach is that exceptions raised in C can bypass Python's exception handling and cause the entire interpreter to crash. These crashes are real faults if they occur when calling a public API. While automated test generation should, in principle, detect such faults, crashes in native code can halt the test process entirely, preventing detection or reproduction of the underlying errors and inhibiting coverage of non-crashing parts of the code. To overcome this problem, we propose separating the generation and execution stages of the test-generation process. We therefore adapt Pynguin, an automated test case generation tool for Python, to use subprocess-execution. Executing each generated test in an isolated subprocess prevents a crash from halting the test generation process itself. This allows us to (1) detect such faults, (2) generate reproducible crash-revealing test cases for them, (3) allow studying the underlying faults, and (4) enable test generation for non-crashing parts of the code. To evaluate our approach, we created a dataset consisting of 1648 modules from 21 popular Python libraries with C-extensions. Subprocess-execution allowed automated testing of up to 56.5

Virtual Reality (VR) applications are increasingly being integrated across a wide range of domains, including surgical training and industrial marketing. However, the long-term adoption and maintenance of VR applications remain limited, particularly due to the lack of effective, systematic, and reproducible software testing approaches tailored to their unique characteristics. To address this issue, we introduce UltraInstinctVR, a novel testing approach for VR applications. Relying on predefined VR models (scenarios), it automates the generation and execution of concrete VR system tests. In our empirical evaluation, we compare UltraInstinctVR with state-of-the-art automated VR testing approaches in terms of coverage and failure detection on 10 open-source VR applications. The results show that UltraInstinctVR outperforms existing automated tools for detecting unique failures and provides valuable insights for identifying real-world bugs in VR applications.

A Pull Request (PR) is a code change proposal submitted by a developer for review by other team members. For a reviewer, prioritizing and selecting a PR to review is not easy: this selection can depend on many factors, including the extent of the change, the time available, their state of mind, etc. Existing tools and platforms like GitHub offer support for visualizing code changes and recommending reviewers to assign a PR to. However, little research has been done on providing support to the reviewer to select a PR to review from a list of (assigned) PRs. In this paper, we propose PRECOG, a Visual Studio Code plugin to help reviewers select the appropriate PR by enabling more informed decisions. PRECOG offers a radar chart visualization of the PR's difficulty across various dimensions, and ranks PRs according to their difficulty. We evaluated PRECOG with 9 developers in a user experiment on the Java Spring Framework. Our results show that although participants differed in how they assess PRs' difficulty, all agreed that the synthetic view provided by PRECOG helps them make rapid, informed decisions about which PR to review first. PRECOG is available at https://github.com/snail-unamur/PRECOG. A demonstration video is available at https://youtu.be/qPlr6lmgyMM.

Green software engineering is emerging as a crucial response to information technology's rising energy impact, especially in continuous development. However, there remain challenges in devising automated methods for identifying energy regressions across commits and their associated code change patterns. In particular, little effort has been put into automatically detecting regressions at the commit level by identifying statistically significant changes in energy consumption. In this paper, we introduce EnergyTrackr, an approach designed to detect energy regressions across multiple commits that can then be used to identify code anti-patterns potentially contributing to the increase of software energy consumption over time. We describe our empirical evaluation, including repository mining and source code analysis, made on 3,232 commits from three Java projects, and show the approach's ability to identify significant energy changes. We also highlight recurring anti-patterns such as missing early exits or costly dependency upgrades. We expect EnergyTrackr to assist developers in accurately monitoring energy regressions and improvements within their projects, identifying code anti-patterns, and helping them optimize their source code to reduce software energy consumption.

Code tours are interactive, onboarding documentation to guide developers through a codebase. Large Language Models (LLMs) can automatically synthesize code tours. Prior work on code tour generation has not studied developer experience or trust calibration when debugging unfamiliar codebases with code tours generated and evaluated by open-weight LLMs. This study surveys how the properties of components in open-weight LLM-authored code tours influence developers' experiences when debugging unfamiliar codebases. We built a pipeline that generated and evaluated code tours from real reproducible bugs. 26 developers with varying backgrounds participated in a user study. In total, 26 code tours were authored from real Java bugs mined from 2025 GitHub commits, with each tour independently judged by two different LLMs, resulting in 52 evaluated configurations. Participants thought aloud as they explored each tour. Three authors qualitatively coded the interviews to identify recurring themes. Developers generally preferred tours that scaled detail with the code length, avoided merely restating code, were easily scannable, and adopted a guiding tone. However, some preferences were mutually exclusive, such as the use of imperative mood. Stack traces were often insufficient to identify all steps developers found relevant. Developers also trusted descriptions they perceived as human-written more than those they believed were AI-generated. Finally, LLM-generated annotations of tour quality were unreliable: sycophancy, confabulation, and incoherence were pervasive. This work lays a basis for future research on fine-tuning open-weight models for code tour generation, personalizing generation to accommodate diverging preferences, selecting relevant steps beyond stack traces, calibrating users' trust to avoid both disuse and misuse, and improving open-weight LLMs' ability to be more trustworthy evaluators

While state-of-the-art large language models (LLMs) show great potential for automating various Behavioral-Driven Development (BDD) related tasks, such as test generation, smaller models depend on high-quality data, which are challenging to find in sufficient quantity. To address this challenge, we adapt the SELF-INSTRUCT method to generate a large synthetic dataset from a small set of human-written high-quality scenarios. We evaluate the impact of the initial seeded scenarios’ quality on the generated scenarios by generating two synthetic datasets: one from 175 high-quality seeds and one from 175 seeds that did not meet all quality criteria. We performed a qualitative analysis using state-of-the-art quality criteria and found that the quality of seeds does not significantly influence the generation of complete and essential scenarios. However, it impacts the scenarios’ capability to focus on a single action and outcome and their compliance with Gherkin syntactic rules. During our evaluation, we also found that while raters agreed on whether a scenario was of high quality or not, they often disagreed on individual criteria, indicating a need for quality criteria easier to apply in practice.

Context: Recent software engineering (SE) research has highlighted the need for sociotechnical research, implying a demand for customized psychometric scales. Objective: We define the concepts of technical and sociotechnical infrastructure in software engineering, and develop and validate a psychometric scale that measures attitudes toward them. Method: Grounded in theories of infrastructure, attitudes, and prior work on psychometric measurement, we defined the target constructs and generated scale items. The scale was administered to 225 software professionals and evaluated using a split sample. We conducted an exploratory factor analysis (EFA) on one half of the sample to uncover the underlying factor structure and performed a confirmatory factor analysis (CFA) on the other half to validate the structure. Further analyses with the whole sample assessed face, criterion-related, and discriminant validity. Results: EFA supported a two-factor structure (technical and sociotechnical infrastructure), accounting for 65

In software engineering, benchmarks are widely used to evaluate and compare the performance, functionality, and reliability of analysis tools. Despite the prevalence of benchmarks in areas such as databases, machine learning, and programming languages, there is a notable absence of publicly available benchmarks for REST APIs, a cornerstone of modern web-based systems. While existing research papers occasionally employ similar REST APIs in their evaluations, opportunistic API selection hampers comparison. Moreover, these studies often rely on API documentation and structural characteristics. Without a reliable benchmark, API data used in evaluations may be outdated or inaccurate, compromising reliability and reproducibility. Hence, this paper addresses a gap in the literature by providing a comprehensive and Public REST API Benchmark (PRAB), to be utilized by researchers in their evaluations. The benchmark contains documentation and structural characteristics of 60 publicly available REST APIs. First, we conduct a systematic mapping study to discover the available and public REST APIs that are utilized in the academic literature. Then, by analyzing the resulting APIs, we report their structural characteristics (e.g., routes, query parameters, HTTP methods, authentication). Finally, we provide their documentation (i.e., OpenAPI Specification, Postman Collection) in a publicly available GitHub repository, to help with future evaluations of REST API studies.

Background: Current state-of-the-art established that cognitive biases appear during code review. They significantly impact the creation of feedback and how developers interpret it. These biases can lead to illogical reasoning and decision-making, violating one of the main hypotheses supporting code review: developers' accurate and objective code evaluation. Objective: This paper explores harmful cases caused by cognitive biases during code review and potential solutions to avoid such cases or mitigate their effects. Method: We design several prototypes covering confirmation bias and decision fatigue. We rely on a developer-centered design approach by conducting usability tests and validating the prototype with a user experience questionnaire (UEQ) and participants' feedback. Results: Our interim findings show that some techniques could be implemented in existing code review tools as reviewers will accept them and help prevent behavior detrimental to code review. Conclusion: This work provides a first approach to treating cognitive bias in code review. The developed prototypes will evolve into fully functional tools, with an extensive evaluation with developers.

Android devices and related applications are increasingly prevalent in our daily routines. Furthermore, these technologies are being used for more than just connecting people around the world. Indeed, Android devices are more and more connected to external sensors or used as sensors, directly gathering data from their environment, which brings them closer to Cyber Physical Systems (CPS). When used for specific purposes such as health, Android devices and related applications can be life-critical (insulin pumps, heart monitoring, etc.), requiring guarantees specific to the application domain. Interestingly, when considering the technical security in domains related to operational technologies, we can see that many standards are available while not directly intended for Android applications. Other regulatory texts can also be valuable to drive an audit process, although they need more effort to reach technically testable requirements from the legal requirements they define. In particular, Android applications are developed using various device permissions (i.e., resource access), external libraries, etc. In this paper, we present A3S3 a tool to link requirements from industry standards and regulatory texts to Android features to drive a security audit. Following research in the cyber-security community, we suggest an approach based on static code analysis of Android applications to retrieve good and bad signals, denoting potential violations, related to non-functional requirements.

The software engineering community has increasingly recognized sustainability as a key research area. However, developers often have limited knowledge of effective strategies to reduce software energy consumption. To address this, we analyze energy consumption in software execution, aiming to raise developer awareness by linking energy consumption with each line of code. We rely on unit test executions to identify energy-intensive executions and manually analyze five hot and five cold spots to identify potentially energy-intensive source code constructs. Our findings suggest a link between the energy consumption of the source code and the number of objects' attributes created within that code. These results lay the groundwork for further analysis of the relationship between object instantiation and energy consumption in Java.

In this paper, we present FETT, a fault injection tool for educational and training purposes addressed to educators and students in cybersecurity. Our tool aims to analyze and inject vulnerabilities into existing Django web applications for educational purposes. Indeed, security education often relies on either abstract theoretical instruction or overly simplistic examples. This tool bridges the gap between theory and practice by modifying real web applications in a targeted, reproducible way. With its user-friendly interface and modular vulnerability injection, instructors can create challenges tailored to specific learning goals, while students engage directly with code that simulates production-level vulnerabilities. We evaluated FETT based on five publicly available GitHub projects and six student projects from the last three academic years (2022-2024). We successfully managed to efficiently inject vulnerabilities inspired by the OWASP top 10:2021 while keeping the core functionalities of the target application operational. The project is publicly available at https://gitlab.com/fabgilson/django-vulnerability-injector. A demonstration is available at https://youtu.be/ZYQs2vLzbyE.

For many years, Odoo, an open-source add-on-based platform offering an extensive range of functionalities, including Enterprise Resource Planning, has constantly expanded its scope, resulting in an increased complexity of its software. To cope with this evolution, Odoo has developed an integration testing system called tour execution, which executes predefined testing scenarios (i.e., tours) on the web user interface to test the integration between the front, back, and data layers. This paper reports our effort and experience in extending the tour system with fuzzing. Inspired by action research, we followed an iterative approach to devise FuzzE, a plugin for Odoo's tour system to create new tours. FuzzE was eventually developed in three iterations. Our results show that mutational fuzzing is the most effective approach when integrating with an existing testing infrastructure. We also reported one issue to the Odoo issue tracker. Finally, we present lessons learned from our endeavor, including the necessity to consider testability aspects earlier when developing web-based systems to help the fuzzing effort, and the difficulty faced when performing triage and root cause analysis on failing tours.

Mapping behaviours to the features they relate to is a prerequisite for variability-intensive systems (VIS) reverse engineering. Manually providing this whole mapping is labour-intensive. In black-box scenarios, only execution traces are available (e.g., process mining). In our previous work, we successfully experimented with variant-based mapping using supervised machine learning (ML) to identify the variants responsible of the production of a given execution trace, and demonstrated that recurrent neural networks (RNNs) work well (≥ 80% accuracy) when trained on datasets in which we label execution traces with variants. However, this mapping (i) may not scale to large VIS because of combinatorial explosion and (ii) makes the internal ML representation hard to understand. In this short paper, we discuss the design of a novel approach: feature-based mapping learning.

Onboarding new developers is a challenge for any software project. Addressing this challenge relies on human resources (e.g., having a senior developer write documentation or mentor the new developer). One promising solution is using annotated code tours. While this approach partially lifts the need for mentorship, it still requires a senior developer to write this interactive form of documentation. This paper argues that a Large Language Model (LLM) might help with this documentation process. Our approach is to record the stack trace between a failed test and a faulty method. We then extract code snippets from the methods in this stack trace using CodeQL, a static analysis tool and have them explained by gpt-3.5-turbo-1106, the LLM behind ChatGPT. Finally, we evaluate the quality of a sample of these generated tours using a checklist. We show that the automatic generation of code tours is feasible but has limitations like redundant and low-level explanations.

RESTful APIs are popular web services, requiring documentation to ease their comprehension, reusability and testing practices. The OpenAPI Specification (OAS) is a widely adopted and machine-readable format used to document such APIs. However, manually documenting RESTful APIs is a time-consuming and error-prone task, resulting in unavailable, incomplete, or imprecise documentation. As RESTful API testing tools require an OpenAPI specification as input, insufficient or informal documentation hampers testing quality. Recently, Large Language Models (LLMs) have demonstrated exceptional abilities to automate tasks based on their colossal training data. Accordingly, such capabilities could be utilized to assist the documentation and testing process of RESTful APIs. In this paper, we present RESTSpecIT, the first automated RESTful API specification inference and black-box testing approach leveraging LLMs. The approach requires minimal user input compared to state-of-the-art RESTful API inference and testing tools; Given an API name and an LLM key, HTTP requests are generated and mutated with data returned by the LLM. By sending the requests to the API endpoint, HTTP responses can be analyzed for inference and testing purposes. RESTSpecIT utilizes an in-context prompt masking strategy, requiring no model fine-tuning. Our evaluation demonstrates that RESTSpecIT is capable of: (1) inferring specifications with 85.05% of GET routes and 81.05% of query parameters found on average, (2) discovering undocumented and valid routes and parameters, and (3) uncovering server errors in RESTful APIs. Inferred specifications can also be used as testing tool inputs.

From business processes to course management, variability-intensive software systems (VIS) are now ubiquitous. One can configure these systems’ behaviour by activating options, e.g., to derive variants handling building permits across municipalities or implementing different functionalities (quizzes, forums) for a given course. These customisation facilities allow VIS to support distinct relevant customer requirements while taking advantage of reuse for common parts. Customisation thus allows realising both scope and scale economies. Behavioural differences amongst variants manifest themselves in event logs. To re-engineer this kind of system, one must know which variant(s) have produced which behaviour. Since variant information is barely present in logs, this paper supports this task by employing machine learning techniques to classify behaviours (event sequences) among variants. Specifically, we train Long Short Term Memory (LSTMs) and Gated Recurrent Units (GRUs) recurrent neural networks to relate event sequences with the variants they belong to on six different datasets issued from the configurable process and VIS domains. After having evaluated 20 different architectures of LSTM/GRU, our results demonstrate that it is possible to effectively learn the trace-to-variant mapping with high accuracy (at least 80% and up to 99% ) and at scale, i.e., identifying 50 variants using 5000+ traces for each variant.

Multimodal interaction has been adopted across various platforms and devices, with supporting tools enhancing the developer experience in developing Multimodal Interfaces (MMI). While traditionally, these tools faced challenges balancing expressiveness and usability, recent progress in Natural Language Processing tends to mitigate this rule. However, adding multimodal interaction still remains challenging, especially when integrating the voice modality, and MMIs remain to be better integrated into today's applications. To address these challenges, we introduce a Unity tool-based system named Ummi. Ummi allows developers to use their knowledge in Object-Oriented Programming to handle the expert knowledge required to create a MMI while allowing end users great flexibility in the natural language they can use to interact. Our contributions are: 1) the presentation of Ummi's architecture and its inherent concepts along with its open-source implementation, and 2) a successful evaluation of its usability for describing MMIs through the System Usability Scale questionnaire with twelve participants.

In today's organizations, software is mission-critical. However, the legacy of past decisions can make tasks related to artifacts increasingly inefficient or risky, creating debt. While most researchers and practitioners mainly focus on technical debt, some have investigated its social dimensions, known as social debt. We argue that organizations developing software need to tackle debt holistically, as it is intrinsically a socio-technical issue. In this short paper, we rely on a definition of socio-technical debt based on the existing literature to define Debt Stories: a tool based on the User Story format, that can help capture debt elements directly from the stakeholders involved in software development. A debt story includes information about the role of the stakeholder in the development process, the social or technical context, and the impact of the debt element on the different tasks performed by the stakeholder. We provide a first empirical evaluation of the usage of Debt Stories in an industrial context, demonstrating the relevance of Debt Stories to express and communicate socio-technical debt.

Researchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and for various platforms (e.g., desktop, web, or mobile applications). Such generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit-testing of libraries vs. system-testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved through the systematic execution of large-scale evaluations of different generators. However, the execution of such empirical evaluations is not trivial and requires a substantial effort to collect benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this paper, we present our JUnit Generation benchmarking infrastructure (JUGE) supporting generators (e.g., search-based, random-based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (e.g., validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, eight editions of a unit testing tool competition, co-located with the Search-Based Software Testing Workshop, have taken place and used and updated JUGE. As a result, an increasing amount of tools (over ten) from both academia and industry have been evaluated on JUGE, matured over the years, and allowed the identification of future research directions.