Background: Ensemble models play an important role in integrating multiple classifiers in a wide range of applications, such as medical diagnosis, sentiment analysis, and financial market trends. In Requirements Engineering (RE), automatic requirements classification can be improved by the utilization of these models. Aims: This paper analyses the performance metrics of voting and stacking ensemble models for requirements classification prediction. Moreover, a cross-dataset validation was performed for the meta-models generated using the stacking ensemble method. Methods: Some previously trained base models and two datasets of software requirements written in Spanish (translated PROMISE_exp and ReSpa dataset) were used to build the ensemble models. Results: The results indicate that the stacking model achieved a weighted F1-score of 0.828 using Support Vector Machine (SVM) and Multi-layer Perceptron (MLP) for translated PROMISE_exp dataset. For the ReSpa dataset, the stacking model achieved a weighted F1-score of 0.890 using Logistic Regression (LR). Conclusion: This study confirms a slight improvement in the performance of binary requirements classification using stacking ensemble methods over voting and most individual base models. Moreover, combining all models outperforms combinations that include only Shallow ML or DL models.
Background: Large language models (LLM) suffer from various forms of biases due to the biased datasets used to train the models. At the same time, human cognitive biases have an equal propensity to express themselves when using LLMs for software engineering tasks. Software testing is a critical phase of the software development life cycle. Confirmation bias is reported to have deteriorated software testing by designing more specification-consistent test cases compared to specificationinconsistent test cases. However, there is a lack of debiasing (mitigation) strategies in this regard. Aims: In this paper, first, we investigate whether the LLM model suffers from confirmation bias while performing software testing tasks. Second, we propose a vision of debasing confirmation bias in software testing via LLM. Method: We conducted an empirical study to detect confirmation bias by an LLM (ChatGPT4.0) in the design of functional test cases. Based on empirical findings, we used the analytical paradigm to design a multi-agent system. Results: We present a vision for debiasing confirmation bias in functional software testing by leveraging LLMs via a multi-agent approach. Conclusions: The proposed vision may improve the performance of LLMs in terms of reduced confirmation bias and serve as a debiasing technique for functional software testing.
Background: Modern software systems are expected to deliver high performance under a variety of different workloads. In order to automatically verify whether a system operates correctly under specific load conditions, load testing has become a widely adopted technique. As software systems evolve, their load requirements-such as performance thresholds and usage patterns-also change, necessitating updates to load tests. Aims: This study investigates the maintenance of load testing scripts to better understand how load requirements evolve and how these changes are reflected in the tests themselves. Method: We analyzed 35 open-source software (OSS) repositories that incorporate load testing. We examined the frequency and nature of load test updates. Results: Our analysis reveals that 45.7% of the studied projects do not update their load testing scripts after initial creation. However, a small subset of projects demonstrates extensive and ongoing maintenance of these scripts. Furthermore, we identified 20 distinct update types across 5 major categories of purposes for load testing script modifications. The most frequent update type is related to “Test Maintenance”, followed by “Test Scenario Modification.” Conclusions: Our findings suggest that load testing scripts are often left unmaintained over time in many projects. The updates, when performed, serve a wide range of purposes, with test maintenance being the most frequent.
Background: Systematic literature reviews (SLRs) require robust search strategies to ensure comprehensive coverage. Although database searches have traditionally been the primary method, snowballing has emerged as an effective alternative strategy in software engineering research. However, the success of snowballing heavily depends on the initial seed set's composition, particularly regarding diversity across authors, publication years, and venues. Objective: This study investigates how different diversity characteristics in seed set creation influence snowballing performance and effectiveness in identifying relevant literature. Method: We conducted replication studies of two existing SLRs, comparing their conventional seed set creation approaches with our diversity-driven methodology, where we systematically incorporated diversity characteristics into constructing the seed sets. Results: Our diversity-based approach demonstrated substantial improvements, with a precision of 0.019 (compared to 0.006 in the original), a relative recall of 0.97 (versus 0.921), and an F-measure of 0.0372 (improving from 0.0119). Conclusions: The empirical evidence suggests that incorporating diversity criteria in seed set creation enhances snowballing efficacy while maintaining comprehensive coverage of relevant literature. This approach offers a systematic and effective method for conducting snowballbased literature reviews in software engineering research.
[Background] Efficient code review is essential in industrial software development, but writing high-quality Pull Request (PR) descriptions aligned with core software engineering competencies remains challenging. [Aims] To investigate Large Language Models (LLMs) as a reliable PR description assistant, we examined two research problems: (1) how developer competencies relate to real-world PR quality, and (2) how developers perceive PR quality and their variants (LLM-Improved). [Methods] We adopted six software engineering competencies to prompt six LLMs to score 212,687 PRs from 82 open-source projects spanning three repository archetypes. From these, we selected the top six PRs for a controlled experiment with 38 software professionals, using a 6x6 Latin square design to measure preferences. The LLMs generated three variants of each original (O) PR: degraded (D), improved from the original (IO), and improved from the degraded (ID). We then compared these variants across LLMs for semantic, lexical, and stylistic similarity. [Results] Surprisingly, our controlled experiment with 38 software professionals showed that the ID variant was rated significantly higher than both the Original PR (O) and the LLM-Degraded (D) one. However, the IO variant was not significantly better, and participants complained about their verbosity and the "AI tone". Concerning generalizability, the six LLMs produced semantically similar PRs, but with high lexical and stylistic variation. [Conclusions] This suggests LLMs can enhance PR by providing missing structure but require guidance to retain human nuance. Overall, our findings support human-centered, model-aware integration of LLMs to strengthen competency-aligned code review.
Background: The rapid adoption of Large Language Models (LLMs) in software engineering, such as in customer service chatbots, has brought substantial benefits but also significant risks. Biased, inappropriate, or harmful responses may occur when integrating LLMs as commercial off-the-shelf (COTS) components into user-facing applications. Aims: This paper aims to present emerging results from an exploratory study that compares commercial guardrail frameworks, assessing their effectiveness in filtering inappropriate content during chat interactions. Method: We empirically evaluated three guardrail frameworks LLM Guard, Llama Guard, and OpenAI Moderation - using two datasets containing toxic and offensive content. Results: The frameworks achieved high accuracy for one dataset (over 90%) but underperformed in other metrics, indicating that toxic or dangerous content could still reach users in certain scenarios, such as deployment in chatbots. Conclusions: These findings $h$ ighlight $t$ he $n$ eed $f$ or further improvements in guardrail frameworks and provide insights for researchers and practitioners to support the selection of appropriate solutions for enhancing harm mitigation in LLMbased applications.
Background: In recent years, with the advancement of software engineering technologies and industry, Open Source Software (OSS) has become a mainstream model for software development and innovation. Increasingly, organizations and developers are adopting and customizing existing OSS to simplify and accelerate development processes. During OSS adoption, recommending suitable software based on user needs is crucial for enhancing development efficiency and addressing diverse requirements. However, the vast number and diversity of OSS make the recommendation task highly challenging. Despite progress in previous research, several issues remain, such as neglect of key software attributes, complexity in extracting multilingual features, and challenges of cold start and data sparsity. Aims: This paper presents AthenaRec, a large-scale OSS recommendation system comprising three core modules: Delphi, Argus, and Hestia. AthenaRec aims to recommend relevant and suitable software from a vast OSS based on user needs. Method: Specifically, Delphi first analyzes user queries to identify intention; Argus employs a heterogeneous ensemble recall approach to retrieve a large set of candidate software relevant to the identified intention; finally, Hestia adopts a two-stage deep ranking strategy. It performs coarse ranking by integrating multilingual modeling with contrastive learning, followed by fine ranking with a large language model, augmented by retrieval-augmented generation to incorporate external evidence. To evaluate the effectiveness of AthenaRec, we use a query dataset from real application scenarios. Results: Experimental results demonstrate that, on the test set of 7,500 queries, AthenaRec achieves superior recommendation performance, with Hits@20, MAP@20, NDCG@20, and MRR scores of 98.27%, 95.60%, 95.05%, and 92.92%, respectively. On average, AthenaRec outperforms other top methods by 10.9% across all evaluation metrics. Conclusions: Additionally, we develop a Visual Studio Code (VSCode) plugin based on AthenaRec, which can be accessed via URL. We intend for this research to provide a reference for software developers, advancing the efficiency and accuracy of OSS recommendation.
Background. Test smells can pose difficulties during testing activities, such as poor maintainability, non-deterministic behavior, and incomplete verification. Existing research has extensively addressed test smells in automated software tests, but little attention has been paid to smells in natural language tests. While some research has attempted to catalog such test smells, there is a lack of investigation into their impact on the effectiveness of test cases. Aims. In this paper, we conduct a controlled experiment with 30 participants from academia and industry to examine the impact of test smells in manual test descriptions. Method. Specifically, we analyze whether the presence of two test smells, Ambiguous Test and Eager Action, result in (1) increased test execution time, (2) a higher number of steps needed to complete the tests, and (3) high divergency on the perceived success of the tests outcomes. Results. Our findings reveal that an Ambiguous Test can increase execution time by up to five times and screen flow by up to seven times. In addition, if the Eager Actions are dependent on one another, there is no increase in execution time and screen flow. Conclusions. It highlights the need for better design of manual test descriptions to improve clarity, consistency, and performance execution.
Background: With limited job openings and growing selectivity in tech, understanding the skills hiring managers expect from entry-level Software Engineering candidates is essential. While job postings emphasize technical abilities, success often hinges on non-technical skills, as interpersonal and adaptive abilities. Yet, mismatches between candidate profiles and employer expectations lead to inefficient hiring and missed opportunities. Aims: We investigated which skills hiring managers expect from early-career Software Engineering (SE) candidates and how those skills are described in job postings. Method: To achieve this goal, we conducted an exploratory case study in Hewlett Packard Enterprise (HPE), a global technology company. We interviewed 12 managers, collected seven HPE job postings related to entry-level positions in Software Engineering-related roles, and employed mixed methods to analyze the data. Results: Our findings show that hiring managers place strong emphasis on non-technical expectations when evaluating entry-level candidates. They commonly believe that technical competencies can be taught on the job, provided the candidate demonstrates the right nontechnical attributes. The non-technical expectations included adaptability, proactivity, dependability, self-direction, passion, collaboration, communication, problem solving, courage, selfawareness, self-confidence, willingness to learn, and having a process-oriented mindset. Conclusions: To secure their first job, Software Engineering students should invest not only in technical competencies but also in developing non-technical skills. Our findings show that hiring managers prioritize these skills, even though they are often underrepresented in job postings. Making such expectations more visible could help bridge the gap between candidates and employers, leading to more efficient and effective hiring.
Background: Accurate classification of nonfunctional requirements (NFRs) is essential for aligning stakeholder expectations with system design and ensuring software quality. While transformer-based models such as PRCBERT and NORBERT have achieved high performance in supervised settings, their generalizability across diverse sources of requirements remains largely unexplored. In practice, requirements originate from heterogeneous platforms, ranging from structured specification documents to informal developer discussions on forums like Stack Overflow. Aim: This study provides the first comprehensive, bidirectional cross-dataset evaluation of domain-specific, embedding-based, and promptbased large language models (LLMs) for NFR classification across two contrasting platforms: PROMISE (structured) and NFR-SO (informal). Method: We evaluate domain-specific finetuned models, sentence embedding models, and prompt-based LLMs (including GPT-4o) in both zero-shot and few-shot settings. Performance is measured both in-domain and in crossplatform transfer scenarios to assess generalization with minimal or no labeled data. Results: Domain-specific fine-tuned models, although effective in-domain, exhibit substantial performance degradation when transferred across platforms. In contrast, LLMs-particularly GPT-4o in few-shot mode-consistently outperform other approaches in cross-platform scenarios, achieving strong generalization with minimal labeled data. In zero-shot mode, GPT-4o also demonstrates robust performance without any supervision. Conclusions: Traditional supervised models face limitations in cross-platform NFR classification. Prompt-based LLMs offer a scalable, low-supervision solution for diverse requirement sources.
Background: Developers often rely on statement coverage to assess test suite quality. However, statement coverage alone may only lead to 10% fault detection, necessitating more rigorous approaches. While mutation testing is effective, its execution and human analysis costs remain high. Identifying covered statements that are not checked by oracles (e.g., assertions) offers a cost-effective alternative; however, the lack of empirical evidence for selecting the appropriate Oracle Gap Calculation Approach (OGCA) prevents developers from making informed choices. Aims: This knowledge-seeking study compares oracle gap characteristics determined by different OGCAs to assist developers in choosing the most valuable approach for their use cases. Method: Using mixed-method empirical analysis, we conduct an in-depth evaluation of the oracle gaps produced using three OGCAs: Checked Coverage using a Dynamic Slicer (CCDS), Checked Coverage using an Observational Slicer (CCOS), and Pseudo-Tested Statement Identification (PTSI). Across 30 Java classes from six open-source projects, we report on a quantitative evaluation of gap prominence, distribution, fault detection correlation and execution times, as well as results from a qualitative manual inspection of the statement types found in the oracle gaps. Results: The qualitative analysis showed data-loading statements, iteration statements and output updates to be most prominent in the oracle gaps. PTSI identified the oracle gaps with the lowest median mutation score (0.32), highlighting areas requiring more fault detection improvement compared to CCDS (0.76) and CCOS (0.50). PTSI also had the shortest median execution time (19.9 seconds), far quicker than both CCDS (273.2 seconds) and CCOS (5957.1 seconds). Conclusions: PTSI quickly reveals the priority testing areas for improved fault detection, making it an effective OGCA for developers to identify where tests fall short.
Background: The sustainability of Open Source Software (OSS) projects relies on attracting and retaining contributors. Interpersonal challenges, whether experienced or witnessed, can discourage participation, alter behavior, or drive contributors away. Aims: This study examines how interpersonal challenges in OSS communities persist over time and explores their behavioral consequences on contributors' decisions and actions. Method: We analyze data from two large GitHub Open Source surveys conducted in 2017 (n=5,495) and 2024 (n=8,452), evaluating changes in reported interpersonal challenges (RQ1) and differential consequences of exposure between 2017 and 2024 (RQ2). Results: Our findings reveal a significant increase in reported interpersonal challenges in 2024 compared to 2017. Contributors more frequently reported severe challenges such as threats of violence, impersonation, sustained harassment, stalking, and doxxing. The behavioral impact has shifted: experiencing rudeness, stalking, and name-calling became strongly linked to stopping contributions, adopting pseudonyms, working privately, and avoiding offline events. Witnessing harmful behaviors like name-calling and impersonation also became stronger predictors of working privately or advocating for Codes of Conduct. These trends show toxicity is not only more pervasive but increasingly damaging to OSS participation and community health. Conclusions: Results highlight a concerning rise in interpersonal challenges within OSS communities, with rudeness emerging as the most impactful. The growing influence of toxic behaviors on contributors' decisions to withdraw, conceal identities, isolate collaboration, and avoid offline engagement underscores the urgent need for stronger, proactive community support. Sustaining healthy OSS projects requires both technical excellence and deliberate investment in social infrastructure to foster respectful collaboration spaces.
Background] Applications of generative artificial intelligence are being proposed at a rapid pace to support various software engineering tasks. Although versatile, the performance of the tools depends on multiple factors that are not always visible to the users, and they tend to camouflage their failure points (i.e., to “hallucinate”). In programming, the existing literature suggests that they shift the effort from writing to reading, comprehending, evaluating, and repairing generated code. The tools can also enable outsourcing these efforts, even when it might be unwise. [Aims] The broad aim of this research is to investigate the interaction between programmers and code generation tools to understand how the tools support the needs of software practitioners. The specific approach of this research is to examine how, when, for what, to what extent, and to what effect programmers read, comprehend, evaluate, repair, and trust AI-generated code, with the goal of theory building for predicting how these will change over time. [Method] Research is expected to consist mainly of qualitative and quantitative analyses of programmers' behaviour and trust in realistic contexts. Research methods include practitioner surveys and interviews, a scoping review, and observational field studies. [Results] The expected results include descriptions of the ways in which programmers evaluate AI-generated code in different scenarios, and a theory on how programmers' evaluation-related behaviour and trust change as they become more skilled users of the technology and/or as the technology matures, with the broader goal of supporting future research on the nature of generative AI assistance.
Background: As Large Language Models (LLMs) reshape software development across industries, they also reshape the associated threat landscape. Traditional threat modeling methods, which assume predictable system behavior, struggle to accommodate the inherent nondeterminism of LLMs. Paradoxically, LLMs themselves offer capabilities, such as pattern recognition, natural language understanding, and semi-structured reasoning, that can support the automation of threat elicitation and mitigation. Aims: This research project, ThreMoLIA, aims to design, develop, and empirically evaluate a threat modeling tool that leverages LLMs to assist practitioners in identifying and analyzing security threats in LLM-integrated applications (LIAs). Method: To this end, we apply a mixed-methods exploratory case study to define and validate threat modeling metrics, and a comparative case study to evaluate the ThreMoLIA tool against existing threat modeling practices. Results: The current prototype of the ThreMoLIA tool uses cloud or local models. We have established, and partiallyvalidated, a measurement framework and a benchmark for the tool evaluation. Conclusions: The project is conducted in close collaboration with industry and contributes to the ESEM community by advancing Security-by-Design practices and sharing reproducible artifacts such as metrics, benchmarks, and threat models.
Background: Testing is a crucial activity in software development, but over time, test suites can accumulate test debt-such as flaky, redundant, or overly complex tests. As a subset of technical debt, test debt represents the long-term costs of suboptimal testing practices. Technical debt (TD) refers to the choice of quick solutions in software development instead of more robust and long-term approaches and it has a negative impact on software projects and has attracted significant interest from both industry and academia. With the increasing complexity of modern software, automated testing is essential for maintaining quality, speed, and customer satisfaction. Like other types of technical debt, test debt silently erodes productivity, increases maintenance costs, and reduces reliability in the software development lifecycle. Despite its impact, identifying and managing test debt remains challenging. Developers often face unclear test failures, redundant checks, and a lack of tools to understand root causes or determine what needs to be fixed. Many researchers have been working on leveraging machine learning to better identify and then reduce technical debt in software development. However, these methods often lack transparency-leaving practitioners unsure of how or why decisions are made. This gap between machine learning and actionable insight highlights the need for more interpretable approaches. Aims: This research aims to address this gap by applying explainable artificial intelligence (XAI) to create a tool that helps software teams detect and interpret test debt in automated testing environments. Method: XAI methods improve traceability, trust, and transparency, enabling developers to better comprehend test behaviour. Expected Results: By providing explanations, teams are empowered to have a better understanding and decision-making about test maintenance and improvement. By improving the visibility of the interpretability of test debt, this research contributes to more resilient testing, better-informed software engineers, and more sustainable software systems.
Background: Security is incorporated late in the Software Development Life Cycle (SDLC), whereas early activities supporting developers in understanding and implementing security measures are difficult to integrate. Aims: The project focuses on empowering developers with tools and practices to seamlessly integrate security and understand the role automation plays in it. Method: During the project, we perform several industrial empirical studies, qualitative and quantitative, under the Design Science Research paradigm. Results: Our studies supporting developers to secure their software supply chain show a positive stance despite low adoption of artifacts such as SBOM and VEX. Other efforts, embedding security in GUIbased testing, are showing promising results. Conclusion: The project covers a broad spectrum of development activities that can be enhanced from a security perspective. Initial results show that despite developers' interest, adoption is limited.
Background: Cognitive biases are systematic errors in reasoning that can lead to inaccurate decision-making across all areas of software production, regardless of the domain, programming language, or development method. Given the central role of software across all sectors, such biases can result in largescale inefficiencies, delays, and increased costs. Goal: While prior software engineering (SE) research predominantly focuses on specific tasks and quantitative methods, the vision of this paper is to study qualitatively how cognitive biases emerge, persist, and can be mitigated to improve decision-making throughout the Software Development Life Cycle (SDLC). Method: This vision uniquely applies the concept of reconception as a theoretical lens to explore how software professionals' cognitive models influence their management of dialectical oppositions, such as project velocity vs product quality and open-source vs proprietary control. Utilising Socio-Technical Grounded Theory (STGT) and dialectical inquiry, it examines how reconceiving these oppositions affects cognitive models and decision-making processes. Expected Outcome: A broad and high-level theoretical framework that explains how selected cognitive biases influence SE decision-making across the SDLC. The framework is developed through a two-phase STGT process: identifying salient bias categories in the first phase and focusing on the role of specific biases in shaping decision patterns in the second. This highlevel theory will open new research opportunities for future investigations into bias-informed decision-making in SE. Conclusion: This theoretical framework will support the development of empirically grounded debiasing strategies for more reliable decision-making in software engineering.
Background: Software migration across programming languages is a critical yet labor-intensive task, often requiring deep code understanding and manual intervention. Aims: In this study, we aim to develop a fully automated agent for end-to-end code translation and validation. Method: First, we generate code comments from Java source code using various large language models (LLMs) to enhance code comprehension and facilitate cross-language translation. Second, leveraging these AI-generated comments, we automatically generate equivalent C# code, demonstrating the potential of AI in software migration and interoperability. Third, we complete both Java and generated C# code and prepare them to execute. Fourth, we apply automated unit testing to assess functional correctness and ensure the reliability of AI-generated code. Results: Our results show that a fully automated LLM agent may effectively bridge programming languages with minimal human input. This approach opens new possibilities for scalable, AIdriven software modernization and cross-platform development. Conclusions: We recommend that such an LLM agent should be used to support human experts during the generation of reliable and correct code.
Background: Review Comment Generators (RCGs) are models trained to automate code review tasks. Prior work shows that RCGs can generate review comments to initiate discussion threads; however, their ability to interact with author responses is unclear. This can be especially problematic if RCGs pose interrogative comments, i.e., comments that ask questions of other review participants. Aims: We set out to study the prevalence of RCG-generated interrogative code review comments, their similarity with the interrogative comments of humans, and the predictability of the generation of interrogative comments. Method: We study three task-specific RCGs and three RCGs based on Large Language Models (LLMs) on data from the Gerrit project using quantitative and qualitative methods. Results: We find that RCGs: (1) generate interrogative comments at a rate of 15.6%-65.26%; (2) differ from humans in generating such comments, which can stifle conversations if RCGs dissuade human reviewers from commenting deeply; and (3) produce interrogative comments with low predictability. Finally, we find that (4) the interrogative comments posed by LLM-based RCGs can differ even more substantially from human behaviour than those of task-specific RCGs. For example, the studied LLM-based RCGs pose rhetorical questions 3.16% of the time, whereas human-submitted interrogative comments pose rhetorical questions 8.74% of the time. Conclusions: Our results suggest that neither task-specific nor LLM-based RCGs can replace human reviewers yet; however, we note opportunities for synergies. For example, RCGs tend to raise pertinent questions about exception handling of common APIs more frequently than human reviewers. Putting greater emphasis on technical comments generated by RCGs (rather than conversational ones, such as interrogative ones) will likely improve their perceived usefulness.
Background: To ensure the quality, dependability, and optimal functioning of software systems, software defect detection (SDD) is a crucial component of software development. Conventional techniques frequently depend on single-modal data sources, which might restrict the range and efficiency of fault identification. Aim: This research investigates the importance of AI-based multimodality, which combines and examines a range of data sources, including source code, design documents, execution logs, and test results. Multimodal frameworks can detect correlations, process complex and diverse data sources, and offer a comprehensive knowledge of software behaviour by utilising artificial intelligence (AI). This capability allows for more comprehensive and precise defect detection at different levels of software development, from design and implementation to testing and deployment. Moreover, AI-based multimodality enables proactive defect prevention techniques, strengthens fault prediction, and advances root cause investigation. Methods: In this context, this study demonstrates how multimodal approaches can revolutionise SDD by tackling the drawbacks of unimodal approaches in the software industry. Results: In addition to demonstrating its effectiveness in comparison to conventional methodologies, it analyses the difficulties in implementing AI-based SDD using multimodality. Conclusion: Further, this paper highlights the implications of AI-based multimodality for producing software systems that are dependable, efficient, and of high quality.
