Background: Ensemble models play an important role in integrating multiple classifiers in a wide range of applications, such as medical diagnosis, sentiment analysis, and financial market trends. In Requirements Engineering (RE), automatic requirements classification can be improved by the utilization of these models. Aims: This paper analyses the performance metrics of voting and stacking ensemble models for requirements classification prediction. Moreover, a cross-dataset validation was performed for the meta-models generated using the stacking ensemble method. Methods: Some previously trained base models and two datasets of software requirements written in Spanish (translated PROMISE_exp and ReSpa dataset) were used to build the ensemble models. Results: The results indicate that the stacking model achieved a weighted F1-score of 0.828 using Support Vector Machine (SVM) and Multi-layer Perceptron (MLP) for translated PROMISE_exp dataset. For the ReSpa dataset, the stacking model achieved a weighted F1-score of 0.890 using Logistic Regression (LR). Conclusion: This study confirms a slight improvement in the performance of binary requirements classification using stacking ensemble methods over voting and most individual base models. Moreover, combining all models outperforms combinations that include only Shallow ML or DL models.
Background: Large language models (LLM) suffer from various forms of biases due to the biased datasets used to train the models. At the same time, human cognitive biases have an equal propensity to express themselves when using LLMs for software engineering tasks. Software testing is a critical phase of the software development life cycle. Confirmation bias is reported to have deteriorated software testing by designing more specification-consistent test cases compared to specificationinconsistent test cases. However, there is a lack of debiasing (mitigation) strategies in this regard. Aims: In this paper, first, we investigate whether the LLM model suffers from confirmation bias while performing software testing tasks. Second, we propose a vision of debasing confirmation bias in software testing via LLM. Method: We conducted an empirical study to detect confirmation bias by an LLM (ChatGPT4.0) in the design of functional test cases. Based on empirical findings, we used the analytical paradigm to design a multi-agent system. Results: We present a vision for debiasing confirmation bias in functional software testing by leveraging LLMs via a multi-agent approach. Conclusions: The proposed vision may improve the performance of LLMs in terms of reduced confirmation bias and serve as a debiasing technique for functional software testing.
Background: Modern software systems are expected to deliver high performance under a variety of different workloads. In order to automatically verify whether a system operates correctly under specific load conditions, load testing has become a widely adopted technique. As software systems evolve, their load requirements-such as performance thresholds and usage patterns-also change, necessitating updates to load tests. Aims: This study investigates the maintenance of load testing scripts to better understand how load requirements evolve and how these changes are reflected in the tests themselves. Method: We analyzed 35 open-source software (OSS) repositories that incorporate load testing. We examined the frequency and nature of load test updates. Results: Our analysis reveals that 45.7% of the studied projects do not update their load testing scripts after initial creation. However, a small subset of projects demonstrates extensive and ongoing maintenance of these scripts. Furthermore, we identified 20 distinct update types across 5 major categories of purposes for load testing script modifications. The most frequent update type is related to “Test Maintenance”, followed by “Test Scenario Modification.” Conclusions: Our findings suggest that load testing scripts are often left unmaintained over time in many projects. The updates, when performed, serve a wide range of purposes, with test maintenance being the most frequent.
Background: Systematic literature reviews (SLRs) require robust search strategies to ensure comprehensive coverage. Although database searches have traditionally been the primary method, snowballing has emerged as an effective alternative strategy in software engineering research. However, the success of snowballing heavily depends on the initial seed set's composition, particularly regarding diversity across authors, publication years, and venues. Objective: This study investigates how different diversity characteristics in seed set creation influence snowballing performance and effectiveness in identifying relevant literature. Method: We conducted replication studies of two existing SLRs, comparing their conventional seed set creation approaches with our diversity-driven methodology, where we systematically incorporated diversity characteristics into constructing the seed sets. Results: Our diversity-based approach demonstrated substantial improvements, with a precision of 0.019 (compared to 0.006 in the original), a relative recall of 0.97 (versus 0.921), and an F-measure of 0.0372 (improving from 0.0119). Conclusions: The empirical evidence suggests that incorporating diversity criteria in seed set creation enhances snowballing efficacy while maintaining comprehensive coverage of relevant literature. This approach offers a systematic and effective method for conducting snowballbased literature reviews in software engineering research.
[Background] Efficient code review is essential in industrial software development, but writing high-quality Pull Request (PR) descriptions aligned with core software engineering competencies remains challenging. [Aims] To investigate Large Language Models (LLMs) as a reliable PR description assistant, we examined two research problems: (1) how developer competencies relate to real-world PR quality, and (2) how developers perceive PR quality and their variants (LLM-Improved). [Methods] We adopted six software engineering competencies to prompt six LLMs to score 212,687 PRs from 82 open-source projects spanning three repository archetypes. From these, we selected the top six PRs for a controlled experiment with 38 software professionals, using a 6x6 Latin square design to measure preferences. The LLMs generated three variants of each original (O) PR: degraded (D), improved from the original (IO), and improved from the degraded (ID). We then compared these variants across LLMs for semantic, lexical, and stylistic similarity. [Results] Surprisingly, our controlled experiment with 38 software professionals showed that the ID variant was rated significantly higher than both the Original PR (O) and the LLM-Degraded (D) one. However, the IO variant was not significantly better, and participants complained about their verbosity and the "AI tone". Concerning generalizability, the six LLMs produced semantically similar PRs, but with high lexical and stylistic variation. [Conclusions] This suggests LLMs can enhance PR by providing missing structure but require guidance to retain human nuance. Overall, our findings support human-centered, model-aware integration of LLMs to strengthen competency-aligned code review.
