Repository-level coding benchmarks typically measure progress in model capability by comparing the resolved rates of later and earlier models. However, this focus overlooks whether the non-functional quality of their generated patches has also changed across model generations. This study investigates whether later models produce functionally correct patches with better non-functional characteristics than earlier models on comparable repository-level repair tasks. We conducted two case studies involving four Claude and DeepSeek models on SWE-bench Lite. Using the same SWE-agent functional repair setting, we evaluated the generated patches with CodeQL, CodeScene, CPU time, and peak memory. Our primary analysis compared the models on commonly resolved instances. The static analysis results showed that most CodeQL paired differences were zero and that no CodeQL or CodeScene comparison remained significant after Holm correction. CPU time differences were small and inconsistent across model families, while peak memory usage was slightly higher for the later models under the benchmark test workload, with small absolute differences. Differences in individual CodeQL rules and CodeScene categories varied across model families and did not survive multiple-comparison correction. Overall, later models resolved more instances but showed no consistent improvement in the measured non-functional indicators on tasks solved by both models. Through this study, we hope to encourage a more comprehensive evaluation of models' practical software engineering capabilities.

Large language models have shown promising capabilities in code refactoring, but existing approaches remain limited to method-level tasks. In this paper, we envision LLM-based refactoring as a continuous component of software maintenance rather than a tool invoked only for occasional manual refactoring. Under this vision, AI agents continuously monitor, evaluate, and improve codebases against explicit and evolving notions of software quality. We present a roadmap organized around five dimensions: the multi-objective optimization problem, quality definition and evaluation, multi-timescale integration of heterogeneous signals, architecture and design pattern, and trust in autonomous refactoring. We further identify integration into continuous delivery pipelines and cost considerations as cross-cutting concerns. For each dimension, we analyze the underlying challenges and pose open research questions. These dimensions define a research agenda for advancing autonomous refactoring from isolated code improvements to system-level quality maintenance.

In recent years, large language models have been widely integrated into software engineering workflows, supporting tasks like code generation. While prior evaluations focus on functional correctness, there is still a limited understanding of the non-functional quality characteristics of generated code.Guided by the ISO/IEC 25010 quality model, this study adopts a multi-methods approach comprising three complementary elements: a literature review of 109 papers, two industry workshops with practitioners from multiple organizations, and an empirical analysis of patching real-world software issues using three LLMs. Motivated by insights from both the literature and practitioners, the empirical study examined the quality of generated patches regarding security, maintainability, and performance efficiency, which were identified as critical code-level quality attributes.Our results indicate that existing research primarily emphasizes security, performance efficiency, and maintainability, while other quality attributes are understudied. In contrast, practitioners prioritize maintainability and readability, warning that generated code may accelerate the accumulation of technical debt. The empirical evaluation demonstrates the instability of optimizing NFQCs through prompts in practical software engineering settings.Overall, our findings expose a misalignment between academic focus, industry priorities, and observed model behavior, highlighting the need to integrate quality assurance mechanisms into LLM code generation pipelines to ensure that future generated code not only passes tests but truly passes with quality.

Generating UML sequence diagrams from natural language requirements is a promising use case for large language models (LLMs), but existing studies often rely on proprietary models and manual evaluation, reducing reproducibility and generalizability. This paper proposes a fully automated, metric-driven evaluation framework for sequence diagram generation, covering semantic correctness, structural fidelity, and annotation quality. We benchmark a diverse set of open-source models across several sizes and families, including Qwen2.5, Qwen3, Mixtral, LLaMA3, Mistral, DeepSeek, and Nemotron, using a curated industrial dataset. Our findings indicate that architecture, instruction tuning, and training data composition drive performance differences, frequently outweighing scale. We also compare few-shot prompting and retrieval-augmented generation (RAG). Corpus-level normalization improves RAG performance, increasing the mean overall score from 0.34 with raw RAG to 0.43 with normalized RAG.

The role of regression testing in software testing is crucial as it ensures that any new modifications do not disrupt the existing functionality and behaviour of the software system. The desired outcome is for regression tests to yield identical results without any modifications made to the system being tested. In practice, however, the presence of Flaky Tests introduces non-deterministic behaviour and undermines the reliability of regression testing results. In this paper, we propose an LLM-based approach for identifying the root cause of flaky tests in C++ projects at the code level, with the intention of assisting developers in debugging and resolving them more efficiently. We compile a comprehensive collection of C++ project flaky tests sourced from GitHub repositories. We fine-tune Mistral-7b, Llama2-7b and CodeLlama-7b models on the C++ dataset and an existing Java dataset and evaluate the performance in terms of precision, recall, accuracy, and F1 score. We assess the performance of the models across various datasets and offer recommendations for both research and industry applications. The results indicate that our models exhibit varying performance on the C++ dataset, while their performance is comparable to that of the Java dataset. The Mistral-7b surpasses the other two models regarding all metrics, achieving a score of 1. Our results demonstrate the exceptional capability of LLMs to accurately classify flakiness in C++ and Java projects, providing a promising approach to enhance the efficiency of debugging flaky tests in practice.

The importance of the team, its internal dynamics, and its performance are widely recognized within the software engineering community. While popular frameworks identify wholeness, stability over time, and smallness as important factors, they offer little guidance on how to form teams that achieve these three characteristics. The objective of this study is to investigate how these team characteristics interact in large-scale software development contexts, particularly focusing on the impact of stable and dynamic teaming approaches. This was done through a multivocal study of literature, followed by individual semi-structured interviews with 19 engineers from two companies and validation workshops with an additional two companies from unrelated industry segments. The study results show that the question of stable versus dynamic approaches to forming software engineering teams is largely unaddressed in industry, with stable teams representing a habitual default option. Meanwhile, both stable and dynamic teams clearly have respective strengths and weaknesses, calling for careful consideration of the most suitable approach in any given situation. To support such consideration, this paper presents a model of how team stability, wholeness, and smallness interact. This model is found relevant, accurate, generalizable, and useful by practitioners.

The characteristics of the test environment are of vital importance to its ability to support the organizations testing objectives. This paper seeks to address the need for a structured and reliable approach, which can be used by companies and other organizations to optimize their test environments in each individual case. The reported study included a series of interviews with 30 individuals, a series of focus groups with in total 31 individuals and a cross‐company workshop with 30 participants from five large‐scale companies, operating in different industry segments. The study resulted in a list of success factors, including not only characteristics and capabilities existing within a test environment (intrinsic success factors) but also properties not inherent to the test environment, but still vital for a successfully implemented test environment (extrinsic success factors). This distinction is important, as the root causes differ and as addressing them requires distinct approaches—not only of technology but also of organization, communication and collaboration. We find that successful implementations of test environments for large‐scale software systems depend primarily on how they support the company's business strategy, test organization and product testability (extrinsic success factors). Based on this, test environments can then be optimized to improve test environment capabilities, usability and stability (intrinsic success factors). The list of intrinsic and extrinsic success factors was well received by all five companies included in the study, supporting that the intrinsic and extrinsic success factors for test environments can be applied to a large segment of the software industry.

Contribution: This article identifies the participation of external stakeholders as a key contributing factor for positive outcomes in project-based software engineering courses. A model for overlapping virtuous circles of lasting positive impact on both stakeholders and students from such courses is proposed. Background: Project-based courses are widespread in software engineering education, and there are numerous designs for such courses presented in literature. It is found that the needs and motivations of external stakeholders, from industry and government sectors, in these courses has received limited attention in related work. Intended Outcomes: A course design that prepares students for graduate level studies and professional life, through close proximity to external stakeholders in a highly realistic setting, working on “live” projects. Application Design: Building on a long tradition of university-industry collaboration dating back to 1977, as well as findings in related work, students are assigned to live projects proposed by external stakeholders from industry and government, working in close proximity with their respective stakeholders throughout the project. The course places great emphasis on coaching over instruction, treating the many unforeseen challenges of such projects as a valuable part of the learning experience. Findings: Based on interviews with stakeholders and students, it is found that stakeholder and student outcomes are interdependent and build upon one another, and that positive outcomes for both groups are necessary for the sustainability of the course over multiple iterations.

Context: Exploratory testing plays an important role in the continuous integration and delivery pipelines of large-scale software systems, but a holistic and structured approach is needed to realize efficient and effective exploratory testing. Objective: This paper seeks to address the need for a structured and reliable approach by providing a tangible model, supporting practitioners in the industry to optimize exploratory testing in each individual case. Method: The reported study includes interviews, group interviews and workshops with representatives from six companies, all multi-national organizations with more than 2,000 employees. Results: The ExET model (Excellence in Exploratory Testing) is presented. It is shown that the ExET model allows companies to identify and visualize strengths and improvement areas. The model is based on a set of key factors that have been shown to enable efficient and effective exploratory testing of large-scale software systems, grouped into four themes: "The testers' knowledge, experience and personality", "Purpose and scope", "Ways of working" and "Recording and reporting". Conclusions: The validation of the ExET model showed that the model is novel, actionable and useful in practice, showing companies what they should prioritize in order to enable efficient and effective exploratory testing in their organization. (C) 2021 Elsevier Inc. All rights reserved.

Continuous integration and delivery are well-established paradigms in the software development community. With these continuous practices come many challenges; while some of these challenges are immediate and well documented in literature, others may be revealed only after sustained application of these practices in large-scale and complex contexts. Based on researcher observations and interviews with 22 senior professionals from four companies, all with significant but varied experiences of continuous practices, we present a set of underreported challenges with continuous practices observed in multiple industry settings. Through a grounded theory approach we construct the Tapco model — Test Automation Progression in Continuous Practices — identifying two distinct ways in which companies progress towards continuous delivery, onto which the studied cases are mapped. This model is then validated by presenting it to the four studied companies, operating in disparate industry segments, and to three additional industry cases, letting them evaluate its relevancy, accuracy and novelty. We find that the model provides industry professionals with essential guidance on how to avoid common pitfalls, as well as an understanding of their causes and possible remediation.

Based on multiple series of interviews and workshops, this paper presents the MaLET model – a representation of the typical evolution path for companies who successfully adopt exploratory testing. The model provides a step-by-step approach to systematically improve exploratory testing over time, and shows how and why capabilities may regress. The MaLET model was validated through a series of interviews with 20 interviewees from eight case study companies in separate industry segments. The interviews also revealed examples on improvement initiatives that had failed in the companies, showing that improvement initiatives tend to fail if they are not planned in an order corresponding to the maturity levels in the model. The MaLET model was well received by the interviewees during the validation, describing the model as sound, relevant and useful in practice.

Mob programming is an emergent practice in the industry, attracting increasing attention from the research community. Superficially a simple concept – similar to pair programming, but with more people – many of the same concerns arise as with pair programming, only more so: can it truly be efficient, is it for everybody, and what are the benefits and risks involved? In this paper we show that while research interest in this topic is increasing, much of the published literature to date constitutes experience reports, with a number of implicit differences in how mob programming is practiced. To transform mob programming from an area of avant-garde experimentation into a well understood practice in the toolbox of mainstream software engineering, such implicit differences need to be exposed, analyzed and documented, and the contextual enabling factors must be investigated. In this paper we take an important step in that direction by identifying variations in the practice and providing concrete guidance for creating an organizational and social environment where mob programming can be practiced both effectively and safely.

This paper identifies three improvement areas related to system design and architecture, where an organization can change to better support continuous integration and continuous delivery: "The product's architecture", "Ways to work with system design and architecture", and "The role of the architect". The three improvement areas are based on a literature review, two series of interviews and a cross-company workshop with three case-study companies. Furthermore, the paper proposes three actionable strategies corresponding to the three identified improvement areas: "Systems with a modular and loosely coupled architecture", "A balanced approach where system design and architecture is focused on the system's most important characteristics", and "Architects shifting perspective from control to facilitation".

Continuous integration is believed by many to improve software quality, including cyclomatic complexity. In this paper an exploratory study investigates the relationship between continuous integration behavior among developers and the cyclomatic complexity - particularly at critical levels - of the source code they commit, as a correct understanding of the consequences of a software development practice, particularly one as popular as continuous integration, is of crucial importance to industry professionals when deciding on improvement efforts. It is found that developer behavior differs significantly, as does their knowledge and awareness of cyclomatic complexity as a concept. We find that even though large commits ("big bangs") contribute proportionately slightly more to cyclomatic complexity, the effect is much less pronounced for critical cyclomatic complexity. It is also found that complex code tends to overlap with high levels of change activity, and that some developers deliberately change their integration behavior in the face of such hotspots, as a strategy to avoid merge conflicts.

Based on 25 interviews with participants from four case study companies that develop large‐scale software embedded systems, this paper presents the Test Activity Stakeholders (TAS) model. The TAS model shows how the continuous integration and delivery pipeline can be designed to include test activities that support four stakeholder interests: “Check changes,” “Secure stability,” “Measure progress,” and “Verify compliance.” The model is developed to show how each of the stakeholder interests are best supported by unit/component tests or system tests, by automated testing or manual testing and by tests executed in simulated environments or on real hardware. The TAS model may serve as a starting point for companies when evaluating and designing their continuous integration and delivery pipeline. The validation of the TAS model included twelve individuals from three case study companies. The validation showed that the TAS model is actionable and useful in practice, enabling the identification of stakeholders and showing where improvement efforts should be focused in order to support all stakeholder interests in the continuous integration and delivery pipeline.

Based on interviews with 20 interviewees from four case study companies, this paper presents a list of key factors that enable efficient and effective exploratory testing of large-scale software systems. The nine factors are grouped into four themes: “The testers’ knowledge, experience and personality”, “Purpose and scope”, “Ways of working” and “Recording and reporting”. According to the interviewees, exploratory testing is a more creative way to work for the testers, and was therefore considered to make better use of the testers. Exploratory testing was also described as a good way to test system-wide and to test large-scale systems, especially exploratory testing with an end-user perspective. The identified key factors were confirmed by a series of follow-up interviews with the 20 interviewees and a cross-company workshop with 14 participants. This strengthens the generalizability of the findings, supporting that the list of key factors can be applied to projects in a large segment of the software industry. This paper also presents the results from a systematic literature review including 129 publications related to exploratory testing. No publication were found that summarizes the key factors that enable efficient and effective exploratory testing, which supports the novelty of the findings.

The popular agile practices of continuous integration and delivery stress the rapid and frequent production of release candidates and evaluation of those release candidates, respectively. Particularly in the case of very large software systems and highly variable systems, these aspirations can come into direct conflict with the need for both thorough and extensive testing of the system in order to build the highest possible confidence in the release candidate. There are multiple strategies to mitigate this conflict, from throwing more resources at the problem to avoiding end-to-end scenario tests in favor of lower-level unit or component tests. Selecting the most valuable tests to execute at any given time, however, plays a critical role in this context: repeating the same static test scope over and over again is a waste that large development projects can ill afford. While a number of alternatives for dynamic test case selection exist - alternatives that may be used interchangeably or even in tandem - many require analysis of large quantities of in situ real-time data in the form of trace links. Generating and analyzing such data is a recognized challenge in industry. In this chapter we investigate one approach to the problem, based on the Eiffel framework for continuous integration and delivery.

Based on interviews with 20 developers from two case study companies that develop large-scale software-intensive embedded systems, this paper presents twelve factors that affect how often developers commit software to the mainline. The twelve factors are grouped into four themes: "Activity planning and execution", "System thinking", "Speed" and "Confidence through test activities". Based on the interview results and a literature study we present the EMFIS model, which allows companies to explicate a representation of the organization's current situation regarding continuous integration impediments, and visualizes what the organization must focus on in order to enable more frequent integration of software. The model is used to perform an assessment of the twelve factors, where the ratings from participants representing the developers are summarized separately from ratings from participants representing the enablers (responsible for processes, development tools, test environments etc.). The EMFIS model has been validated in workshops and interviews, which in total included 46 individuals in five case study companies. The model was well received during the validation, and was appreciated for its simplicity and its ability to show differences in rating between developers and enablers.

Chapter 16 Dynamic Test Case Selection in Continuous Integration Test Result Analysis using the Eiffel Framework Daniel Ståhl, Daniel StåhlSearch for more papers by this authorJan Bosch, Jan BoschSearch for more papers by this author Daniel Ståhl, Daniel StåhlSearch for more papers by this authorJan Bosch, Jan BoschSearch for more papers by this author Book Editor(s):Ron S. Kenett, Ron S. Kenett KPA, Israel and Samuel Neaman Institute, Technion, IsraelSearch for more papers by this authorFabrizio Ruggeri, Fabrizio Ruggeri CNR-IMATI, ItalySearch for more papers by this authorFrederick W. Faltin, Frederick W. Faltin The Faltin Group and Virginia Tech, USASearch for more papers by this author First published: 06 July 2018 https://doi.org/10.1002/9781119357056.ch16 AboutPDFPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShareShare a linkShare onFacebookTwitterLinked InRedditWechat Summary This chapter describes how the open source continuous integration and delivery framework Eiffel was developed by Ericsson to address the challenges of scalability and traceability. It discusses the dynamic selection of test cases as a method to reduce time and resource usage of, particularly, continuous delivery testing. The chapter discusses the need for dynamic test case selection and how it requires traceability while touching upon selection strategies and handling of manual and automated tests on a conceptual level. It posits that the traceability data generated by Eiffel can in fact be used to great effect to facilitate a wide range of dynamic test selection methods, and have shown through examples how this can be achieved. The chapter lists several examples of methods for test case selection. It suggests that all of these may favorably be achieved through analysis of Eiffel events and their relationships. Analytic Methods in Systems and Software Testing RelatedInformation

This extended abstract summarizes an article, which has been published in Information and Software Technology and was selected for the Journal-First presentations at the International Conference on Software and System Process (ICSSP 2018).