It is our pleasure to welcome you to the Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI 2019) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP 2019). Due to their closeness, CESI and SER&IP merged into CESSER-IP for 2019. Both look into challenges in the cooperation between software engineering industry and research. Practitioners often believe that researchers mostly work on theoretical and “ivory tower” challenges that do not consider the complexity of real-world projects and therefore doubt the results produced by researchers. On the other hand, researchers often believe that practitioners are only interested in quick fixes to their problems. Hence, empirical studies are necessary to ensure the relevance and applicability of software engineering research. The joint workshop (CESSER-IP) will explore if and how results obtained in specific industry contexts can help advance the industry in general. We believe that this will further stimulate empirical research in software engineering and increase industry’s appreciation of empirical research.
A microservice architecture is expected to provide a better modularization and management of small and autonomous services. Other expected benefits include increased availability and time to market. There is a growing interest of both industry and academia on streamlining the migration of existing systems to a microservice architecture. However, the success of this migration is largely dependent on the use of appropriate criteria for extracting microservices from a code base. Recent studies indicate the selection and decomposition of microservices represent the main challenge along the migration. Academic techniques tend to support the extraction of microservices with either one or two conventional criteria, namely coupling and cohesion. There is limited knowledge on the criteria actually considered as useful by practitioners. Thus, we have performed an exploratory online survey with 15 specialists experienced on migrating systems to a microservices architecture. In particular, we question the relative usefulness of seven possible criteria for supporting decision-making along microservice extraction. The participants were also questioned about tools they have used, their limitations, and whether the decisions on extracted microservices were considered unsuccessful. Overall, the survey results suggest academic techniques do not totally satisfy the needs of practitioners. Practitioners often need to consider simultaneously at least four dominant criteria as well as their trade-offs to support their decisions. Most practitioners consider existing tooling support insufficient or even irrelevant to support their microservice extraction decisions.
Context. Software testing is the area of software engineering focused on determining whether a software meets the planned requirements and on evaluating its quality. Lately, academic researchers have increased their attention in this topic due to the impact of its success on software projects. However, recent studies have discussed that practitioners and researchers might have different views regarding what is important to explore and study in order to improve the software testing process. Goal. This study aims to investigate the differences of interests between academic researchers and practitioners in software testing, pointing out observable convergences and divergences between the two communities. Method. A mixed-method approach based on a mapping study, a quantitative study and a focus group was applied to collect quantitative and qualitative data from professionals and academic sources. Results. Our results confirm the existence of a gap between the two communities and the findings suggest that, while researchers are mainly focused on the proposition of novel tools and techniques, practitioners are more interested in issues related to the evaluation and discussions of existing approaches, tools and techniques. Therefore, academic researchers might consider identify, understand and modify the existing tools and strategies, instead of building new ones. Conclusion. In general, the distinction between the two groups is noticeable and there is only one strong mutual interest between both practitioners and researchers, namely, test automation. Therefore, there is a need for the development of strategies that reduce the gap between academia and industrial practice and bring them closer in order to increase the quality of the software testing processes.
Practitioners and scholars often face new software engineering phenomena which lack sufficient theoretical grounding. When studying such nascent and emerging topics, it is important to establish an initial and rudimentary understanding, leaving a more precise understanding of underpinning mechanisms till later. Controlled experiments, for example, might lead to insights into the specific mechanisms underpinning a certain practice, such as distributed development, pair programming, and test-driven development. However, at an initial stage of research, such highly controlled studies may not be feasible. In other domains, it may not be clear what the key constructs are, so that effective measurement cannot be done. Instead, researchers might opt for pragmatic alternative research approaches that do not require experimental control or active intervention in a study's setting. In this paper we advocate the use of soft theory (based on soft modeling techniques) for quantitative studies in software engineering research. We discuss the use of soft theory and position it within an existing taxonomy of quantitative data analysis techniques. Soft modeling and soft theory affords us a pragmatic approach to developing inferential and predictive research models, rather than aiming to develop a causal understanding. Soft theory approaches are grounded in robust quantitative data analysis techniques. We argue that these techniques can be effectively used in industry settings which are not amenable to highly controlled studies.
Artificial intelligence and machine learning (AI/ML) are some of the newest trends to hit the software industry, compelling organizations to evolve their development processes to deliver novel products to their customers. In this talk, I describe a study in which we learned how Microsoft software teams develop AI/ML-based applications using a nine-stage AI workflow process informed by prior experiences developing early AI applications (e.g. search and NLP) and data science tools (e.g. application telemetry and bug reporting). Adapting this workflow into their pre-existing, well-evolved, Agile-like software engineering processes and job roles has resulted in a number of engineering challenges unique to the AI/ML domain, some universal to all teams, but others related to the amount of prior AI/ML experience and education the teams have. I tell you about some challenges and the solutions that teams have come up with. The lessons that Microsoft has learned can help other organizations embarking on their own path towards AI and ML.
There is increasing interest in machine learning (ML) techniques and their applications in recent years. Although there has been intensive support by frameworks and libraries for the implementation of ML-based systems, investigation into engineering disciplines and methods is still at the early phase. The most pressing issue in this field is identifying the essential challenges for the software engineering research community as engineering of ML-based systems requires novel approaches due to the essentially different nature of ML-based systems. In this paper, we analyze the results of a questionnaire administered to 278 people who have worked on ML-based systems in practice, clarify the essential difficulties and their causes as perceived by practitioners, and suggest potential research directions.
An intuitive method is needed to achieve buy-in from all sectors of Engineering for a way to gauge release-over-release change for a given product's sequence of releases. Also, customers need to know if there are extant releases that are more reliable than the ones they already rely on in their networks. A new Release-Over-Release (RoR) metric can both enable customers to clearly understand the reliability risk of migrating to other available releases, and also enable Engineering to understand if their software engineering efforts are actually improving release reliability.
Background: A grand challenge for Requirement Engineering (RE) research is to help practitioners understand which RE methods work in what contexts and why. RE researchers recognize that for an RE method to be adopted in industry, RE practitioners should be able to evaluate the relevance of empirical studies to their practice. One possible approach to relevance evaluation is the set of perspective-based checklists proposed by Kitchenham et al. Specifically, the checklist from the practitioner's perspective seems to be a good candidate for evaluating the relevance of RE studies to RE practice. However, little is known about the applicability of this checklist to the RE field. Moreover, this checklist also requires a deeper analysis of its reliability. Aim: We propose a perspective-based checklist to the RE community that allows evaluating the relevance of experimental studies in RE from the practitioner's/consultant's viewpoint. Method: We followed an iterative design-science based approach in which we first analyzed the problems with a previously published checklist and then developed an operationalized proposal for a new checklist to counter these problems. We performed a reliability evaluation of this new checklist by having two practitioners apply the checklist on 24 papers that report experimental results on software requirements specifications' comprehensibility. Results: We report first-hand experiences of practitioners in evaluating the relevance of primary studies in RE, by using a perspective-based checklist. With respect to the reliability of the adjusted checklist, 9 of out 19 questions show an acceptable proportion of agreement (between two practitioners). Conclusions: Based on our experience, the contextualization and operationalization of a perspective-based checklist helps to make it more useful for the practitioners. However, to increase the reliability of the checklist, more reviewers and more discussion cycles are necessary.
To assess the benefits of introducing Agile practices, it is important to get a clear understanding of the baseline situation, i.e. the situation before their introduction. Without a clear baseline, we cannot properly assess the extent of impacts, both positive and negative, of introducing Agile practices. This paper provides a preliminary guideline to help researchers in capturing and reporting baseline situations. The guideline has been developed through the study of literature and interviews with industry practitioners, and validated by experts in academia.
Context: Case studies are a useful approach for conducting empirical studies of software engineering, in part because they allow a phenomenon to be studied in its real-world context. However, given that there are several kinds of case studies, each with its own strengths and weaknesses, researchers need to know how to choose which kind to employ for a specific research study. Aim: The objective of this research is to compare two case study approaches: embedded, longitudinal case studies, and multi-case studies. Approach: We compared two actual software engineering case studies: a multi-case study involving interviews with 46 practitioners at 9 international companies engaged in offshoring and outsourcing, and a single case, participant observation embedded case study lasting 13 months in a mid-sized Irish software company. Both case studies were exploring similar problems of understanding the activities performed by members of scrum development teams. Results: We found that both multi-case and embedded case studies are suitable for exploratory research (hypothesis development) but that embedded research may also be more suitable for explanatory research (hypothesis testing). We also found that longitudinal case studies offer better confirmability, while multi-case studies offer better transferability. Conclusion: We propose a set of illustrative research questions to assist with the selection of the appropriate case study method.
Software Inspection is an important approach to find defects in Software Engineering (SE) artifacts. While there has been extensive research on traditional software inspection with pen-and-paper materials, modern SE poses new environments, methods, and tools for the cooperation of software engineers. Technologies, such as Human Computation (HC), provide tool support for distributed and tool-mediated work processes. However, there is little empirical experience on how to leverage HC for software inspection. In this vision paper, we present the context for a research program on this topic and introduce the preliminary concept of a theory-based ex-periment line to facilitate designing experiment families that fit together to answer larger questions than individual experiments. We present an example feature model for an experiment line for Soft-ware Inspection with Human Computation and discuss its expected benefits for the research program, including the coordination of research, design and material reuse, and aggregation facilities.
Conducting empirical research in software engineering industry is a process, and as such, it should be generalizable. The aim of this paper is to discuss how academic researchers may address some of the challenges they encounter during conducting empirical research in the software industry by means of a systematic and structured approach. The protocol developed in this paper should serve as a practical guide for researchers and help them with conducting empirical research in this complex environment.
Researchers perform empirical studies in industry to gain qualitative insights into a real-world problem. However, common critics are the diversity and selection process of participants. To address these issues, we propose to improve the integration of question-answering systems into empirical study. In this paper, we i) describe approaches to conduct studies in such systems, ii) exemplify corresponding challenges, and iii) discuss their potential. We illustrate the approaches on existing works in which they were partly implemented to show that they can succeed.
Background: Industry experiments are typically associated with higher external validity compared to academic experiments. However, when conducting industry experiments, dropouts and incomplete experimental tasks are quite common, which is unusual in academic experiments. To the best of our knowledge, this phenomenon has not been reported in the literature. Aim: Identify the circumstances that explain why some experimental subjects exhibit poor or null participation during experimental sessions. Method: An industry experiment with experienced programmers at the Universidad de las Fuerzas Armadas ESPE of Ecuador was performed. Several post hoc analyses of the experimental data revealed relationships that could explain the subjects' behavior. Results: A high percentage of older experienced programmers did not perform meaningful work in their task assignments, even though they were present during the entire experiment. Longer overall (i. e., not only programing) experience and poor knowledge of the programming language and integrated development environment have a negative influence in the degree of task completion as well. Conclusions: Several experienced professionals were found to live a two, mixed-factors reality: old age and technological lapse. This negatively influenced (to a greater or lesser extent, depending on the person) attitudes regarding performance of activities that differ from daily professional work.
Context: Conducting a Grounded Theory study is rigorous, demanding, and challenging. Misperceptions exist within the software engineering community [1]. Objective: The purpose of this paper is to describe one extended participant observation Grounded Theory study for aiding new empirical researchers wanting to run similar research studies. Method: Following Constructivist Grounded Theory, we conducted a two-year five-month participant-observation of eight software development projects at Pivotal, a software development organization; interviewed 33 software engineers, interaction designers, and product managers; and analyzed one year of retrospection topics. We iterated between analysis and theoretical sampling until achieving theoretical saturation, publishing papers on team code ownership [2], sustainable software development through overlapping code ownership [3], and software development waste [4]. Results: This paper describes the missteps, challenges, and unique insights that occurred while conducting a Grounded Theory study. Limitations: While the results are highly relevant to the researcher, the outcomes might not apply to other researchers. Conclusion: Conducting my own Grounded Theory research study, attending Glaser's Seminar, and reading and rereading Charmaz's and Glaser's books helped the researcher overcome misperceptions about Grounded Theory research.
Context: Agile software processes are more frequently being integrated with aspects of human centered design given the commonalities that the two approaches share, such as user involvement, importance attributed to testing and prototyping practices, iterative design and so on. Goal: tailor a SCRUM-based process by integrating it and adapting it with Human Centered Design (HCD) practices. Method: action research experience carried out in an industrial case study involving researchers, an Italian SME and stakeholders of the final product (a web portal). Results: The research has led to the definition of the HCD-SCRUM process. The classic process has been tailored by: including staff dedicated to the project, considering flexible duration of sprints based on the workload and priority of features to be developed, using an iterative-incremental progression of the SCRUM development practices and including customer interviews. Conclusion: The tailored process outlined in this paper has been substituted to the classic SCRUM process that the company was adopting before this research study.
The software engineering industry should be the laboratory of much, perhaps most, of the empirical software engineering research. Not only would this create a more realistic context and higher external validity of the empirical research, it would also ease the result transfer and make the results more convincing for the industry. Unfortunately, this is currently not the case. About 90% of software engineering experiments are, for example, conducted with students instead of software professionals as subjects. One reason for the lack of industry studies may be that an efficient and sustainable give-and-take-based collaboration between research and industry can be difficult to establish. The collaborations are frequently fragile, end before the research is completed, and lead to a waste of resources for both the researchers and the industrial partners. This paper presents stories and lessons learned from failed and successful research-industry collaborations. It has a focus on experience with the use of non-traditional collaboration types, such as payment to get industry participation in experiments, trade-based collaboration, lightweight collaborations at industry venues, and network-based collaborations. It is argued that empirical software engineering research should more often consider the use of alternative types of research–industry collaborations than those traditionally chosen.
Software development teams apply security practices to prevent vulnerabilities in the software they ship. However, vulnerabilities can be difficult to find, and security practices take time and effort. Stakeholders can better guide software development if they have empirical data on how security practices are applied by development teams. The goal of this paper is to inform managers and developers on the use of security practices through a case study of an industrial software team so that managers and developers can base their security practice adoption decisions on empirical evidence. We present a case study of security practice use in a typical software development project at IBM. We collected empirical data from three perspectives: qualitative observations, a survey of the team members, and text mining of the team's development history. The team's top three practices were "Track Vulnerabilities", "Apply Secure Coding Standards" and "Apply Security Tooling". Our results indicate that the IBM team applied all investigated security practices and had good security outcomes, that tools and automation in the development and testing processes are applied and valued, that most practices required regular effort from the team and support from the organization, and that the team recognizes opportunities for improvement. We provide a list of lessons learned.
Getting companies involved in and committed to empirical research projects is a major challenge for many researchers. This paper discusses several cases of lessons learned about how to involve and get the commitment of companies in different types of empirical research projects. This paper will discuss the experiences of two very large consortium projects and one medium-size research project. In addition, the roles of meetings, workshops, and off-site meetings are also discussed.
Context: Surveys constitute an valuable tool to capture a large-scale snapshot of the state of the practice. Apparently trivial to adopt, surveys hide, however, several pitfalls that might hinder rendering the result valid and, thus, useful. Goal: We aim at providing an overview of main pitfalls in software engineering surveys and report on practical ways to deal with them. Method: We build on the experiences we collected in conducting many studies and distill the main lessons learnt. Results: The eight lessons learnt we report cover different aspects of the survey process ranging from the design of initial research objectives to the design of a questionnaire. Conclusions: Our hope is that by sharing our lessons learnt, combined with a disciplined application of the general survey theory, we contribute to improving the quality of the research results achievable by employing software engineering surveys.
