Abstract : This report discusses the impact of cloud computing and the broader revolution in computing on systems, on the disciplines of systems engineering that have evolved over the last half century, and on new opportunities for these disciplines. This report sees the need for a new synthesis of traditional and computer-science-based variants of systems engineering. It recommends that the Department of Defense (DoD), perhaps in collaboration with other agencies, invest in new research and educational initiatives to develop and implement a new function, and perhaps a new hybrid discipline, of computational systems engineering. As software and computing have moved to the very center of system design, it is no longer feasible to treat these issues as mere component-level concerns. Rather, they must now be addressed at the highest level of system definition and development. That said, the traditional systems engineering issues of integrating across many other areas of expertise have not gone away. To succeed now requires an integration of expertise from traditional systems engineering disciplines and from the de facto systems engineering disciplines that have evolved in parallel in the computer sciences, notably but not only software engineering and cyber-physical systems. Neither discipline is configured to succeed on its own. A new synthesis is required.

There are many technical challenges in ensuring high life-time quality of NASA's systems. Some of NASA's software-related challenges could potentially be addressed by the many powerful technologies that are being developed in software research laboratories. However, most such research technologies do not make the transition from the research lab to the software lab because research infusion and technology transfer is difficult. For example, there must be evidence that the technology works in the practitioner's particular domain, and there must be a potential for great improvements and enhanced competitive edge for the practitioner, for such infusion to take place. NASA IV&V's Research Infusion initiative strives to facilitate such infusion. In 2006, a research infusion project involving Johns Hopkins University Applied Physics Laboratory (JHU/APL) and the Fraunhofer Center for Experimental Software Engineering Maryland, was successfully completed infusing Fraunhofer's software architecture visualization and evaluation (SAVE) tool. The infusion project helped improve JHU/APL's software architecture and produced evidence that SAVE is applicable to software architecture problems in the aerospace domain, spawning a series of related research infusion projects. The project also led to the discovery of other needs that could not be addressed by current technologies and, therefore, spawned the research and development of a new technology that will be ready for infusion in the future. This paper describes the SAVE technology followed by a description of the infusion of SAVE at JHU/APL and the other projects that followed, as well as the newly started Dynamic SAVE research and development project. Lessons learned related to various aspects of research infusion conclude the paper.

Developing and maintaining software is difficult and error prone. This can at least partially be attributed to its constantly evolving nature. Requirements are seldom finalized before the development begins, but evolve constantly as new details about the system become known and stakeholder objectives change. As requirements are altered, the software architecture must be updated to accommodate these changes. This includes updating the architecture documentation, which serves as the design specification as well as a means of comprehending complex systems. Furthermore, the software architecture of the implementation must be adapted in order to ensure that the system complies with both functional and non-functional requirements.In practice, however, software changes are often applied in an ad-hoc manner. As a result, the implementation frequently deviates from the architecture documentation, rendering the latter useless for supporting system engineers in comprehending the system and aiding maintenance tasks. Furthermore, errors that are introduced during the implementation lead to discrepancies between the system and the intended architecture design. Consequently, it cannot be guaranteed that the system meets the desired quality objectives, such as reliability and dependability.We present the behavioral reflexion model approach, which aims to support the system engineer in identifying and resolving discrepancies between software architecture representations. In our approach, the system engineer is supported in producing architecture documentation that reflects the intended architecture. Furthermore, discrepancies between the implementation and documentation are identified. These discrepancies are then illustrated graphically in a reflexion model, which guides debugging activities.In this research, we are concerned with architecture representations of system behaviors and focus in particular on distributed systems. In this thesis, we describe how architecture discrepancies are introduced and the implications for the reliability and maintainability of the system. We then discuss the individual components of the behavioral reflexion model approach in detail. Finally, we provide an evaluation of our approach in the form of two case studies. In these studies, we applied the behavioral reflexion model approach to two space-mission systems with the goal to resolve problems in their reliability and maintainability.

This paper describes a method for extracting functional requirements from tests, where tests take the form of vectors of inputs (supplied to the system) and outputs (produced by the system in response to inputs). The approach uses data-mining techniques to infer invariants from the test data, and an automated-verification technology to determine which of these proposed invariants are indeed invariant and may thus be seen as requirements. Experimental results from a pilot study involving an automotive-electronics application show that using tests that fully cover the structure of the software yield more complete invariants than structurally-agnostic black-box tests.

This extended poster abstract presents a software visualization tool (CodeVizard) that helps researchers and managers to analyze software repository data. The tool focuses on identifying areas of risks in software development projects, such as: Code Smells, degrading architectures, increasing software complexity, lack of documentation, process violations, and issues of code ownership. CodeVizard has been used to support six empirical studies, whose results have been published. This extended abstract gives a short summary of the important empirical results and features.

Software architecture has become essential in the struggle to manage today's increasingly large and complex systems. Software architecture views are created to capture important system characteristics on an abstract and, thus, comprehensible level. As the system is implemented and later maintained, it often deviates from the original design specification. Such deviations can have implication for the quality of the system, such as reliability, security, and maintainability. Software architecture compliance checking approaches, such as the reflexion model technique, have been proposed to address this issue by comparing the implementation to a model of the systems' architecture design. However, architecture compliance checking approaches focus solely on structural characteristics and ignore behavioral conformance. This is especially an issue in Systems-of-Systems. Systems-of-Systems (SoS) are decompositions of large systems, into smaller systems for the sake of flexibility. Deviations of the implementation to its behavioral design often reduce the reliability of the entire SoS. An approach is needed that supports the reasoning about behavioral conformance on architecture level.In order to address this issue, we have developed an approach for comparing the implementation of a SoS to an architecture model of its behavioral design. The approach follows the idea of reflexion models and adopts it to support the compliance checking of behaviors. In this paper, we focus on sequencing properties as they play an important role in many SoS. Sequencing deviations potentially have a severe impact on the SoS' correctness and qualities. The desired behavioral specification is defined in UML sequence diagram notation and behaviors are extracted from the SoS implementation. The behaviors are then mapped to the model of the desired behavior and the two are compared. Finally, a reflexion model is constructed that shows the deviations between behavioral design and implementation. This paper discusses the approach and shows how it can be applied to investigate reliability issues in SoS.

NASA systems need to be highly dependable to avoid catastrophic mission failures. This calls for rigorous engineering processes including meticulous validation and verification. However, NASA systems are often highly distributed and overwhelmingly complex, making the software portion of these systems challenging to understand, maintain, change, reuse, and test. NASA's systems are long-lived and the software maintenance process typically constitutes 60-80% of the total cost of the entire lifecycle. Thus, in addition to the technical challenges of ensuring high life-time quality of NASA's systems, the post-development phase also presents a significant financial burden. Some of NASA's software-related challenges could potentially be addressed by some of the many powerful technologies that are being developed in software research laboratories. Many of these research technologies seek to facilitate maintenance and evolution by for example architecting, designing and modeling for quality, flexibility, and reuse. Other technologies attempt to detect and remove defects and other quality issues by various forms of automated defect detection, architecture analysis, and various forms of sophisticated simulation and testing. However promising, most such research technologies nevertheless do not make the transition from the research lab to the software lab. One reason the transition from research to practice seldom occurs is that research infusion and technology transfer is difficult. For example, factors related to the technology are sometimes overshadowed by other types of factors such as reluctance to change and therefore prohibits the technology from sticking. Successful infusion might also take very long time. One famous study showed that the discrepancy between the conception of the idea and its practical use was 18 years plus or minus three. Nevertheless, infusing new technology is possible. We have found that it takes special circumstances for such research infusion to succeed: 1) there must be evidence that the technology works in the practitioner's particular domain, 2) there must be a potential for great improvements and enhanced competitive edge for the practitioner, 3) the practitioner has to have strong individual curiosity and continuous interest in trying out new technologies, 4) the practitioner has to have support on multiple levels (i.e. from the researchers, from management, from sponsors etc), and 5) to remain infused, the new technology has to be integrated into the practitioner's processes so that it becomes a natural part of the daily work. NASA IV&V's Research Infusion initiative sponsored by NASA's Office of Safety & Mission Assurance (OSMA) through the Software Assurance Research Program (SARP), strives to overcome some of the problems related to research infusion.

In this chapter, we propose an architectural design evaluation technique called EBEAM (Expert‐Based Evaluation of Architecture for Modifiability) that assists experts in articulating their knowledge of architectural designs and expressing the knowledge in measurable terms. EBEAM supports the evaluation of different architectural design versions for modifiability. In addition, EBEAM supports relative comparison between these design versions and the target design. We develop EBEAM as a generalized technique that is reusable for evaluating other architectural design attributes, apart from modifiability. We discuss EBEAM in detail and report on two case studies that investigate its applicability, and one study that validates the results of the evaluations made using EBEAM.

In this paper, we analyze software that we inherited from another party. We analyze its architecture and use common design principles to identify critical changes in order to improve its flexibility with respect to a set of planned extensions. We describe flexibility issues that we encountered and how they were addressed by a redesign and re-implementation. The study shows that basic and well-established design concepts can be used to guide the design and redesign of software.

This paper discusses the results of an application of a formally based verification technique, called Instrumentation-Based Verification (IBV), to a production automotive lighting controller. The goal of the study is to assess, from both a tools as well as a methodological perspective, the performance of IBV in an industrial setting. The insights obtained as a result of the project include a refinement of a previously developed architecture for requirements specifications; observations about changes to model-based design workflows; insights into the role of requirements during development; and the capability of automated verification to detect inconsistencies among requirements as well as between requirements and design models.

This paper presents experiences of verifying architectural design rules of the NASA Core Flight Software (CFS) product line implementation. The goal is to check whether the implementation is consistent with the CFS' architectural rules derived from the developer's guide. The results indicate that consistency checking helps a) identifying architecturally significant deviations that were eluded during code reviews, b) clarifying the design rules to the team, and c) assessing the overall implementation quality. Furthermore, it helps connecting business goals to architectural principles, and to the implementation. This paper is the first step in the definition of a method for analyzing and evaluating product line implementations from an architecture-centric perspective.

In model-based design verification, software models are checked against functional and nonfunctional requirements. Many of the mathematically well-founded theories developed for functional verification suffer from limitations (poor integration with industrial modeling tools and inability to scale to larger, more complicated systems) that restrict their practical adoption. Nonfunctional verification approaches, because of their inherent subjectivity (“what does it mean for a system to be modifiable?”), remain largely ad hoc and manual. There is also another overarching problem in that functional and nonfunctional verification take place on widely different representations of the system making it difficult to ensure that these representations are consistent with each other. This chapter highlights the latest research done into (1) a practical, formal, coverage-based functional verification approach called instrumentation-based verification, (2) quality attribute reasoning, a semiautomated, nonfunctional verification theory, and (3) an integrated functional and nonfunctional verification approach for model-based development.

Prominent characteristics of systems in the aerospace domain are that they are inherently complex, they must operate under tight resource constraints, and are often parts of a larger system of systems that must be reliable. These systems communicate with each other to exchange data and control information to together fulfill a larger task. In such a setup, the reliability of the communication channel plays a central role in the reliability of the entire system of systems and thus determines the success of fulfilling the larger task. Ensuring such a reliable communication is difficult due to several reasons: (1) the systems are developed independently by different teams at different locations, (2) the specification of the expected communication behavior is ambiguous, and (3) issues in the communication are often subtle and remain uncovered for a long time with the effect that bandwidth and other precious resources are wasted. We are proposing an approach called Dynamic Software Architecture Visualization and Evaluation (DynSAVE) to detect problems in the communication between systems by analyzing their communication behavior. The approach is divided into three main steps. The first step is the non-intrusive monitoring and recording of low level network traffic, the second step converts these raw communication records into meaningful messages, and the third step visualizes this abstracted information in such a way that issues can be detected. In this paper we discuss how the approach was applied to the Consultative Committee for Space Data Systems (CCSDS) File Delivery Protocol (CFDP), which is used for satellite communication by the JHU/APL Common Ground System. The approach has proven to be useful for understanding the communication behavior and uncovering subtle issues due to emerging system behaviors.

In this chapter, we propose an architectural design evaluation technique called EBEAM (Expert-Based Evaluation of Architecture for Modifiability) that assists experts in articulating their knowledge of architectural designs and expressing the knowledge in measurable terms. EBEAM supports the evaluation of different architectural design versions for modifiability. In addition, EBEAM supports relative comparison between these design versions and the target design. We develop EBEAM as a generalized technique that is reusable for evaluating other architectural design attributes, apart from modifiability. We discuss EBEAM in detail and report on two case studies that investigate its applicability, and one study that validates the results of the evaluations made using EBEAM.

This paper presents an approach for recovering application-level views of the interaction behaviors between systems that communicate via networks. Rather than illustrating a single behavior, a sequence diagram is constructed that describes the characteristics of multiple combined behaviors. The approach has several properties that make it particularly suitable for analyzing heterogeneous systems. First, since the interactions are retrieved from observing the network communication, our technique can be applied to systems that are implemented in different languages and run on different platforms. Second, it does not require the availability or modification of source code. After the behaviors are extracted, we employ methods to merge multiple observed behaviors to a single sequence diagram that illustrates the overall behavior.The contributions of this paper are a technique for observing and processing the network communication to derive a model of the behavior. Furthermore, it describes a series of model transformations to construct a sequence diagram view of all observed behaviors.

This paper reports on an industrial case study in a large Norwegian Oil and Gas company (StatoilHydro ASA) involving a reusable Java-class framework and an application that uses that framework. We analyzed software changes from three releases of the framework and the application. On the basis of our analysis of the data, we found that perfective and corrective changes account for the majority of changes in both the reusable framework and the non-reusable application. Although adaptive changes are more frequent and has longer active time in the reusable framework, it went through less refactoring compared to the non-reusable application. For the non-reusable application we saw preventive changes as more frequent and with longer active time. We also found that designing for reuse seems to lead to fewer changes, as well as we saw a positive effect on doing refactoring.

Systems often collaborate to form a system-of-systems (SoS) and together fulfill some larger task. Correctness and performance issues in the interaction between participating systems are frequent occurrences and decrease the reliability of the entire SoS. We are currently developing an analysis framework to automatically compare a model of the desired interaction behavior (specification) to a model that is retrieved from the system execution and detect deviations between the two. The specification, the observed interaction behavior, and the evaluation result are presented in behavioral diagrams to be analyzed by the user.

Many software systems are evolving complex system of systems (SoS) for which inter-system communication is mission-critical. Evidence indicates that transmission failures and performance issues are not uncommon occurrences. In a NASA-supported Software Assurance Research Program (SARP) project, we are researching a new approach addressing such problems. In this paper, we are presenting an approach for analyzing inter-system communications with the goal to uncover both transmission errors and performance problems. Our approach consists of a visualization and an evaluation component. While the visualization of the observed communication aims to facilitate understanding, the evaluation component automatically checks the conformance of an observed communication (actual) to a desired one (planned). The actual and the planned are represented as sequence diagrams. The evaluation algorithm checks the conformance of the actual to the planned diagram. We have applied our approach to the communication of aerospace systems and were successful in detecting and resolving even subtle and long existing transmission problems.