The quality assurance of large scale integrative systems often requires complex testbed environments and simulations that allow to test the overall functionality and enables various experiments towards systematically verifying the realization of the identified user and system requirements. Thereby, an integration setup and resulting activities lead to another level of quality assurance, whereby the integrator deals with the quality examination of the single components and their integrative interplay according to a set of overall system and user requirements. In such context, it is often the case that the testing activity is conducted by various partners (e.g. single companies and legal entities) with complementing know how required for specific sub-tasks - e.g. PKI, chip cards, special network protocols, firewall, security architectures, and penetration testing. This leads to the emergence of a large number of proprietary testbeds focusing on specific aspects resulting in the lack of a unified testbed configuration, versioning and technological foundations (e.g. operating system, network stack implementations, hypervisor technology ...). In this paper, we present our experiences drafted from a large scale industrial project with 600-700 requirements relating to a critical eHealth infrastructure within a telecom provider context. Thereby, various sub-contractors had to be unified in their approach to testbed management in order to achieve reproducible and traceable (with respect to system requirements) test results based on a test architecture accommodating various quality assurance activities (unit testing, development tests, component testing, integration testing, security testing ...). We gradually analyze the project situation with respect to testbed management and argue on the need for unified testbed management across multiple teams and stakeholders in a large scale telecom integration setup. Subsequently, we propose possible solutions and conduct a series of experiments highlighting the advantages of the proposed approach and belonging solution.

In testing, the term arbitration describes the process of calculating the verdict after the execution of a test case or test suite. The calculation of the verdict follows a defined rule set that clearly specifies which verdict to produce under which conditions. In many situations, these rules simply follow the scheme that any deviation between the expected and the actual response of the system under test leads to fail. UTP 2 introduces the concept of arbitration specifications on various hierarchy levels to determine the final verdict of an arbitration target. It provides default arbitration specifications that adhere to the above-mentioned straightforward calculation of verdicts, but allows for overriding these default ones with user-defined arbitration specifications. Unfortunately, this override mechanism adversely affects the maintainability of test cases and test actions because of its high degree of intrusion. Arbitration targets, such as test sets, test cases and procedural elements, and arbitration specifications are tightly coupled with each other, losing the ability to reuse these arbitration targets in a different context with different arbitration specifications. In this paper, we suggest to replace this highly intrusive override mechanism with a decoupling binding mechanism. This binding mechanism increases both comprehensibility and maintainability of test specifications on one hand, because arbitration targets remain independent of any potential arbitration specification. On the other hand, it offers a high degree of reusability and flexibility to the user because of a cascading override mechanism inspired by W3C cascading style sheets.

The design and implementation of complex network infrastructures requires early and extensive planning. However, increased complexity may make it more difficult to anticipate potential risks associated with the implementation and later with the operation of the network in question. In order to minimize the risk and potential effort fault/error/failure/alarm analysis during network operation, there exist a variety of network simulation tools, which allow to evaluate the feasibility of complex network environments and can be used to examine the envisioned infrastructure before implementation. If one wants to go one step further and simulate networks as realistically as possible for an extensive analysis, the choice of available tools is rather small. An interaction of emulated routers - i.e. with their virtualized operating systems and configuration interfaces - and a simulation environment can remedy this. The fact is that there is only limited existence of such software solutions on the market. Indeed, these products usually lack an essential function - the visual representation of the traffic flows for investigating network behavior under various test or failure scenarios. This article aims to show ways to solve this problem and to investigate its applicability to other scenarios based on the utilization of various standards.

The quality of Web services is an important factor for businesses that advertise or sell their services in the Internet. Failures can directly lead to fewer costumers or security problems. However, the testing of complex Web services that are organized in service-oriented architectures is a difficult and complex problem. Model-based testing (MBT) is one solution to deal with the complexity of the testing. With MBT, testers do not define the tests directly, but rather specify the structure and behavior of the System Under Test using models. Then, a test strategy is used to derive test cases automatically from the models. However, MBT yields a large amount of tests for complex systems which require lots of resources for their execution, thereby limiting its potential. Within this article, we discuss how cloud computing can be used to provide the required resources for scaling up test campaigns with large amounts of test cases derived using MBT.

Fuzz testing is an established technique in order to find zero-day-vulnerabilities by stimulating a system under test with invalid or unexpected input data. However, fuzzing techniques still generate far more test cases than can be executed. Therefore, different kinds of risk-based testing approaches are used for test case identification, selection and prioritization. In contrast to many approaches that require manual risk analysis, such as fault tree analysis, failure mode and effect analysis, and the CORAS method, we propose an automated approach that takes advantage of an already shown correlation between interface complexity and error proneness. Since fuzzing is a negative testing approach, we propose a complexity metric for the negative input space that measures the boundaries of the negative input space of primitive types and complex data types. Based on this metric, the assumed most error prone interfaces are selected and used as a starting point for fuzz test case generation. This paper presents work in progress.

In systems and service engineering testing is an important part to get confidence in quality and trust in security issues. Standardized testing techniques support the unique definition of abstract test models, configurations and behavior scenarios that can be executed automatically. This contribution presents the state of the art and future directions of two international standards for testing: the Testing and Test Control Notation (TTCN-3) from the European Telecommunication Standardization Institute (ETSI), and the UML testing profile (UTP) from the Open Management Group (OMG). Special emphasize is given to the translation from UTP to TTCN-3 test models, automated test execution using standard-compliant tool support and related examples from European projects.

The increasing complexity of software-intensive systems raises a lot of challenges demanding new techniques for ensuring their overall quality. The risk of not meeting the expected level of quality has negative impact on business, customers, environment and people, especially in the context of safety/security-critical systems. The importance of risk assessment, analysis and management has been well understood both in the literature and practice, which has led to the definition of a number of well-known standards. In the recent years, Risk-Based Testing (RBT) is gaining more attention, especially focusing on test prioritization and selection based on risks. On the other hand, model-based testing (MBT) provides a systematic and automated way to facilitate rigorous testing of software-intensive systems. MBT has been an intense area of research and a large number of MBT techniques have been developed in literature and practice in the last decade. In this paper, we study the feasibility of combining RBT with MBT by using the upcoming version of UML Testing Profile (UTP 2) as the mechanism. We present potential traceability between RBT and UTP 2 concepts.

UML modeling environments for doing model-based testing are often not very comfortable to use and burden some knowledge about the internals of UML to the users, respectively test engineers. Test engineers, however, are seldom experts in UML, thus, the gain of efficiency model-based testing approaches entail, is reduced by a too generic tooling. The tool Fokus!MBT, developed by the competence center MOTION of Fraunhofer FOKUS, is a multi-paradigmatic test modeling environment based on the UML Testing Profile, an OMG-adopted industry-driven notation for model-based testing. Fokus!MBT simplifies the creation and authoring of test models with methodology-specific support. It is built on top of Eclipse Papyrus, a powerful open source UML modeling environment, which, in turn, relies on the Eclipse Modeling Framework and the Graphical Modeling Framework. This paper provides deep insights into the basic concepts and technical realization of Fokus!MBT as well as into the lessons we have learned during development and application.

UML modeling environments for doing model-based testing are often not very comfortable to use and burden some knowledge about the internals of UML to the users, respectively test engineers. Test engineers, however, are seldom experts in UML, thus, the gain of efficiency model-based testing approaches entail, is reduced by a too generic tooling. The tool Fokus!MBT, developed by the competence center MOTION of Fraunhofer FOKUS, is a multi-paradigmatic test modeling environment based on the UML Testing Profile, an OMG-adopted industry-driven notation for model-based testing. Fokus!MBT simplifies the creation and authoring of test models with methodology-specific support. It is built on top of Eclipse Papyrus, a powerful open source UML modeling environment, which, in turn, relies on the Eclipse Modeling Framework and the Graphical Modeling Framework. This paper provides deep insights into the basic concepts and technical realization of Fokus!MBT as well as into the lessons we have learned during development and application.

Since software systems become more and more complex, the efforts for developing, documenting and executing meaningful test cases increases. Testing is a vital, but time- and resource-consuming activity. To avoid running out of time or budget, new test methodologies had to be established in order to increase reliable, yet maintainable test scenarios. In the last years the Model-Driven idea matures to the most promising approaches to solve current problems in the software development domain. Model-Based Testing adopts these concepts to exploit their benefits for testing area. In this paper an integrated tool chain (called FOKUS!MBT) is discussed, to enable a Model-based development of testing scenarios. It is based on a canonical metamodel for testing concerns and a service-oriented model storage and exchange infrastructure, that allows a flexible, yet extensible adaptation to different test process requirements. Its premise is to establish a tooling architecture for the specification and development of a domain-independent Model-based testing scenario.

The European Space Agency (ESA) as many other companies is interested in capitalizing its business assets. With the space programmes often lasting 10 to 20 years, the software system migration problems arise frequently. The Object Management Group promotes the Model Driven Architecture (MDA) concept and proposes the Architecture Driven Modernization (ADM) approach for model-based platform migration. SOFTEAM, Fraunhofer FOKUS and GTI6 performed an ESA-funded study on Round Trip Engineering for Space Systems. During this study the state-of-the-art methods and tools for ADM and MDA were combined with state-of-the-art model based testing (MBT) approaches to safeguard the modernization process. Both techniques were assessed by applying them to a real-life use case – the migration and testing of a distributed archive and versioning system. In this article we overview the combined platform migration and testing methodology used in the project and summarize our experience during its application to the ESA’s File Archive System. We specially focus on real life experience with MBT and discuss lessons learned.

Software developed for the space domain often has to deal with extremely long mission times (sometimes in the order of 15 to 20 years). During the lifetime of a mission programming platforms evolve and sometimes disappear forcing migrations or updates. Migration can also be triggered by the appearance of new platforms that can improve scalability, performance. European Space Agency (ESA) is interested in modernization approaches that simplify platform migration and that preserve the business values of systems. The Architecture Driven Modernization (ADM) promoted by the Object Management Group (OMG) proposes to recover the models which represent the business value and proceed with the platform migration in a forward MDA process. This article provides results of a study dedicated to assess the state-of-the-art tools and methods for model driven platform migration, including model-based testing and metrication.

This contribution addresses systematic test development methods to include an algorithm to retrieve a test suite execution control in order to run test cases with high priority earlier than others. The approach uses a model that allows both the introduction of user-defined weightings for system features within the test model and an automatic calculation of the test ordering. Based on an algorithm for the calculation of test case weights first results from the application of a tool implementation in pilot projects have been described.

This paper considers conformance testing of TINA service components in the realm of the ITU-T conformance testing methodology, which provides the only standardized, well accepted and widely used test methodology and test notation TTCN. It discusses the use of TTCN for testing computational objects of service components and proposes an implementation approach to derive executable tests from TTCN, which can be executed on any CORBA compliant ORB (with minor efforts for ORB specific adaptation) to check the functional correctness of deployed service components in distributed object environments. The TTCN/CORBA gateway is a general approach for testing distributed systems. TINA is taken as an example in this paper only. The work presented has been partially supported by the EC ACTS project TOSCA-TINA Open Service Creation Architecture, AC237.

Following the success of CORBA based systems the OMG has standardized the CORBA Component Model (CCM) to improve the implementation process of large distributed systems. The European project COACH [16] has been set up to build an Open Source development platform to construct CCM applications. As part of COACH a toolset for CCM components and system testing has been defined and implemented. This paper introduces the various components and features which have been foreseen and implemented for test activities such as interactive component testing, test trace visualization, or the application of abstract test specifications. The resulting test infrastructure addresses the CCM specifics but also benefits from CCM, e.g. by incorporating component communication facilities.

The Reference Model for Open Distributed Processing (RM-ODP) [5] describes an architecture for the design of distributed services where the basic idea is to split the design concerns into several viewpoints. This is in order to overcome the immense complexity of today’s distributed systems by structuring the design process.