Effective system testing of applications with a Graphical User Interface (GUI) front-end demands careful generation of event sequences as well as providing relevant test data for parameterized widgets, i.e., widgets that accept input values such as textboxes and textareas. Current GUI testing techniques either manipulate the source code of the application under test (AUT) to generate the test data, or blindly use a set of random string values. In this paper, we propose a third novel way to generate relevant test data for GUI testing. We exploit the information provided in the GUI structure to extract a set of key identifiers for each parameterized widget. These identifiers are used to compose appropriate search phrases and collect relevant test data from the Internet. The results of an empirical study on five GUI-based applications show that the proposed approach is applicable and can get some hard-to-cover branches in the subject programs to execute. The proposed technique works from the black-box perspective and is entirely independent from GUI modeling and event sequence generation, thus it does not need access to the source code of AUT and provides an opportunity to be integrated with the existing GUI testing frameworks.
Architectural component models are frequently used as a central view of architectural descriptions of software systems and therefore play a crucial role in the whole development process and in achieving the desired software qualities. The components in those models represent important high level structural units that are often used to group either lower-level sub-components or classes in object-oriented design views. In this paper we present a study that examines the relationships between the effort required to understand a component, measured through the time that participants spent on studying a component, and a number of information theory based and the corresponding counting based metrics on graphs at the component level. The results show a statistically significant correlation between all of the metrics and the effort required to understand a component. In a multivariate regression analysis we obtained some reasonably well-fitting models that can be used to estimate the effort required to understand a component.
Model checking has been successfully used for checking model design where the specification is given by a temporal logic formula. In this paper, we develop an approach to bounded model checking Linear Temporal Logic (LTL) properties of Hybrid Automata Pushdown System (HAPS) over finite traces. Such HAPS models are suitable formal representations for Cyber/Physical co-verification, verifying software controller with controlled plant together. We convert the LTL formula into a C program, which is interleaved with the execution of the HAPS under analysis. Our approach checks both safety and livens uniformly within the framework of bounded model checking through symbolic execution. We have realized this approach and applied it to real-world control systems. The evaluation has shown that our approach has major potential in verifying system-level LTL properties of cyber-physical systems.
Spectral fault localization (SFL) is one automatic fault-localization technique, which uses ranking metric to rank the risk of fault existence in each program entity after dynamically collecting the testing information. The effectiveness evaluation and comparison of ranking metrics are two important research problems. In this paper, we provide a uniformly theoretical investigation framework on longitudinally evaluating ranking metrics and horizontally comparing them for SFL techniques under any single fault scenario. We propose a generic vector table model as a novel device of thoroughly understanding various SFL techniques. By investigating rankings' mathematical formula of statements in the vector table model, the performance of different SFL techniques could be systematically analysed and compared. Under table model-driven evaluation framework, seven typical metrics as examples are explored, the existing equivalent group is extended, and the new relation of two equivalent groups is found. Our framework overcomes limitations of current empirical and theoretical approaches, and can theoretically evaluate the advantage and disadvantage of a SFL technique and compare different SFL techniques.
It is well known that an appropriate requirement specification is essential for the success of software development projects. In the model-driven development context, the requirement models can be used as entry artifacts for aligning the final software products with the stakeholders' needs. Thus, it is necessary to solve any modeling issue that may prevent the correct translation of the input requirement artifacts, i.e., the requirement models need to be verified for assuring the completeness of the design (MDD-oriented) models generated. In this paper, we face this issue by presenting an approach to integrate specific measures that automate the verification of a goal-oriented requirement approach - the i* framework - in the context of a MDD development process.
This paper proposes an approach to improve the quality of AspectJ applications which supports translating name-based point cuts to analysis-based point cuts automatically. Traditional name-based point cuts rely on the names of fields, methods, and classes in the program, which are well known to be fragile against software evolution. Analysis-based point cuts rely on the properties of the program via user-defined static program analysis, which therefore are free from such fragility. However, writing such programs manually is difficult because user has to write her analysis suitable for corresponding properties. Our approach solves the problem by translating name-based point cuts into analysis-based ones, this can be achieved by simply calling the method from within the conditional point cuts. We demonstrate the usefulness of analysis-based point cuts in multiple versions of several open-source AspectJ programs. The results show that the analysis-based point cuts generated by our approach are more robust than their original name-based counterparts.
There exists two types of developers in Open Source Software (OSS) projects: 1) Committers who have permission to commit edited source code to the Version Control System (VCS), 2) Developers who contribute source code but cannot commit to the VCS directly. In order to develop and evolve high quality OSS, projects are always in search of new committers. OSS projects often promote strong developers to become committers. When existing committers find strong developers, they propose their promotion to a committer role. Delaying the committer-promotion might lead to strong developers departing from an OSS project and the project losing them. However early committer-promotion comes with its own slew of risks as well (e.g., the promotion of inexperienced developers). Hence, committer-promotion decisions are critical for the quality and successful evolution of OSS projects. In this paper, we examine the committer-promotion phenomena for two OSS projects (Eclipse and Firefox). We find that the amount of activities by future committers was higher than the amount of activities by developers who did not become committers). We also find that some developers are promoted to a committer role very rapidly (within a few month) while some of developers take over one year to become a committer. Finally, we develop a committer-identification model to assist OSS projects identifying future committers.
Performance evaluation is an important approach for various systems to guarantee the quality of their services. However, most performance evaluation tasks face a problem: how to model the system workload? Traditional workload models have limitations when it comes to modeling different workloads. In this paper, we propose a workload model for characterizing and generating synthetic web workloads. First, we introduce a Context-based Sequential Action Model to describe users that exhibit similar access patterns. Next, we present a Workload Parameter Specification Language to describe workload parameters for workload generation. Then, we introduce our load-testing framework based on the proposed model. The representativeness and features of our model are demonstrated by comparing it to other models. Experiments show that our framework can generate accurate and stable synthetic workloads.
Configuration errors, similar to bugs, are often the dominant causes of errors in deployed applications, leading to application failures, exceptions and even crashes. Misconfigurations are hard to detect and diagnose. To address this issue, we propose a method to detect the potential misconfigurations. Unlike the existing work, we focus on the misconfigurations introduced by the relations between components. The options in configuration files of an application are extracted and classified into several groups. Directed by the relations, the possible inter-related parameters are discovered and checked with some proposed checking rules. We implement a prototype and make evaluations. The case study shows that this method can detect and check a majority of inter-related configuration options with less human effort.
Technical debt has become a popular term in the software engineering community in recent years for labelling issues and development risks incurred either intentionally or unintentionally throughout the entire software development process. There are some approaches for calculating and/or managing the costs related with different kinds of technical debt. Current research lacks a clear classification of these existing approaches. We therefore, in a first step, developed a classification scheme including respective categories and dimensions for this purpose and derived in a second step the need for a new approach that is able to (1) consider several levels of required target quality a project shall reach, and (2) base on a calculation mechanism that allows to regard experiences with known reference projects. In an experiment with two open source projects we find that the results of our model are in-line with an external quality judgment of these projects. It also shows how the resulting remediation costs depend on the actual quality level of the projects and the target quality level. We conclude this paper with future work regarding improvements of our calculation model and planned enhancements to cover design and documentation aspects.
In the recent years, web applications have become increasingly popular for delivering security critical services. Because web applications are exposed to various threats and attacks, numerous tools, including commercial tools and open source software, have been developed for detecting web application vulnerabilities, called web vulnerability scanner. Many studies have focused on evaluating web vulnerability scanners by comparing the vulnerability coverage, precision, recall, and time complexity. However, tremendous new attack scenarios and various hacking techniques usually cause erroneous judgement by the scanners and a comprehensive scan often results in redundant vulnerability alerts. Therefore, an efficient detection tools is essential and can be extremely helpful to the users. In this paper, we propose the advanced confusion matrix to estimate the performance of Web vulnerability scanners and then propose a cost-effective approach with three main phases to evaluating vulnerability scanners by additionally considering the reduction of redundant vulnerability alert. We define the redundant alert problem in scanner evaluation based upon two attributes, true duplication (TD) and false duplication (FD). Accordingly, we build up the Web Vulnerability Scanner Testbed, W-VST. Two experiments have been made to evaluate the performance. The experimental results indicate that our evaluation approach can verify the performance of scanners and W-VST is efficient in tool evaluation.
We present a requirement-centered analysis and testing framework that integrates methods and tools for capturing and formalizing textual customer requirements, analyzing requirements consistency, generating test cases from formalized requirements and executing them on the implementation model. The framework preserves a fine grained traceability of informal and formal requirements, test cases and implementation models throughout every step of the workflow. We instantiate the framework with concrete tools that we integrate via a file repository and Open Services for Lifecycle Collaboration (OSLC). The standardized integration ensures that the framework remains generic -- any specific tool used in our instantiation can be replaced by another one with compatible functionality. We apply our framework on an industrial airbag control chip case study that we use to illustrate step-by-step our requirements-driven analysis and test methodology.
In recent years, more organizations began to address usability problems to improve their competitive position and foster customer loyalty. Although there are some classification schemes to address usability problems, there is no cause-effect framework to enable in-process usability feedback and improvement. Orthogonal defect classification (ODC) had been widely used in software development process, but with insufficient attention to usability problems. In this paper, we present a classification and analysis framework to integrate usability practices with ODC framework. With our framework, the cause-effect relationship can be identified. By addressing the usability problems during the whole development cycle, human computer interface (HCI) and software process can be improved together. A case study was conducted to demonstrate the viability and effectiveness of our framework.
Search-based approaches are successfully used for generating unit tests for object-oriented programs in Java. However, these approaches may struggle to generate sequence method calls with specific values to achieve high coverage due to the large size of the search space. This paper proposes a memetic algorithm (MA) approach in which static analysis is used to identify method dependence relations (MDR) based on the field access. This method dependence information is employed for reducing the search space and used to guide the search towards regions that lead to full (or at least high) structural coverage. Our approach, MAMDR, combines both a genetic algorithm (GA) and Hill Climbing (HC) to generate test data for Java programs. The former is used to produce test cases that maximize the branch coverage of the CUT, while minimizing the length of each test case. The latter is used to target uncovered branches in the preceding search phase using static information that guides the search to generate sequences of method calls and values that could cover target branches. We compare MAMDR with pure random testing, a well-known search based approach (EvoSuite), and a simple MA on several open source projects and classes, and show that the combination of MA and MDR is effective.
Architectural component models represent high level designs and are frequently used as a central view of architectural descriptions of software systems. In this paper we present a study that examines the effect of the level of abstraction of the software architecture representation on the architecture-level understandability of a software system. Three architectural representations of the same software system that differ in the level of abstraction (and hence in the number of components used in the architecture) are studied. Our results show that an architecture at the abstraction level that is sufficient to adequately maps the system's relevant functionalities to the corresponding architectural components (i.e., each component in the architecture corresponds to one system's relevant functionality) significantly improves the architecture-level understanding of the software system, as compared to two other architectures that have a low and a high number of elements and hence tangles or scatters the system's relevant functionalities into several architectural components.
In previous studies on the application of Metamorphic Testing (MT), testing was usually conducted at the system level directly, without any well-planned integration strategy. This could lead to a difficulty in constructing Metamorphic Relations (MRs), as well as tracking and debugging the faults. On the other hand, traditional integration testing requires extra cost. In this paper, we combine the bottom-up integration approach with MT to address all the above problems. We use the Feature Selection (FS) system to illustrate our method and a 2-phase MT is conducted. With this 2-phase MT method, we can ease the MR identification and provide a clear hierarchy to easily track and isolate the faults for a complex system. Moreover, since such an integration approach is logically achieved with MRs, it does not need to have those specific set-ups as required in traditional integration testing.
In-Parameter-Order (IPO) algorithm is an effective strategy of combinatorial testing. And several variants of the algorithm have been developed for reducing the runtime and size of test cases or for dealing with certain problems in test case generation, such as IPOG, IPOG-F and IPOG-F2. In this paper, the general optimization strategies, which can be applied to these variants of the algorithm, are proposed to make each value of all parameters more evenly distributed in the test cases. The proposed optimization strategies mainly focus on choosing values for the extension to an additional parameter during the horizontal growth of the algorithm and filling values for don't care positions. Experimental results show that the proposed optimization strategies are effective in reducing runtime and producing smaller size of test suites with the increase of the domain size.
Nowadays, systems are becoming increasingly complex and large and the process of developing such large-scale systems is becoming complicated with high cost and enormous effort required. Such a complicated process has a prominent challenge to ensure the quality of delivered artifacts. Therefore there is clearly a need to facilitate reuse of developed artifacts (e.g., requirements, architecture, tests) and enable automated analyses such as risk analyses, prioritizing test cases, change impact analysis, with the objective to reduce cost, effort and improve quality. Model-based engineering provides a promising mechanism to facilitate reuse and enable automation. The key idea is to use models as the backbone of structuring repositories that contain reusable artifacts (e.g., test cases, requirements). Such a backbone model is subse-quently used to enable various types of automation such as model-based testing and automated rule verification. In this paper, we report 12 industrial projects from five different industry domains that all require the construction of model-based repositories to enable various types of automation. We believe using models as the backbone to structure repositories for the purpose of enabling different types of automation in different contexts is a new and non-conventional model-based development research approach. This exploratory paper will serve the basis for future research to derive a generic model-based repository.
Class cohesion is a design quality that has a great impact on maintenance of object-oriented systems. There has been much work on measuring class cohesion based on internal class relationships only. However, object-oriented systems involve many external class relationships carrying functional semantics which should be also taken into account in measuring class cohesion. In this paper, we present a novel metric for measuring class cohesion based on an iterative process using both internal and external class relationships.
Performance is an important facet of software quality, and application profiling tools are the instruments used to measure software performance at the function and application levels. The most powerful measurement method available in application profiling tools today is sampling-based profiling, where a potentially unmodified application is interrupted based on some event to collect data on what it was doing when the interrupt occurred. It is well known that sampling introduces statistical uncertainty that must be taken into account when interpreting results, however, factors affecting the variability have not been well-studied. In attempting to validate two previously published analytical estimates, we obtained negative results. Furthermore, we found that the variability is strongly influenced by at least one factor, self-time fragmentation, that cannot be determined from the data yielded by sampling alone. We investigate this and conclude with recommendations for obtaining valid estimates of uncertainty under the conditions that exist.
