The modeling of power and thermal behavior of processors requires challenging validation processes, which may be complex and undependable. In order to ameliorate some of the difficulties associated with the validation of power and thermal models, this paper describes an infrared measurement setup that simultaneously captures run-time power consumption, thermal characteristics, and performance activity counters from modern processors. We use infrared cameras with high spatial resolution (10x10/ μm ) and high frame rate (125Hz) to capture thermal maps. Power measurements are obtained with a multimeter, while performance counters are obtained after modifying the operating system (Linux), both at a sampling rate of 1 KHz . The synchronized traces can then be used in the validation process of possible thermal, power, and processor activity models.
Pipeline Spectroscopy is a new technique that allows us to measure the cost of each cache miss. The cost of a miss is displayed (graphed) as a histogram, which represents a precise readout showing a detailed visualization of the cost of each cache miss throughout all levels of the memory hierarchy. We call the graphs 'spectrograms' because they reveal certain signature characteristics of the processor's memory hierarchy, the pipeline, and the miss pattern itself. We show that in a memory hierarchy with N cache levels (L1, L2, ..., L N , and memory) and a miss cluster of size C, there are (C+N/C) possible miss penalties. This represent all possible sums from all possible combinations of the miss latencies from each level of the memory hierarchy (L2, L3, ... Memory) for a given cluster size. Additionally, a theory is presented that describes the shape of a spectrogram, and we use this theory to predict the shape of spectrograms for larger miss clusters. Detailed analysis of a spectrograph leads to much greater insight in pipeline dynamics, including effects due to prefetching, and miss queueing delays.
XML simplifies data exchange among heterogeneous computers, but it is notoriously verbose and has spawned the development of many XML-specific compressors and binary formats. We present an XML test corpus and a combined efficiency metric integrating compression ratio and execution speed. We use this corpus and linear regression to assess 14 general-purpose and XML-specific compressors relative to the proposed metric. We also identify key factors when selecting a compressor. Our results show, XMill or WBXML may be useful in some instances, but a general-purpose compressor is often the best choice.
The overhead of a context switch is typically associated with multitasking, where several applications share a processor. But even if only one runnable application is present in the system and supposedly runs alone, it is still repeatedly preempted in favor of a different thread of execution, namely, the operating system that services periodic clock interrupts. We employ two complementing methodologies to measure the overhead incurred by such events and obtain contradictory results. The first methodology systematically changes the interrupt frequency and measures by how much this prolongs the duration of a program that sorts an array. The overall overhead is found to be 0.5-1.5% at 1000 Hz, linearly proportional to the tick rate, and steadily declining as the speed of processors increases. If the kernel is configured such that each tick is slowed down by an access to an external time source, then the direct overhead dominates. Otherwise, the relative weight of the indirect portion is steadily growing with processors' speed, accounting for up to 85% of the total. The second methodology repeatedly executes a simplistic loop (calibrated to take 1ms), measures the actual execution time, and analyzes the perturbations. Some loop implementations yield results similar to the above, but others indicate that the overhead is actually an order of magnitude bigger, or worse. The phenomenon was observed on IA32, IA64, and Power processors, the latter being part of the ASC Purple supercomputer. Importantly, the effect is greatly amplified for parallel jobs, where one late thread holds up all its peers, causing a slowdown that is dominated by the per-node latency (numerator) and the job granularity (denominator). We trace the bizarre effect to an unexplained interrupt/loop interaction; the question of whether this hardware mis-feature is experienced by real applications remains open.
Computational experiments often discard large amounts of valuable data, such as invocation parameters and the lineage of output. Our goal is to identify, manage, capture, and organize this information. These data can be used to make the scientific process simpler and more efficient, and to increase the value of the research by making it more rigorous and reproducible. Research Assistant (RA) is an open source Java programming tool that helps to plug this information leak. RA ensures that all console output is valid XML; saves invocation parameters, the random seed, and code version information; automatically checkpoints intermediate results; creates runnable experiment packages; and keeps meticulous notes. This paper presents the design and implementation of RA, and shows how RA easily scales to make complex experiments repeatable.
In surprisingly many computer science research projects, system outcomes may be influenced by computerized or human agents with different economic incentives. Such studies include P2P networks, routing protocols, agent systems, and attacker-defender security games. Even the most technical system raises a pressing economic question: what incentives will drive the success or failure of deployment? Traditional computer science techniques, in isolation, may overlook factors that are critical to the overall outcome: In these situations, human-subject experiments may form a useful complement, broadening our understanding through the formulation and testing of economically motivated hypotheses. I argue that these efforts can benefit from the large body of work that has been conducted in the field of experimental economics in the last 30 to 40 years. I discuss the methodology of experimental economics and review recent work that falls on the boundary of computer science and economics experiments and is likely unfamiliar to many professionals and researchers in technical fields.
In this paper, we present the design of a performance isolation benchmark that quantifies the degree to which a virtualization system limits the impact of a misbehaving virtual machine on other well-behaving virtual machines running on the same physical machine. Our test suite includes six different stress tests - a CPU intensive test, a memory intensive test, a disk intensive test, two network intensive tests (send and receive) and a fork bomb. We describe the design of our benchmark suite and present results of testing three flavors of virtualization systems -- an example of full virtualization (VMware Workstation), an example of paravirtualization (Xen) and two examples of operating system level virtualization (Solaris Containers and OpenVZ). We find that the full virtualization system offers complete isolation in all cases and that the paravirtualization system offers nearly the same benefits -- no degradation in many cases with at most 1.7% degradation in the disk intensive test. The results for operating system level virtualization systems are varied -- illustrating the complexity of achieving isolation of all resources in a tightly coupled system. Our results highlight the difference between these classes of virtualization systems as well as the importance of considering multiple categories of resource consumption when evaluating the performance isolation properties of a virtualization system.
To date, the measurement of user-perceived degradation of quality of service during denial of service (DoS) attacks remained an elusive goal. Current approaches mostly rely on lower level traffic measurements such as throughput, utilization, loss rate, and latency. They fail to monitor all traffic parameters that signal service degradation for diverse applications, and to map application quality-of-service (QoS) requirements into specific parameter thresholds. To objectively evaluate an attack's impact on network services, its severity and the effectiveness of a potential defense, we need precise, quantitative and comprehensive DoS impact metrics that are applicable to any test scenario. We propose a series of DoS impact metrics that measure the QoS experienced by end users during an attack. The proposed metrics consider QoS requirements for a range of applications and map them into measurable traffic parameters with acceptable thresholds. Service quality is derived by comparing measured parameter values with corresponding thresholds, and aggregated into a series of appropriate DoS impact metrics. We illustrate the proposed metrics using extensive live experiments, with a wide range of background traffic and attack variants. We successfully demonstrate that our metrics capture the DoS impact more precisely than the measures used in the past.
Measuring the indirect cost of context switch is a challenging problem. In this paper, we show our results of experimentally quantifying the indirect cost of context switch using a synthetic workload. Specifically, we measure the impact of program data size and access stride on context switch cost. We also demonstrate the potential impact of OS background interrupt handling on the measurement accuracy. Such impact can be alleviated by using a multi-processor system on which one processor is employed for context switch measurement while the other runs OS background tasks.
The evaluation of new network server architectures is usually performed experimentally using either a simulator or a hardware prototype. Accurate simulation of the hardware-software interface within the network subsystem is challenging due to the interactions of multiple asynchronous systems. Small timing inaccuracies in such a system can perturb the hardware and software state yielding potentially misleading results. Hardware prototypes show more promise because they are real-world implementations, not simplifications. Existing Ethernet network interface cards (NICs) are unsuitable for prototyping as they lack the capability and/or flexibility for advanced networking research. RiceNIC is an open network interface prototyping platform for public use. This reconfigurable and programmable Gigabit Ethernet NIC is designed to address the dilemma of how to accurately evaluate new ideas in network server architecture, and is built for use in experimental research and education. The flexibility and capability of RiceNIC has proven invaluable in recent research efforts.
Simulation has been the de facto standard method for performance evaluation of newly proposed ideas in computer architecture for many years. While simulation allows for theoretically arbitrary fidelity (at least to the level of cycle accuracy) as well as the ability to monitor the architecture without perturbing the execution itself, it suffers from low effective fidelity and long execution times. We (and others) have advocated the use of empirical experimentation on reconfigurable hardware for computer architecture performance assessment. In this paper, we describe an empirical performance assessment subsystem implemented in reconfigurable hardware and illustrate its use. Results are presented that demonstrate the need for the types of performance assessment that reconfigurable hardware can provide.
Predictors are inherent components of state-of-the-art microprocessors. Branch predictors are discussed actively from diverse perspectives. Performance of a branch predictor largely depends on the dynamic behavior of the executing program. Nevertheless, we have no effective metrics to represent the nature of program behavior quantitatively. In this paper, we introduce an information entropy idea to represent program behavior and branch predictor performance. Through simple application of Shannon's information entropy, we introduce new entropy, Branch History Entropy, which quantitatively represents the regularity level of program behavior. We show that the entropy also represents an index of prediction performance that is independent of prediction mechanisms. We further discuss branch predictor performance from a stereoscopic view of their typical organization. We propose two entropies: Table Reference Entropy and Table Entry Entropy. The former represents an unbalanced level of references of table entries. The latter offers the maximum expectation in prediction performance. We evaluated the proposed three entropies and prediction performance in various situations. Artificially generated branch patterns, as preliminary experiments, show an overview of the entropies and prediction performance. Subsequently, we present a comparison to the 2nd Championship Branch Predictor competition results and show the high potential of the proposed entropy. Finally, we present an actual view of our entropies and prediction performance as application results to SPEC CPU2000 benchmarks.
Combinatorial optimization problems expressed as Boolean constraint satisfaction problems (BCSPs) arise in several contexts, ranging from the classical unate set-packing problems to the binate minimum cover problems, including the Haplotype Inference by Pure Parsimony (HIPP) problem. These problems are being solved under different formulations and in different formats. Results of experiments that are reported can be seldom compared and replicated. This paper is not about 'the best BCSP solver'. Rather, it is a case study of how the scientific method can be applied to comparing the performance of not only BCSP solvers but also other solvers that address NP-hard problems. The approach is founded on two premises: (1) the introduction of instance isomorphs as families of equivalence classes, based on randomized replicas of a given reference instance, and (2) the use of isomorph classes for the design of reproducible experiments with BCSP solvers that includes performance testing hypotheses. We introduce a number of BCSP reference instances from different domains, generate isomorph classes and use various versions of cplex to characterize the solver performance and the isomorph classes themselves. This methodology may make it easier to (1) reliably improve the performance of combinatorial solvers and, (2) report results of experiments under the proposed schema.
Utility programs, which perform similar and largely independent operations on a sequence of inputs, include such common applications as compilers, interpreters, and document parsers; databases; and compression and encoding tools. The repetitive behavior of these programs, while often clear to users, has been difficult to capture automatically. We present an active profiling technique in which controlled inputs to utility programs are used to expose execution phases, which are then marked, automatically, through binary instrumentation, enabling us to exploit phase transitions in production runs with arbitrary inputs. We demonstrate the effectiveness and programmability of active profiling via experiments with six utility programs from the SPEC benchmark suite; compare to code and interval phases; and describe applications of active profiling to memory management and memory leak detection.
In this paper, we introduce EXACT, the EXperimental Algorithmics Computational Toolkit. EXACT is a software framework for describing, controlling, and analyzing computer experiments. It provides the experimentalist with convenient software tools to ease and organize the entire experimental process, including the description of factors and levels, the design of experiments, the control of experimental runs, the archiving of results, and analysis of results. As a case study for EXACT, we describe its interaction with FAST, the Sandia Framework for Agile Software Testing. EXACT and FAST now manage the nightly testing of several large software projects at Sandia. We also discuss EXACT's advanced features, which include a driver module that controls complex experiments such as comparisons of parallel algorithms.
Developing kernel-level file systems is a difficult task that requires a significant time investment. For experimental file systems, it is desirable to develop a prototype before investing the time required to develop a kernel-level file system. We have built a ptrace monitoring infrastructure for file system development. Because our system runs entirely in user-space, debugging is made easier and it is possible to leverage existing tested user-level libraries. Because our monitor intercepts all OS entry points (system calls and signals) it is able to provide more functionality than other prototyping techniques, which are limited by the VFS interface (FUSE) or network protocols (user-level NFS servers). We have developed several example file systems using our framework, including a pass-through layered file system, a layered encryption file system, and a user-level ISO9660 file system. We analyzed the complexity of our code using cyclomatic complexity and other metrics. We show savings for a pass-through file system of 53% compared to existing user-level pass-through file systems and a factor of 4.7 reduction for an in-kernel pass-through file system. Our performance evaluation demonstrates that our infrastructure has an acceptable overhead of 18.4% for a pass-through file system.
Context switching imposes a performance penalty on threads in a multitasking environment. The source of this penalty is both direct overhead due to running the context switch code and indirect overhead due to perturbation of caches. We calculate indirect overhead by measuring the running time of tasks that use context switching and subtracting the direct overhead. We also measure the indirect overhead impact on the running time of tasks due to processor interrupt servicing. Experiment results are presented for the Linux kernel running on an ARM processor based mobile device platform.
Experimental computer systems research typically ignores the end-user, modeling him, if at all, in overly simple ways. We argue that this (1) results in inadequate performance evaluation of the systems, and (2) ignores opportunities. We summarize our experiences with (a) directly evaluating user satisfaction and (b) incorporating user feedback in different areas of client/server computing, and use our experiences to motivate principles for that domain. Specifically, we report on user studies to measure user satisfaction with resource borrowing and with different clock frequencies in desktop computing, the development and evaluation of user interfaces to integrate user feedback into scheduling and clock frequency decisions in this context, and results in predicting user action and system response in a remote display system. We also present initial results on extending our work to user control of scheduling and mapping of virtual machines in a virtualization-based distributed computing environment. We then generalize (a) and (b) as recommendations for incorporating the user into experimental computer systems research.
We present an active measurement study of the routing dynamics induced by AS-path prepending, a common method for controlling the inbound traffic of a multi-homed ISP. Unlike other inter-domain inbound traffic engineering methods, AS-path prepending not only provides network resilience but does not increase routing table size. Unfortunately, ISPs often perform prepending on a trail-and-error basis, which can lead to suboptimal results and to a large amount of network churn. We study these effects by actively injecting prepended routes into the Internet routing system using the RIPE NCC RIS route collectors and observing the resulting changes from almost 200 publicly-accessible sources of BGP information. Our results show that our prepending methods are simple and effective and that a small number of ASes is often responsible for large amounts of the route changes caused by prepending. Furthermore, we show that our methods are able to reveal hidden prepending policies to prepending and tie-breaking decisions made by ASes; this is useful for further predicting the behavior of prepending.1
