The Standard Performance Evaluation Corporation (SPEC) CPU benchmark has been widely used as a measure of computing performance for decades. The SPEC is an industry-standardized, CPU-intensive benchmark suite and the collective data provide a proxy for the history of worldwide CPU and system performance. Past efforts have not provided or enabled answers to questions such as, how has the SPEC benchmark suite evolved empirically over time and what micro-architecture artifacts have had the most influence on performance?—have any micro-benchmarks within the suite had undue influence on the results and comparisons among the codes?—can the answers to these questions provide insights to the future of computer system performance? To answer these questions, we detail our historical and statistical analysis of specific hardware artifacts (clock frequencies, core counts, etc.) on the performance of the SPEC benchmarks since 1995. We discuss in detail several methods to normalize across benchmark evolutions. We perform both isolated and collective sensitivity analyses for various hardware artifacts and we identify one benchmark (libquantum) that had somewhat undue influence on performance outcomes. We also present the use of SPEC data to predict future performance.

Cybersecurity education has been emphasized by several national organizations in the United States, including the National Academy of Engineering, which recognizes securing cyberspace as one of the 14 Engineering Grand Challenges. To prepare students for such challenges and to enhance cybersecurity education opportunities at our large research university, we implemented an NSF-funded cybersecurity education project. This project is a collaborative effort between faculty and graduate students in the Engineering Education, Computer Science (CS) and Computer Engineering (CPE) departments at a major US research university. In this effort, we integrated cybersecurity learning modules into multiple existing core CS and CPE courses following Jerome Bruner's spiral-theory model, which has previously been used to reformulate several academic curricula. In this paper, we present our cybersecurity curriculum initiative, describe the spiral-theory based process we developed to implement the curriculum and provide an in-depth description of four reusable cybersecurity learning modules that we developed. A core tenet of spiral theory holds to revisit topics as students advance through their curriculum. This work applies this approach to Cybersecurity education by carefully designing the learning objectives of the modules and its contents. For evaluating these learning modules we implemented pre and post-tests to assess students' technical knowledge, their perceptions towards the modules' learning objectives, and how it influenced their motivation to learn cybersecurity. Our findings are overwhelmingly positive and the students' feedback has helped us improve these learning modules. Since its inception, our initiative has educated more than $2\,000$ students and is currently being used to revise the affected courses' syllabi.

Lightning talks of EduHPC are a venue where HPC educators discuss work in progress. This paper summarizes the EduHPC 2020 lightning talks, which cover four very different areas: (i) The simulation-based pedagogy of the EduWRENCH project, including motivations for using simulation to teach High Performance Computing, the design principles underlying EduWRENCH modules, a survey of the available modules, a look at a particular module, plus a conclusion including lesson learned thus far and future plans. (ii) The use of the software-tuning component from Student Cluster Competitions in the HPC master’s program at the University of Liverpool. (iii) Steps being taken by the Computer Systems Genome Project at Virginia Tech to foster a community atmosphere among the diverse students working to catalog the lineage of computer system performance over time. (iv) A 3-semester master’s degree program titled Computational Engineering, focused on HPC training, being offered at the University of Warsaw.

Performance variability can have a significant impact on many applications of computing. Cloud computing, high performance computing, and computer security communities each exert considerable effort managing and analyzing variability throughout the system stack. This work presents and evaluates a methodology for predicting precise characteristics of the computational performance variability of an input/output (I/O) application over varying system configurations. Results demonstrate that the presented methodology is capable of precisely modeling performance variability, which could allow applications that tighten service level agreements, maximize computational throughput, and obfuscate system configurations against malicious users.

A rapid increase in the quantity of data available is allowing all fields of science to generate more accurate models of multivariate phenomena. Regression and interpolation become challenging when the dimension of data is large, especially while maintaining tractable computational complexity. This paper proposes three novel techniques for multivariate interpolation and regression that each have polynomial complexity with respect to number of instances (points) and number of attributes (dimension). Initial results suggest that these techniques are capable of effectively modeling multivariate phenomena while maintaining flexibility in different application domains.

The current emphasis on cybersecurity worldwide, demonstrates the importance of this topic. This poster describes a unique NSF funded project that aims to create cybersecurity education opportunities at Virginia Tech (VT). It is a collaborative effort among faculty and graduate students in the Engineering Education, Computer Science (CS), Electrical and Computer Engineering (includes two majors, Electrical Engineering (EE) and Computer Engineering (CPE)) Departments, and the Hume Center in the College of Engineering at VT. The goal is to integrate cybersecurity modules into eight required CS and CPE courses, from freshman to junior levels, utilizing Jeremy Bruner's spiral-theory-based1,2 curriculum model. A spiraling theme of "handling threats to software for securing information" is chosen that will be returned to repeatedly as learners advance in their knowledge and intellectual capacity. Cybersecurity goals of the Confidentiality, Integrity, Availability, Authenticity, Anonymity, Assurance (CIA/AAA) triad, as appropriate for various academic levels, are adopted to develop the cybersecurity modules. Each module engages students in an authentic activity that reinforces the cybersecurity concepts. The project includes an engineering education research component, which is focused on evaluating the effectiveness of the curriculum in enhancing students' knowledge, skills, and motivation in cybersecurity concepts. The first year of the project has been completed by introducing cybersecurity modules into four courses (CS: Introduction to Software Design, and Software Design and Data Structures, and CPE: Engineering Problem Solving with C++, and Data Structures and Algorithms) impacting ~1600 students. The details of curriculum development, implementation and, preliminary findings of the research will be presented.

Computer systems form the backbone of computing from very small, mobile devices to the huge datacenters that power the digital economy. These systems often exhibit large degrees of variability in their performance that is little understood, but such variability threatens to severely diminish the effectiveness of critical systems upon which society relies. Funded by a large NSF grant, the VarSys project at Virginia Tech researches the sources of variability in computer systems and develops methods to overcome it. We believe it is crucial to raise awareness of the phenomena surrounding variability in computer systems at the undergraduate level. Towards this end, we are connecting the research techniques developed as part of this NSF award to ongoing classroom projects in a core systems course. Our key insight is to expose students to the phenomenon as it occurs in the systems software modules (e.g. a memory allocator, a fork-join thread pool) they are themselves developing in the course. We have implemented a web-based system that allows students to submit their own systems-level code to a specialized cluster which then benchmarks it while systematically varying a number of ordinal and categorical variables. These variables reflect environmental factors that can influence the performance of complex systems. Students are then presented with a visual statistical analysis of the results and asked to interpret those results. We have successfully deployed this system in 2 semesters to over 250 students and collected student data about their experience with this system and are documenting our progress towards these important learning objectives.

Computer systems form the backbone of computing from very small, mobile devices to the huge datacenters that power the digital economy. These systems often exhibit large degrees of variability in their performance that is little understood, but such variability threatens to severely diminish the effectiveness of critical systems upon which society relies. Funded by a large NSF grant, the VarSys project at Virginia Tech researches the sources of variability in computer systems and develops methods to overcome it. We believe it is crucial to raise awareness of the phenomena surrounding variability in computer systems at the undergraduate level. Towards this end, we are connecting the research techniques developed as part of this NSF award to ongoing classroom projects in a core systems course. Our key insight is to expose students to the phenomenon as it occurs in the systems software modules (e.g. a memory allocator, a fork-join thread pool) they are themselves developing in the course. We have implemented a web-based system that allows students to submit their own systems-level code to a specialized cluster which then benchmarks it while systematically varying a number of ordinal and categorical variables. These variables reflect environmental factors that can influence the performance of complex systems. Students are then presented with a visual statistical analysis of the results and asked to interpret those results. We have successfully deployed this system in 2 semesters to over 250 students and collected student data about their experience with this system and are documenting our progress towards these important learning objectives.

When interpolating computing system performance data, there are many input parameters that must be considered. Therefore, the chosen multivariate interpolation model must be capable of scaling to many dimensions. The Delaunay triangulation is a foundational technique, commonly used to perform piecewise linear interpolation in computer graphics, physics, civil engineering, and geography applications. It has been shown to produce a simplex based mesh with numerous favourable properties for interpolation. While computation of the two- and three-dimensional Delaunay triangulation is a well-studied problem, there are numerous technical limitations to the computability of a high-dimensional Delaunay triangulation. This paper proposes a new algorithm for computing interpolated values from the Delaunay triangulation without computing the complete triangulation. The proposed algorithm is shown to scale to over 50 dimensions. Data is presented demonstrating interpolation using the Delaunay triangulation in a real world high performance computing system problem.

Each of high performance computing, cloud computing, and computer security have their own interests in modeling and predicting the performance of computers with respect to how they are configured. An effective model might infer internal mechanics, minimize power consumption, or maximize computational throughput of a given system. This paper analyzes a four-dimensional dataset measuring the input/output (I/O) characteristics of a cluster of identical computers using the benchmark IOzone. The I/O performance characteristics are modeled with respect to system configuration using multivariate interpolation and approximation techniques. The analysis reveals that accurate models of I/O characteristics for a computer system may be created from a small fraction of possible configurations, and that some modeling techniques will continue to perform well as the number of system parameters being modeled increases. These results have strong implications for future predictive analyses based on more comprehensive sets of system parameters.

Following an exhaustive set of experiments, we identify slowdowns in I/O performance that occur when processor power and frequency are increased. Our initial analyses indicate slowdowns are more likely to occur and more acute when the number of parallel I/O threads increases and the variability between runs is high. We use a micro benchmark-driven methodology to simplify isolation of the root causes of I/O performance loss. We classify the observed performance loss into two categories: file synchronization and file write delays. We introduce LUC, a runtime system to Limit the Unintended Consequences of power scaling and dynamically improve I/O performance. We demonstrate the effectiveness of the LUC system running on two platforms for two critical parallel transaction-oriented workloads including a mail server (vermeil) and online transaction processing (lotp).

Libraries have long been adding content and customizations to vendor-provided web-based search interfaces, including discovery systems such as ProQuest’s Summon(™). Unlike solutions based on using an API, these approaches augment the vendor-designed user interface using library-provided JavaScript code. Recently, vendors have been implementing such user interfaces using client-centric model-view-controller (MVC) frameworks such as AngularJS, which are characterized by the use of modern software engineering techniques such as domain-specific markup, data binding, encapsulation, and dependency injection. Consequently, traditional approaches such as reverse-engineering the document model (DOM) have become more difficult or even impossible to use because the DOM is highly dynamic, the templates used are difficult to discern, the vendor-provided JavaScript code is both encapsulated and partially obfuscated, and the data binding mechanisms impose a strict separation of model and view that discourages direct DOM manipulation. In fact, practitioners have started to complain that AngularJS-based websites such as Summon 2.0 are very difficult to enhance with custom content in a robust and efficient manner. In this article, we show how to reverse-engineer the AngularJS-based Summon 2.0 interface to discover the modules, directives, controllers, and services it uses, and we explain how we can use AngularJS’s built-in mechanisms to create new directives and controllers that integrate with and augment the vendor-provided ones to add desired customization and interactions . We have implemented several features that demonstrate our approach, such as a click-recording script, COinS and facet customization, and the integration of eBook public notes. Our explanation and code should be of direct use for adoption or as examples for other Summon 2.0 customers, but they may also be useful to anyone faced with the need to add enhancements to other vendor-controlled MVC-based sites.

Most operating systems provide protection and isolation to user processes, but not to critical system components such as device drivers or other system code. Consequently, failures in these components often lead to system failures. VirtuOS is an operating system that exploits a new method of decomposition to protect against such failures. VirtuOS exploits virtualization to isolate and protect vertical slices of existing OS kernels in separate service domains. Each service domain represents a partition of an existing kernel, which implements a subset of that kernel's functionality. Unlike competing solutions that merely isolate device drivers, or cannot protect from malicious and vulnerable code, VirtuOS provides full protection of isolated system components. VirtuOS's user library dispatches system calls directly to service domains using an exceptionless system call model, avoiding the cost of a system call trap in many cases.We have implemented a prototype based on the Linux kernel and Xen hypervisor. We demonstrate the viability of our approach by creating and evaluating a network and a storage service domain. Our prototype can survive the failure of individual service domains while outperforming alternative approaches such as isolated driver domains and even exceeding the performance of native Linux for some multithreaded workloads. Thus, VirtuOS may provide a suitable basis for kernel decomposition while retaining compatibility with existing applications and good performance.

LibX is a platform that allows libraries to create customized web browser extensions that simplify direct access to library resources and services. LibX provides multiple user interfaces, including popups, context menus, and contextualized cues to direct the user’s attention to these resources. LibX is supported by two toolbuilder applications - the Edition Builder and the LibApp Builder – which allow anyone to create, manage, and share LibX configurations and applications. These tools automate the process of software creation and distribution, allowing librarians to become software distributors. This chapter provides background and history of the LibX project, as well as in-depth analysis of the design and use of the LibX Edition Builder that has helped enable its success.

This experience report describes using a validated survey instrument to measure changes in student attitudes toward computing across a CS1 course. The choice of the survey instrument is described, with links to online resources for several existing instruments. The setup of a within-subjects design using preand post-tests at each end of a semester-long programming course allows changes in attitudes to be assessed. Results from analyzing 197 surveys received over a two-semester period are presented. Most attitude measures were quite high. The only significant change was an increase in the perceived importance of computing among the students. However, student aversion scores were a significant predictor of student success. Lessons learned are presented along with future plans.

CloudBrowser is a web application framework that supports the development of rich Internet applications whose entire user interface and application logic resides on the server, while all client/server communication is provided by the framework. CloudBrowser thus hides the distributed nature of these applications from the developer, creating an environment similar to that provided by a desktop user interface library. CloudBrowser preserves the user interface state in a server-side virtual browser that is maintained across visits. Unlike other server-centric frameworks, CloudBrowser's exclusive use of the HTML document model and associated JavaScript execution environment allows it to exploit existing client-side user interface libraries and toolkits while transparently providing access to other application tiers. We have implemented a prototype of CloudBrowser as well as several example applications to demonstrate the benefits of its server-centric design.

Improving recruitment and retention depends on linking activities to real-world contexts that illustrate the social and societal impact of computer science. The sleek interaction and underlying technologies that define Web 2.0 applications offer a unique opportunity to teach CS concepts in a real-world context. Learn how students can create engaging Web 2.0 applications using simple Java programming and basic XHTML, without learning JavaScript or AJAX. See a full set of CS1 assignments leveraging this strategy. See how students can write a personal "Facebook-lite" they can show to friends. Play with live demos yourself. Leave with new assignment ideas.

Virtualization is a powerful technique used for variety of application domains, including emerging cloud environments that provide access to virtual machines as a service. Because of the interaction of virtual machines with multiple underlying software and hardware layers, the analysis of the performance of applications running in virtualized environments has been difficult. Moreover, performance analysis tools commonly used in native environments were not available in virtualized environments, a gap which our work closes. This paper discusses the challenges of performance monitoring inherent to virtualized environments and introduces a technique to virtualize access to low-level performance counters on a per-thread basis. The technique was implemented in perfctr-xen, a framework for the Xen hypervisor that provides an infrastructure for higher-level profilers. This framework supports both accumulative event counts and interrupt-driven event sampling. It is light-weight, providing direct user mode access to logical counter values. perfctr-xen supports multiple modes of virtualization, including paravirtualization and hardware-assisted virtualization. perfctr-xen applies guest kernel-hypervisor coordination techniques to reduce virtualization overhead. We present experimental results based on microbenchmarks and SPEC CPU2006 macrobenchmarks that show the accuracy and usability of the obtained measurements when compared to native execution.