Mutation testing measures a test suite's ability to detect bugs by inserting bugs into the code and seeing if the tests behave differently. Mutation testing has recently seen increased adoption in industrial and open-source software but sees limited use in education. Some instructors use manually-constructed mutants to evaluate student tests and provide general automated feedback. Additional tutoring requires more intensive instructor interaction such as in office hours, which requires substantial resources at scale. Prior work suggests that students benefit from frequent, actionable feedback, and our work focuses on the challenge of leveraging automation to give students high-quality feedback when they need it. We deployed an automated hint system that provides instructor-written hints related to mutants that student-written tests do not detect. We evaluated our hint system in a controlled experiment across four assignments in two introductory programming courses, comprising 4,122 students. We also analyzed student test suite revisions and conducted a mixed-methods analysis of student hint ratings and comments collected by the automated hint system. We observed a small, statistically significant increase in the mean number of mutants detected by students who received hints (experiment group) compared to those who did not (control group). In 25% of instances where students received a hint, they detected the mutant in a single revision to their test suite. We conclude with recommendations based on our analysis as a starting point for instructors who wish to deploy this type of automated feedback.

A common practice in computer science courses is to evaluate student-written test suites against either a set of manually-seeded faults (handwritten by an instructor) or against all other student-written implementations (“all-pairs” grading). However, manually seeding faults is a time consuming and potentially error-prone process, and the all-pairs approach requires significant manual and computational effort to apply fairly and accurately. Mutation analysis, which automatically seeds potential faults in an implementation, is a possible alternative to these test suite evaluation approaches. Although there is evidence in the literature that mutants are a valid substitute for real faults in large open-source software projects, it is unclear whether mutants are representative of the kinds of faults that students make. If mutants are a valid substitute for faults found in student-written code, and if mutant detection is correlated with manually-seeded fault detection and faulty student implementation detection, then instructors can instead evaluate student test suites using mutants generated by open-source mutation analysis tools. Using a dataset of 2,711 student assignment submissions, we empirically evaluate whether mutation score is a good proxy for manually-seeded fault detection rate and faulty student implementation detection rate. Our results show a strong correlation between mutation score and manually-seeded fault detection rate and a moderately strong correlation between mutation score and faulty student implementation detection. We identify a handful of faults in student implementations that, to be coupled to a mutant, would require new or stronger mutation operators or applying mutation operators to an implementation with a different structure than the instructor-written implementation. We also find that this correlation is limited by the fact that faults are not distributed evenly throughout student code, a known drawback of all-pairs grading. Our results suggest that mutants produced by open-source mutation analysis tools are of equal or higher quality than manually-seeded faults and a reasonably good stand-in for real faults in student implementations. Our findings have implications for software testing researchers, educators, and tool builders alike.

Computer Science has a persistent lack of women's participation. In order to best effect change, we require a more fine-grain analysis of the gender disparity as it changes throughout an undergraduate Computer Science curriculum. In this paper, we use a quantitative approach to highlight, with greater specificity, the point in an undergraduate career where gender balance changes. We also examine the role of grades in students' decisions to stay in the course sequence. Our goal is to enable targeted interventions that will make Computer Science a more welcoming discipline. Our study examines 30,890 unique student records over ten years at a large, public research institution. The records include students who took a Computer Science course over the past ten years. The dataset contains information about gender, majors, minors, academic level, and GPA. The dataset also includes a record from each course taken by each student and their final grade. We observed a modest increase in women's participation in all Computer Science courses over the past ten years. Despite this increase, the gender disparity is still large. Through our analysis, we found that women consistently choose not to continue through the Computer Science sequence at a higher rate than men. This higher attrition could be linked to women receiving lower grades in most introductory CS courses despite having the same or higher GPAs than men. Our results reveal specific areas where intervention can be the most effective in changing the stubborn gender disparity in Computer Science.

An inclusive community is one factor in growing and maintaining a diverse student body. This proposal aims to measure changes in the sense of community among students as they progress through an introductory computing sequence. A five year program is aimed at increasing the enrollment and graduation rates of women and underrepresented minority (URM) students in computer science and engineering at a competitive public research institution. Three obstacles to diversity in computer science and computer engineering have been identified: stereotyped traits, perceived abilities, and learning environment. Identifying implicit bias and imposter syndrome as components of these obstacles, we include a series of class activities designed to lessen the impact of implicit bias and imposter syndrome on our students in large-enrollment introductory computer programming courses. One element of assessing the success of our program is to use entry and exit surveys to gauge the change in students’ perceptions of their abilities and learning environment. Previously, we investigated the difference between men’s and women’s perceptions of their abilities and the learning environments in introductory computing courses. We found a statistically significant association between gender and perception of self-efficacy, but not in how those perceptions change over the course of the term. We then investigated the differences between underrepresented minority (URM) students’ and non-URM students’ perceptions of their abilities and the learning environments in introductory computing courses. We found no statistically significant differences between URM and non-URM students; however, both groups showed slight decreases in feelings of inclusion for the courses included in the study. Our program wishes to increase feelings of inclusion; therefore, we propose to conduct a more nuanced investigation of individual paths through the introductory computer programming sequence at our institution. Our research question is: How do student’s sense of community change over time through an introductory computer programming sequence? Our data set is comprised of entry and exit survey data for five semesters of a three course introductory computing sequence. We plan to measure changes in survey responses from individual students as they progress from CS1 through CS2 to CS3. The survey data will be analyzed using mixed model ANOVA for repeated measures of questions on the student’s experiences in their undergraduate studies up to the point of when they took the survey. The results and analysis will be presented in this paper.

Computer science and software engineering courses commonly use automated grading systems to evaluate student programming assignments. These systems provide various types of feedback, such as whether student code passes instructor test cases. The literature contains little data on the association between feedback policies and student learning. This work analyzes the association between different types of feedback and student learning, specifically on the topic of software testing. Our study examines a second-semester computer programming course with a total of 1,556 students over two semesters. The course contained five programming projects where students wrote code according to a specification as well as test cases for their code. Students submitted their code and test cases to an automated grading system. These test cases were evaluated by running them against intentionally buggy instructor solutions. The first semester comprised the control group, while the second semester comprised the experiment group. The two groups received different kinds of feedback on their test cases. The control group was shown whether their tests were free of false positives. In addition to the same feedback as the control group, the experiment group was shown how many intentionally buggy instructor solutions their tests exposed. Our results measured the quality of student test cases for the control and experiment groups. After students in the experiment group completed two projects with additional feedback on their test cases, they completed a final project without the additional feedback. Despite not receiving additional feedback, their test cases were of higher quality, exposing on average 5% more buggy solutions than students from the control group. We found this difference to be statistically significant after controlling for GPA and whether students worked alone or with a partner.

Executive Summary This report summarizes our study of anomaly detection techniques using readings from the AC power cord on an infusion pump. Our goal was to measure the effectiveness of our power analysis techniques in identifying anomalous pump behavior that could include intentional tampering via cyber-physical attacks. Our system was able to detect simulated cyberattack scenarios with close to 100% accuracy. The tunable model is configurable to generate near-zero false positives.

Human vs. Automated Coding Style Grading in Computing Education Computer programming courses often evaluate student coding style manually. Static analysis tools provide an opportunity to automate this process. In this paper, we explore the tradeoffs of human style graders and general-purpose static analysis tools to evaluate student code. We investigate the following research questions: - Are human coding style evaluation scores consistent with static analysis tools? - Which style grading criteria are best evaluated with existing static analysis tools and which are more effectively evaluated by human graders? We analyze data from a second-semester programming course at a large research institution with 943 students enrolled. Hired student graders evaluated student code with rubric criteria such as “Lines are not too long” or “Code is not too deeply nested.” We also ran several static analysis tools on the same student code to evaluate the same criteria. We then analyzed the correlation between the number of static analysis warnings and human style grading score for each criterion. In our preliminary results, we see that static analysis tools tend to be more effective at evaluating objective code style criteria. We found a weak negative or no correlation between the human style grading score and number of static analysis warnings. Note that we expect student code with more static analysis warnings to receive fewer human style grading points. When comparing the “Lines are not too long” human style grading criterion to a related line-length static analysis inspection, we see a Pearson correlation score of r=-0.21. We also see trends in the distributions of human style grading scores that suggest human graders perform inconsistently. For example, 50% of students who received full human style grading points for the line-length criterion had 3 or more static analysis warnings from a related line-length inspection. Additionally, 23% of students who received no points on the same criterion had no static analysis warnings for the line-length inspection. We also found that some code style criteria are not well suited to the general-purpose static analysis tools we investigated. For example, none of the static analysis tools we investigated provide a robust way of evaluating the quality of variable and function names in a program. Some tools provide an inspection for detecting variable names that are shorter than a user-specified length threshold; however, this inspection fails to identify low-quality variable names that happen to be longer than the minimum allowed length. Furthermore, there are some common scenarios where a short variable name is acceptable by convention. Static analysis tools have the benefit of integration with an automated grading system, facilitating faster and more frequent feedback compared to human grading. The literature suggests that frequent feedback encourages students to actively improve on their work (Spacco et al. 2006). There is also evidence to suggest that increased engagement is most beneficial to students with less experience (Carini et al. 2006). Our results suggest that automated code quality evaluation could be one tool that benefits student learning in intro CS courses, helping most those students with least access to CS training pre-college. References - Carini, R.M., Kuh, G.D. & Klein, S.P. Res High Educ (2006) 47: 1. - Spacco, Jaime and Pugh, William. Helping students appreciate test-driven development (TDD). Proceedings of OOPSLA, pages 907–913, 2006.

"The only thing that scales with undergrads is undergrads". As Computer Science course enrollments have grown, there has been a necessary increase in the number of undergraduate and graduate teaching assistants (TAs, and UTAs). TA duties often extend far beyond grading, including designing and leading lab or recitation sections, holding office hours and creating assignments. Though advanced students, TAs need proper pedagogical training to be the most effective in their roles. Training strategies have widely varied from no training at all, to semester-long prep courses. We will explore the challenges of TA training across both large and small departments. While much of the effort has focused on teams of undergraduates, most presenters have used the same tools and strategies with their graduate students. Training for TAs should not just include the mechanics of managing a classroom, but culturally relevant pedagogy. The panel will focus on the challenges of providing "just in time", and how we manage both intra-course training and department or campus led courses.

Increasing participation of women and underrepresented minorities is a key challenge in the field of Computer Science Education. Balanced representation of these groups among teaching assistants in Computer Science courses influences recruitment and retention of underrepresented students. At the same time, the status-quo reduced participation of these students makes it more difficult to hire instructional staff from underrepresented groups. In this paper, we describe our experience evaluating candidates with teaching-demonstration videos, followed by in-person interviews, to hire a gender-balanced set of undergraduate TAs for a large-scale CS2 course. Our research goal is to quantitatively assess gender balance throughout the hiring process. Our initial applicant pool is just one-sixth women, but we found that women applicants perform better in our application process than men, resulting in a gender-balanced course staff without making hiring decisions based on the gender of applicants. We show that our approach results in a more gender-balanced teaching staff than hiring based on applicant GPA. We also use course-evaluation data to demonstrate that women perform as well as men as teaching assistants in CS2, and that the overall quality of our teaching assistants has remained high after the hiring-process change.

Diagnosing an inherited disease often requires identifying the pattern of inheritance in a patient's family. We represent family trees with genetic patterns of inheritance using hypergraphs and latent state space models to provide explainable inheritance pattern predictions. Our approach allows for exact causal inference over a patient's possible genotypes given their relatives' phenotypes. By design, inference can be examined at a low level to provide explainable predictions. Furthermore, we make use of human intuition by providing a method to assign hypothetical evidence to any inherited gene alleles. Our analysis supports the application of latent state space models to improve patient care in cases of rare inherited diseases where access to genetic specialists is limited.

Contribution: This paper provides evidence for the benefits of pair programming early in the curriculum on student performance later in the curriculum. It also confirms the short term benefits of partnerships at scale. Background: Engineers often work in teams, both in industry and in academia. Previous work has shown that partnered programming yields higher student performance during the course in which students partner. Research Questions: This paper investigates the long term effects of early curriculum pair programming on student performance in a following course. Specifically, do student partnerships impact long-term student performance in a later course? Are previously observed effects of partner programming robust to a larger sample size? Methodology: This quantitative analysis examines 2468 students in an introductory computer science sequence at a large, public research institution. The data set comprises two academic years and includes partnership participation, project and exam scores, withdraw rates, time between courses, GPA, and gender. Findings: A positive relationship is observed between partnering in an introductory course, and higher project scores in a future course, where all students worked alone (N=1003). Students with the lowest GPAs experienced the greatest benefits. Additionally, results with a large population of students confirm the observations of previous research, showing that partnerships are associated with an overall positive grade impact during the course in which the partnership takes place (N=2468).

These conditions are clinically and genetically heterogeneous. (Figure 1) Over 170 genes and thousands of different mutations have been implicated as diseasecausing in individuals with various retinal dystrophies.3 Identifying the genetic diagnosis for a given patient is important for a number of reasons. It can confirm the diagnosis, direct clinical management, provide a more accurate prognosis, inform genetic counseling, and identify patients for whom molecular-based therapy might be available.2

Computer scientists often work in teams on complex software projects and their education often includes group work or pair programming. In the literature, group work and pair programming have been shown to improve student learning outcomes. For example, students programming in pairs have been shown to produce better programs and have similar exam scores during the same semester. In this paper, we explore the impact of student partnerships in introductory programming projects on future student performance in subsequent courses. We examine the data from 2,234 students enrolled in introductory computer science courses at a major public research institution, including participation in group work and performance in two different courses. The first is a second-semester CS2 programming and introductory data structures course, with 4 programming projects and 2 exams. During this course, some students worked in partnerships while others worked alone. Students then advanced to a CS3 course, which covers data structures and algorithms. This course also consists of 4 independent programming projects and 2 exams. Both courses are part of the core computer science curriculum for all computer science majors and minors, and are considered gateway courses to specialized upper level courses. In our results, we observed an association between student partnerships and increased project performance during the same semester, confirming the observations of prior work. When examining student performance in the following CS course, our strongest statistically significant association was between students in the lowest GPA quartile and improved project scores.

Software-based medical devices enable fast product-development cycles, constructive information sharing, and configurable therapy delivery, resulting in better patient outcomes overall. An unfortunate drawback is that software is complicated and difficult to maintain correctly. Devices with inadequate software maintenance may pose operational risks to network security and patient safety and privacy. This paper describes a noninvasive approach to medical-device monitoring that can address some of the shortcomings of conventional approaches.Protecting software-based medical devices from malware infections or network-based mischief is a growing concern for clinical engineers and healthcare information technology (IT) practitioners. Unlike desktop PCs and laptops, medical devices often lack support for antivirus systems or operating-system patches, despite running off-the-shelf operating systems and commercial third-party software. Manufacturers have cited previous regulatory approval as a reason not to support software updates [1], despite the Food and Drug Administration's clarifications to the contrary [2].Medical devices are often in use for decades in clinical settings, during which time the software they are based on continues to change. For example, Microsoft Windows has undergone four major product revisions since the release of Windows XP in 2001, but new medical devices are shipped with Windows XP as recently as 2012 [3], and anecdotal evidence suggests many more are still in use [4]. Microsoft halted support for Windows XP in early 2014. Even when patches are available, administrators tend to emphasize functionality and efficacy over security and avoid applying patches [4] for fear of breaking systems or voiding warranties.Without adequate patching, the threat to connected devices increases with time as more vulnerabilities are discovered. Manufacturers have little incentive to retest devices once they are in the field [5], and testing is not guaranteed to catch vulnerabilities. While proactive manufacturers have been steadily improving their design and maintenance processes for new devices to prevent security holes and permit software patches, healthcare IT practitioners are often left with a mess they cannot effectively maintain.Third-party software also poses challenges for manufacturers and device owners. Devices often ship with commercial or open-source libraries that are maintained separately from a device's main code, often by completely separate teams. Popular libraries that are easy to use become widespread if they add new capabilities to devices. For example, the OpenSSL library for encrypting communications, which is compatible with a range of systems from embedded to server-class, appears in at least 74 different kinds of devices, each with different update mechanisms. A 2014 Internet scan for a particularly high-impact vulnerability called Heartbleed found that 56% of vulnerable devices were embedded systems [6], which are typically more difficult to update than PCs or servers. Modern applications bring together tens of libraries that must all be considered separate sources of potential security problems.The National Institute of Standards and Technology (NIST) recently published a medical-device security use case [5] describing the security risks of a modern medical device, an infusion pump with wireless network connectivity. Network connectivity gives this device a means of adjusting therapy, feeding data into electronic health records (EHRs), and working with centralized management tools. NIST also pointed out the risk of bad actors changing infusion rates to harm patients, a potentially dangerous criminal offense. Independent researchers have confirmed that commercially available devices are vulnerable to such tampering [7]. Malicious hackers may be tempted by a potential foothold into a hospital network [8], a desire to harm specific patients or other mischievous ends [9].The remainder of this brief summarizes an approach to medical-device monitoring that can be added to medical devices in postmarket scenarios, i.e., after they are deployed in clinical settings. Monitoring is a component of security, along with prevention and remediation. In light of the NIST use case, the example in this brief focuses on monitoring infusion pumps for unauthorized dosage changes.Our system, which draws inspiration from previously published work [10,11], comprises hardware and software dedicated to nonintrusive monitoring. We use machine learning to match a device's activity patterns to previously observed behaviors. The monitoring point for device behaviors is the AC power outlet, a common interface to many medical devices. For many kinds of plugged-in devices, distinct activities on the device correspond to distinct patterns of power consumption. In the case of an infusion pump, the relevant question is whether different infusion rates can be distinguished by examining power consumption.We studied three commercially available infusion pumps from two manufacturers: two large-volume pumps and a low-volume syringe-type pump. Each runs an off-the-shelf operating system. Our system flags anomalous behavior by comparing behavioral samples against a model derived from a training set. For each pump, we first established a set of training data by recording only the pump's normal behavior with our prototype hardware. We used a proprietary feature-engineering workflow to determine which properties of the collected signals revealed the most about the system's aggregate behavior. With a model derived from the training data, we commenced “live” measurement of the pump to compare behavioral samples to the trained model using the selected features. If a series of new measurements were sufficiently outside the normal range with respect to a sufficient number of features, our detectors automatically deemed the pump's behavior as anomalous.We qualified “normal” behavior on the two large-volume pumps as an infusion rate of 50 ml/hr, and on the syringe-type pump as an infusion rate of 0.1 ml/hr. Normal depends on the drug being delivered, the patient, and other clinical decision processes, but in keeping with the NIST threat model, we focused on scenarios in which an adversary would be able to change a low infusion rate to an inappropriately high infusion rate. We qualified an “abnormal” rate on the large-volume pumps as 999 ml/hr, the maximum configurable, and on the syringe-type pump as its maximum 20 ml/hr for a simulated morphine sulfate 5 ml/hr syringe.Over all the infusion pumps we tested, our tools automatically generated models that discriminated among the various infusion rates with low rates of false positives (1.0% and lower) and false negatives (close to 0.0%).Figure 1 is a visualization of an infusion pump anomaly detection model (a one-class support vector machine with two features) that our prototype toolchain produced. The X- and Y-axes plot the two features used by this model of a high-volume infusion pump. Each dot represents one measurement of the pump's power consumption over 100 s, with respect to two features we used to build the model. Normal data collected from the pump at a 50 ml/hr delivery rate (white dots) were used as training data. Next, we evaluated the model with new measurements of normal 50 ml/hr infusion rates (green dots), which fall inside the red boundary, indicating no anomaly. Finally, we evaluated the model on new measurements that simulated abnormal (attack) activity at a 999 ml/hr rate. These dots fall outside the red boundary, indicating an anomaly. Our error rate on the training data was 1.0%, with 0.0% error for normal test data (false positives) and 0.0% error on abnormal test data (false negatives). The trade-off between false negatives and false positives is configurable in the model. For a second high-volume pump with a similar training and testing regimen, the error on our training data was 0.1% (false positives), and error on the test data was 0.0% (false negatives).In addition to anomaly detection, we also built a regression model to predict the infusion rate using only features measured from the AC power line; the plot is omitted for space. The input to this model is the X-axis feature from Fig. 1, measured over 100 s. The output is the predicted infusion rate, which is a continuous value in contrast to the discrete normal versus abnormal output of the anomaly detector. The model suggests that infusion rate is correlated (R2 = 0.82) to the features we measured on the pump's AC power traces.Our preliminary study demonstrates that commercially available infusion pumps' patterns of AC power consumption are correlated with their infusion rates, suggesting that monitoring power at fine granularity is a potentially viable approach to postmarket security maintenance for these medical devices. Postmarket strategies cannot replace coherent patching and update strategies by manufacturers, but they can help decrease healthcare IT practitioners' reliance on manufacturers to provide updates in lockstep with known vulnerabilities in off-the-shelf components.This material is based upon work supported by the National Science Foundation under the SBIR Grant No. 1448440.

Verification is a critical bottleneck in the time to market of a new digital design. As complexity continues to increase, post-silicon validation shoulders an increasing share of the verification/validation effort. Post-silicon validation is burdened by large volumes of test failures, and is further complicated by root cause bugs that manifest in multiple test failures. At present, these failures are prioritized and assigned to validation engineers in an ad-hoc fashion. When multiple failures caused by the same root cause bug are debugged by multiple engineers at the same time, scarce, time-critical engineering resources are wasted. Our scalable bug triage technique begins with a database of test failures. It extracts defining features from the failure reports, using a novel, topology-aware approach based on graph partitioning. It then leverages unsupervised machine learning to extract the structure of the failures, identifying groups of failures that are likely to be the result of a common root cause. With our technique, related failures can be debugged as a group, rather than individually. Additionally, we propose a metric for measuring verification efficiency as a result of bug triage called Unique Debugging Instances (UDI). We evaluated our approach on the industrial-size OpenSPARC T2 design with a set of injected bugs, and found that our approach increased average verification efficiency by 243%, with a confidence interval of 99%.

Shared-memory chip-multiprocessor (CMP) architectures define memory consistency models that establish the ordering rules for memory operations from multiple threads. Validating the correctness of a CMP's implementation of its memory consistency model requires extensive monitoring and analysis of memory accesses while multiple threads are executing on the CMP. In this paper, we present a low overhead solution for observing, recording and analyzing shared-memory interactions for use in an emulation and/or post-silicon validation environment. Our approach leverages portions of the CMP's own data caches, augmented only by a small amount of hardware logic, to log information relevant to memory accesses. After transferring this information to a central memory location, we deploy our own analysis algorithm to detect any possible memory consistency violations. We build on the property that a violation corresponds to a cycle in an appropriately defined graph representing memory interactions. The solution we propose allows a designer to choose where to run the analysis algorithm: 1) on the CMP itself; 2) on a separate processor residing on the validation platform; or 3) off-line on a separate host machine. Our experimental results show an 83% bug detection rate, in our testbed CMP, over three distinct memory consistency models, namely: relaxed-memory order, total-store order, and sequential consistency. Finally, note that our solution can be disabled in the final product, leading to zero performance overhead and a per-core area overhead that is smaller than the size of a physical integer register file in a modern processor.

The exponentially growing complexity of modern processors intensifies verification challenges. Traditional pre-silicon verification covers less and less of the design space, resulting in increasing post-silicon validation effort. A critical challenge is the manual debugging of intermittent failures on prototype chips, where multiple executions of a same test do not yield a consistent outcome. We leverage the power of machine learning to support automatic diagnosis of these difficult, inconsistent bugs. During post-silicon validation, lightweight hardware logs a compact measurement of observed signal activity over multiple executions of a same test: some may pass, somemay fail. Our novel algorithm applies anomaly detection techniques similar to those used to detect credit card fraud to identify the approximate cycle of a bug's occurrence and a set of candidate root-cause signals. Compared against other state-of-the-art solutions in this space, our new approach can locate the time of a bug's occurrence with nearly 4x better accuracy when applied to the complex OpenSPARC T2 design.