Code plagiarism is an unfortunate problem which often plagues introductory level programming courses. In this poster, we study the relationship between plagiarism and student engagement using Submitty, our open-source course management, homework submission, and autograding system. We measure student participation through responses to live-lecture polls, posts in the course discussion forum, and daily and weekly office hours attendance patterns. While our assignments have strict deadlines, students are empowered to adapt deadlines by using penalty-free "late day" deadline extensions. Students who regularly participate in lecture polls are rewarded with additional late days. Submitty includes an opensource plagiarism detection module, Lichen, that detects software similarity that suggests academic dishonesty. Our implementation can be configured to examine every version of the assignment submitted by students and search for similarities between any version for any other student. We leverage the advantages of having a single, unified platform that combines submission timestamp data, course participation metrics, and code similarity data. Having many types of data stored in the same place allows us to effectively identify at-risk students earlier. We will present an exploration and analysis of data from an introductory level computer science course.

We present a system for the automated testing and grading of computer graphics applications. Our system runs, provides input to, and captures image and video output from graphical programming assignments. Instructors use a simple set of commands to script automated keyboard and mouse interactions with student programs at fixed times during execution. The resultant output, including plaintext standard output and mid-execution screenshots and GIFs, are displayed to the student to aid in debugging and ensure compliance with assignment specifications. Student output is automatically evaluated by basic text and image difference operations, or via an instructor-written validation method. We evaluate the success, implementation, and robustness of our design through deployment of this work in our university's senior undergraduate/graduate computer graphics course. In this course, students implement a variety of graphical assignments using OpenGL in C++. We summarize student feedback about the system gathered from anonymous end-of-term course evaluations. We provide anecdotal and quantitative evidence that the system improves student experience and learning by clarifying instructor expectations, building student confidence, and improving the consistency and efficiency of manual grading. This research has been implemented as an extension to Submitty, an open source, language-agnostic course management platform which allows automated testing and automated grading of student programming assignments. Submitty supports all levels of courses, from introductory to advanced special topics, and includes features for manual grading by TAs, version control, team submission, discussion forums, and plagiarism detection.

Fuzzing, testing a codebase against a set of randomly generated inputs, has become a promising model of testing across the industry due to its ability to reveal difficult to detect bugs. Separately, the use of randomized inputs when testing student code submissions removes the potential for student hard-coding behavior. Motivated by these factors, we present a solution for the automated generation of testcase inputs and expected outputs within Submitty, an open source automated grading system from Rensselaer Polytechnic Institute. We detail an enhanced workflow that allows instructors to provide our testing system with an assignment-specific input generation script and an assignment solution. The input generation script is run at student test-time, providing students with either entirely generated inputs or a combination of generated and hand-crafted testcases. The instructor solution is run against the same inputs to produce expected results. This model of testcase specification carries the benefit of simple regeneration of expected output files if an assignment's specification changes after submissions open or between semesters. We present preliminary results of the use of random input generation in our large introductory programming courses, and evaluate the ability of random inputs to curb student hardcoding behavior as it relates to an early submission incentive system, which grants students an extension for achieving a target assignment score early in the week an assignment is due. We examine random input generation's ability to reveal bugs in student submissions from previous semesters.

Grading web-based assignments poses many unique challenges when compared with other types of programming assignments. For introductory courses, grading involves not just validation of source code, but also performing some level of functional testing through a browser environment, where one clicks on content, and validates the browser state. In upper level courses, assignments increasingly use several different services, such as a web server and database, running concurrently, each potentially exposing ports for user access. Finally, for some assignments where students are encouraged to be creative, an instructor must then be able to view and interact with the running code, which has historically meant downloading, setting it up on their local machine, and running it -- which can prove burdensome. In this work, we present a system that can perform the task of automated grading, create a web-accessible environment on the server for the instructor to manually grade the assignment, and that can scale to handle many concurrent instances of both. To evaluate the effectiveness of our system, we will demonstrate its usage within the context of several assignments ranging over several different levels and courses of our Information Technology curriculum.

Submitty is an open-source course management platform for assignment submission with automated testing and grading, immediate feedback with an option for resubmission, and manual grading from course instructors and TAs. We have added an integrated discussion forum to the Submitty environment, which enables the students to communicate with their peers and have a public dialogue with the teaching staff outside of the classroom. Compared to a collection of closed-source external applications, our single-system login for distribution of course materials, assignment submission, and discussion forum participation is more convenient and accessible for students and simplifies administrative tasks. Students use the forum to ask questions about course logistics, homework help, exam review problems, and Submitty-related questions. The forum supports image and code segment posts, which facilitates targeted debugging help and sharing of additional reference material. An instructor-moderated forum is especially helpful in larger classes where students can feel it is otherwise difficult to get their questions answered. Every thread is tagged with course-specific categories and we provide full-text search of the forum. The forum becomes a curated collection of relevant frequently asked questions, which reduces the number of duplicate questions that must be answered by the staff. The forum includes customizable system and email notifications for important events such as: instructor announcement, assignment posted, team invitation, forum question answered, grades released, etc. Our current work focuses on private channels to facilitate team assignments (which can be monitored for participation by staff) and a grading interface for discussion-based assignments within the forum.

With the continued growth of enrollment within computer science courses, it has become an increasing necessity to utilize autograding systems. These systems have historically graded assignments through either a jailed sandbox environment or within a virtual machine (VM). For a VM, each submission is given its own instantiation of a guest operating system and virtual hardware that runs atop the host system, preventing anything that runs within the VM communicating with any other VM or the host. However, using these VMs are costly in terms of system resources, making it less than ideal for running student submissions given reasonable, limited resources. Jailed sandboxes, on the other hand, run on the host itself, thus taking up minimal resources, and utilize a security model that restricts the process to specified directories on the system. However, due to running on the host machine, the approach suffers as new courses utilize autograding and bring their own set of potentially conflicting requirements for programming languages and system packages. Over the past several years, \em containers have seen growing popularity in usage within the software engineering industry as well as within autograding systems. Containers provide similar benefits of isolation as a VM while maintaining similar resource cost to running within a jailed sandbox environment. We present the implementation of both a jailed sandbox and container-based autograder, compare the running time and memory usage of the two implementations, and discuss the overall resource usage.

We present a container-based system to automatically run and evaluate networked applications that implement distributed algorithms. Our implementation of this design leverages lightweight, networked Docker containers to provide students with fast, accurate, and helpful feedback about the correctness of their submitted code. We provide a simple, easy-to-use interface for instructors to specify networks, deploy and run instances of student and instructor code, and to log and collect statistics concerning node connection types and message content. Instructors further have the ability to control network features such as message delay, drop, and reorder. Running student programs can be interfaced with via stream-controlled standard input or through additional containers running custom instructor software. Student program behavior can be automatically evaluated by analyzing console or file output and instructor-specified rules regarding network communications. Program behavior, including logs of all messages passed within the system, can optionally be displayed to the student to aid in development and debugging. We evaluate the utility of this design and implementation for managing the submission and robust and secure testing of programming projects in a large enrollment theory of distributed systems course. This research has been implemented as an extension to Submitty, an open source, language-agnostic course management platform with automated testing and automated grading of student programming assignments. Submitty supports all levels of courses, from introductory to advanced special topics, and includes features for manual grading by TAs, version control, team submission, discussion forums, and plagiarism detection.

Prior education research, including Computer Science, has established that students will attempt to cheat and violate academic integrity, with one of the more common forms being code plagiarism. The majority of existing tools for software plagiarism are closed source, requiring instructors to use them in a prescribed configuration and sending student code to a third-party server for analysis. At the core of this analysis is the need to perform a language-specific tokenization of the input program and then to use "digital fingerprinting" on the code to identify significant markers. This has required developers to write their own parser for each supported language, which is time-consuming to create and keep up-to-date, and thus a barrier to creation of these tools. Instead we bootstrap new languages into our plagiarism system by leveraging the "Language Server Protocol", an initiative to create open-source parsers and tokenizers for many languages (principally to be used within a range of popular IDEs). In this poster, we present our work on Lichen, the open source plagiarism detection tool that is integrated into the Submitty course management platform we use at Rensselaer Polytechnic Institute. This tool is a pipeline of modules for the specific tasks of tokenizing, fingerprinting, and then comparing the fingerprints for any number of files. Through this, a similarity score is generated for pairs of files, and these are used to help instructors determine to what extent code plagiarism has occurred.

A flexible late submission policy can be used to help address breaches of academic integrity and student stress as they relate to programming assignments. To this end, we examine the use of late days -- earned, penalty free deadline extensions -- during our large format, two-semester introductory programming sequence. In particular, we examine correlations between patterns of student assignment submission and student plagiarism. Lexical-analysis-based techniques are used to identify probable cases of plagiarism across student submissions. Furthermore, we present metrics for estimating student stress at submission time. Our metric utilizes the automated grading score of the student"s current and previous submission attempts, the imminence of the assignment"s deadline, and the availability of late days. We corroborate these patterns with self-reported student surveys conducted at the end of the academic year.

Traditionally, automated testing and grading of student programming assignments has been done in either a jailed sandbox environment or within a virtual machine (VM). For a VM, each submission is given its own instantiation of a guest operating system (OS) running atop the host OS, with no ability for a given submission to affect anything outside the VM. However, using a VM is expensive in terms of system resource usages, especially for RAM and memory, making it less than ideal for solutions without unlimited resources. Jailed sandboxes on the other hand allow student submissions to run directly on the server. Sufficient security measures must be implemented to ensure that students cannot access each other's submissions or the server at large, and must prevent runaway programs, over-utilization of system resources. Jailed sandboxes have a larger attack vector than VMs. Within the past several years, container systems have been gaining popularity and usage within the computer science industry, primarily through solutions such as Docker. These containers give similar security protections as a VM, but with better performance due to being able to utilize of resources installed within the host OS and other containers. However, containers do not have the full isolation of a VM, and thus implementing Docker for autograding ends up facing its own set of security concerns, as well as with the increased system resource usage. In this poster, we will analyze how well containers work, measuring system resources and throughput of submissions of containers against the traditional jailed environment.

Submitty is an open source programming assignment submission system from the Rensselaer Center for Open Source Software (RCOS) at Rensselaer Polytechnic Institute (RPI) accessed via an online interface. Submitty allows students to submit their code through file upload or version control, such as an internal Git/SVN server or Github, where it is then tested with a highly configurable and customizable automated grader. For each assignment, instructors can specify whether or not students can work in teams. For team assignments, the instructor can either assign teammates or allow the students to choose. In addition to the auto-grading for submissions, Submitty supports human grading. The human graded rubric is developed by the graders as they work, allowing reuse of common feedback messages and partial credit points. The rubric can be searched and modified during and after grading is complete for consistency. By default, grading is handled by instructors and TAs who are assigned to sections of students, which can be rotated through the semester. However, an instructor can choose to incorporate peer grading, which will allow students to anonymously view and submit grades for each other, receiving multiple peer grades per assignment. Submitty has been used at RPI for several years for a variety of courses, serving over 1500 students and 50 instructors and TAs each semester, and has recently been used by several other universities. We will present "case studies" of assignment configurations for autograding and manual grading and demonstrate the grading interface in support of team submissions and peer grading.

With surging enrollment in Computer Science courses at both the introductory and advanced level, it is critical to leverage automated testing and grading to ensure consistent assessment of student learning. Program analysis tools allow us to streamline the grading process so instructors and TAs can spend more time teaching, one-on-one tutoring, and mentoring students. We present complex use cases of automated assignment testing and grading within the open-source homework submission system, Submitty. Students receive immediate and detailed feedback from the automated grader, and can resubmit to correct errors. Submitty uses custom-built grading tools, including difference checking of plaintext program output, instructor authored assignment-specific custom graders, and static analysis tools that reason about program structure. In addition, it employs a variety of external tools, including version control (Git and SVN), unit testing frameworks (JUnit), memory debugging tools (Valgrind and DrMemory), and code coverage tools (Emma). In this poster we describe our experience with memory debugging and code coverage tools, and outline plans to include immutability inference and verification.

Many computer science programs have capstone experiences or project courses that allow students to integrate knowledge from the full breadth of their major. Such capstone projects may be student-designed, instructor-designed, designed in conjunction with outside companies, or integrated with ongoing free and open source (FOSS) projects. The literature shows that the FOSS approach has attracted a great deal of interest, in particular when implemented with projects that have humanitarian goals (HFOSS). In this article, we describe five unique models from five distinct types of institutions for incorporating sustained FOSS or HFOSS (alternatively H/FOSS) project work into capstone experiences or courses. The goal is to provide instructors wishing to integrate open source experiences into their curriculum with additional perspectives and resources to help in adapting this approach to the specific needs and goals of their institution and students. All of the models presented are based on sustained engagement with H/FOSS projects that last at least one semester and often more. Each model is described in terms of its characteristics and how it fits the needs of the institution using the model. Assessment of each model is also presented. We then discuss the themes that are common across the models, such as project selection, team formation, mentoring, and student assessment. We examine the choices made by each model, as well as the challenges faced. We end with a discussion how the models have leveraged institutional initiatives and collaborations with outside organizations to address some of the challenges associated with these projects.

Submitty (http://submitty.org) is an open source programming assignment submission system from the Rensselaer Center for Open Source Software (RCOS) at Rensselaer Polytechnic Institute (RPI). Students can submit their code via a web interface in a variety of ways, where it is then tested with a highly configurable and customizable automated grader. Students receive immediate feedback from the grader, and can resubmit to correct errors as needed. Through an online interface, TAs can access detailed grading results and supplement the automated scores with manual grading (numeric and written feedback) of overall program structure, good use of comments, reasonable error checking, etc. and any non-programming components of the assignment. The instructor can also configure the system to allow for a configurable late day policy on a per assignment and per student basis. We currently use Submitty in eight different courses (spanning from introductory through advanced topics) serving over 1500 students and 35+ instructors and TAs each week. We will present a range of "case study" assignment configurations in a hands-on demo, going from simple through complex, using a variety of different automated grading methods including per-character and per-line output difference checkers, external unit testing frameworks (such as JUnit), memory debugging tools (Valgrind and DrMemory), code coverage (e.g., Emma), static analysis tools, and custom graders. Submitty can be customized per test case as appropriate to apply resource limits (running time, number of processes, output file size, etc.) and to display or hide from students the program output, autograding results, and testing logs.

We present a non-grade-penalty late day policy used in many of the large lecture, required courses in our computer science department. We study the effectiveness of this late day policy in reducing student stress, distributing demand for teaching assistant resources in peak hours before the homework deadline, and in maintaining or improving student understanding and homework grades. A complex late day policy can be efficiently implemented and managed within our open-source homework submission system that utilizes automated testing and grading, allowing students to submit and resubmit homeworks as they make progress on the assignment.

Student experience in introductory computer science classes can be enhanced by applying static analysis techniques to automatically grade assignments. At Rensselaer Polytechnic Institute (RPI), introductory computer science classes (using Python) exceed 650 students in size. As resources are limited, it is infeasible to have teaching staff individually examine each student's answer for small in-lecture exercises; however, qualitative data regarding student code independent from execution is still valuable (and in some cases required) to assess progress. When static analysis utilities were made available to instructors and integrated with automatic assignment testing, instructors were able to judge student performance and provide feedback at a scale that would otherwise be infeasible. There are clear advantages to applying static analysis techniques in comparison to less sophisticated methods (e.g. regular-expression based search). For one, students are unable to subvert grading by placing certain keywords within comments or string literals. Static analysis can also be applied to easily grade students on patterns that would be nontrivial to detect using a more naive method, for example in enforcing a rule that all member variables of a C++ class must be private, or verifying that a function takes the appropriate number and type of arguments.

We present a method for identifying inexpensive, off-the-shelf laser pointers in a multi-user interaction environment on large-scale displays. We identify a laser pointer's personality, a measure of its output in a particular context. Our method requires a set of inexpensive and unmodified green lasers, a large screen, a projector, and a camera with an infrared (IR) filter. The camera detects the IR spillover from the green laser beam, while ignoring color information projected onto the screen. During a calibration phase, a radial histogram of each laser's IR spillover are used to represent the laser's personality. Our system is able to identify the spots of a specific laser, allowing multiple users to simultaneously interact in the environment. In addition, we present a series of applications that take advantage of tracked and identified laser pointers to demonstrate large-scale, multi-user interactions.

The objective of this research is to develop tools that would improve the understanding of the process of levee failure because of erosion and reduce the risk of failure. Hydraulic erosion is a complicated phenomenon and depends on many different parameters. To improve design criteria for levees, embankments, and earthen structures, the development of realistic computer models that can simulate the erosion process is necessary. Verification of these computer simulations, as with any simulation, is a necessity. In this research, a large number of physical levee erosion tests were performed at 1g and at high g's using a geotechnical centrifuge. Centrifuge tests were performed to simulate real (prototype) size levees, and thus to obtain a more realistic model. The erosion was modeled physically in detail. Conventional three-dimensional scanning was used to precisely verify the calculated dimensions of initial and final computer model geometries, but did not yield interim data or measurements of the quantity of eroded soil during the tests. A Kinect device was used to scan and evaluate the volume of eroded soil and variation of the shape of the channels as a function of time. Three-dimensional images were obtained, and variations of different parameters were plotted. Various quantities were measured as a function of time. Based on recorded videos and pictures taken during the tests, it was discovered that the Kinect results agreed well with the physical models. The Kinect is a low-cost sensor, and enables the measurement of the rate of soil erosion, which, if done at all, usually requires expensive equipment. The Kinect device was also used in the centrifuge experiments, and functioned well in the high g environment. It is believed to be the first use of a Kinect device in a centrifuge. The application of this method in other laboratory experiments was also investigated.

The Rensselaer Polytechnic Institute (RPI) Homework Submission Server is an ongoing open source project used to collect, compile, and automatically grade programming homeworks for students in our introductory and sophomore computer science classes. It allows viewing of homework, lab, test, and overall grades and late submissions and excused absences on homework. Our first hypothesis is that an electronic submission server is the preferred way for students to submit their coding homeworks because it provides immediate feedback about the correctness of their code and ensures fair, consistent grading since their code is compiled and run with the same test cases, on the same computer. Our second hypothesis is that students appreciate courses with a flexible policy for late submission of homeworks, allowing them to use a specific number of "late days" throughout the semester without penalty. We recently conducted a survey to test these hypotheses and will incorporate specific feedback from the students as we continue development and expand the server to more courses at RPI and other universities.