The call for alternative grading practices has been made both inside and outside the computing education community. Various practices exist to provide assessment and feedback to students that do not rely strictly on points out of one hundred percent, weighted averages, high stakes assignments, and grading for behaviors instead of learning. However, modern classrooms, especially computer science classrooms, rely on a myriad of digital tools to organize and maintain the course structure. Tools like learning management systems, automatic grading systems, submission systems, and practice systems all exist for computing students and faculty to use to help support the learning of programming concepts. By and large, these systems all rely on an underlying mechanism of points and aggregating points for scoring. In the face of such technological choices, adopting alternative grading practices can prove challenging for instructors and confusing for students. In this position paper, we advocate addressing key research problems to make these systems easier to use with alternative grading practices. These include comprehensive support for categorical grading, comprehensive support for rework and resubmission, and improved protocols for communication of scores and feedback. We propose an extension to LTI to support the needs of alternative grading practices, and we provide an initial design for this LTI extension. We discuss current problems and potential solutions and challenge the community to work on these problems and consider the design of future systems to embrace grading approaches that go beyond just points-based scoring.

Computer science as a discipline is well-placed to research and adopt new pedagogical technologies as they are developed. Many popular software platforms that have been adopted across institutions have started as research projects in computer science programs. Additionally, because of the growth in demand for computer science as a program of study, CS instructors have been on the forefront of adopting new approaches and technologies to be able to meet the requirements of teaching at scale. However, being at this intersection creates challenges of its own. There are hundreds of tools and approaches, many with some level of research behind them. These tools come from vendors large and small, with costs both large and small, and with a wide variety of features that may or may not meet the needs of the instructor and the approach they want to pursue. From infinite diversity comes infinite combinations, with the attendant number of questions that instructors must answer when designing their courses. What tools do we use? How do we pick? How can we integrate them? Can we get support? Will my institution approve them? How can I adopt a technique like peer instruction in a class of 600 ? Will the tools I pick support 50,000 assignment submissions per term? How do I manage change? In this paper we will report our experience redesigning our CS 2 course from a mix of disparate tools to a fully-automated, integrated course delivery system. We combined multiple teaching platforms into a unified design, that is delivered to the student as a coherent whole, rather than piecemeal. We will discuss the design decisions we made, and some we didn't, and why. We will discuss our approach, the challenges we faced, and also provide instructors with a set of “lessons learned”, questions they should ask themselves and their colleagues as they embark on any course redesign.

This innovative practice paper describes our experiences with alternative grading practices in introductory computing courses and two large public universities in the United States. Computing classrooms often use traditional grading practices involving allocating points to assignments, deducting points for mistakes and tardiness, and combining assignment scores using a weighted average to determine grades. Recent research suggests that these practices may diminish achievement, discourage students, and suppress effort to such an extent that they are considered by some as detrimental. We approach our work as an exploratory case study, without predefined research questions or hypotheses. Our experiences began with the adoption of specifications grading. We outline the grading scheme applied to traditional programming assignments and exams/quizzes, and discuss the initial integration of these schemes with conventional auto-grading tools. We delve into student perceptions of alternative grading, their utilization of flexible deadlines, and resubmission opportunities. We conclude with a discussion of two challenges encountered during our exploration: student acceptance of a novel grading form, and the adaptation of tools designed for traditional grading to support alternative grading mechanisms. Our early exploration aims to inspire further research on the use of alternative grading in computing. It is clear from our observations that simply implementing the practices does not ensure the equitable and inclusive outcomes that can be achieved with these practices. If students are not prepared to use these practices, they find them difficult to understand and can feel that they are not being treated fairly. Additionally, we wish to foster a community of practice to assist faculty members exploring these changes, with the goal of creating more equitable and inclusive classrooms.

Measuring instructors' adoption of learning management system (LMS) tools is a critical first step in evaluating the efficacy of online teaching and learning at scale. Existing models for LMS adoption are often qualitative, learner-centered, and difficult to leverage towards institutional support. We propose depth-of-use (DOU): an intuitive measurement model for faculty's utilization of a university-wide LMS and their needs for institutional support. We hypothesis-test the relationship between DOU and course attributes like modality, participation, logistics, and outcomes. In a large-scale analysis of metadata from 30000+ courses offered at Virginia Tech over two years, we find that a pervasive need for scale, interoperability and ubiquitous access drives LMS adoption by university instructors. We then demonstrate how DOU can help faculty members identify the opportunity-cost of transition from legacy apps to LMS tools. We also describe how DOU can help instructional designers and IT organizational leadership evaluate the impact of their support allocation, faculty development and LMS evangelism initiatives.

Background and ContextSelf-regulated learning (SRL) skills are critical aspect of learning to program and are predictive of academic success. Early college students often struggle to use these skills, but can improve when given targeted instruction. However, it is not yet clear what skills are best to prioritize.ObjectiveWe seek to create a shortlist of SRL skills that help computer science students with an ultimate goal of giving future researchers and educators a set of promising targets for future study.MethodThis work details a three-phase Delphi-Process study wherein a panel educators to reach consensus on SRL skills needed for success in CS.FindingsWe identified fourteen skills broadly categorized into five areas: Planning, Executing, Monitoring, Responding, and Reflecting. There are several factors to indicate these skills are all highly valuable areas for future work.ImplicationsWe highlight Knowing When to Seek Help especially for future research within the Computing Education Research community.

It is often the case that computer science classrooms use traditional grading practices where points are allocated to assignments, mistakes result in point deductions, and assignment scores are combined using some form of weighted averaging to determine grades. Unfortunately, traditional grading practices have been shown to reduce achievement, discourage students, and suppress effort to such an extent that some common elements of traditional grading practices have been termed toxic. Using grades to reward or punish student behavior does not encourage learning and instead increases anxiety and stress. These toxic elements are present throughout computing education and computer science classrooms in the form of late penalties, lack of credit for code that doesn't compile or pass certain unit tests, among others. These types of metrics, that evaluate behavior are often influenced by implicit bias, factors outside of the classrooms (e.g., part-time employment), and family life situations (e.g., students who are caregivers). Often, students in these situations are disproportionately from low-socioeconomic backgrounds and predominantly students of color. Through this paper, we will present a case for adoption of equitable grading practices and a call for additional support in classroom and teaching technologies as well as support from administrations both at the department and university level. By adopting a community of practice approach, we argue that we can support new faculty making these changes, which would be more equitable and inclusive. Further, these practices have been shown to better support student learning and can help increase student learning gains and retention.

The process of sharing content among instructors at different institutions is not straightforward. In most contexts, "shared" material is unidirectional: a more experienced instructor shares their materials with a more novice instructor; a book publisher provides resources to instructors who have adopted their textbook. In a fully-realized sharing ecosystem, this flow is bi-directional. Materials can be shared, modified, corrected or edited, and then the changes are committed back to the repository for use by everyone who has adopted the material. There are many issues that must be addressed regarding the sharing process, related to both the actual content, as well as the context in which the sharing occurs. In fact, there is a wide opinion on what exactly "sharing" means. As such, we endeavoured to identify sharing opportunities and address issues that sharing course content raises. The primary goal of this initiative was to inform the development of ways in which shared courses and content can be made available online to share across higher education institutions in Virginia, as a potential model to be expanded to other localities and disciplines. A multi-institutional team developed a framework for sharing course materials within the context of Computer Science at the university-level. A number of areas of consideration have been identified as directly impacting any sharing scheme. Additionally, non-content related issues were uncovered during our investigation. These issues impact the context and process in which the sharing may occur. Finally based on the discussions conducted by the working group, a preliminary framework has been developed.

Historically, computing instructors and researchers have developed a wide variety of tools to support teaching and educational research, including exam and code testing suites and data collection solutions. Many are then community or individually maintained. However, these tools often find limited adoption beyond their creators. As a result, it is common for many of the same functionalities to be re-implemented by different instructional groups within the CS Education community. We hypothesize that this is due in part to accessibility, discoverability, and adaptability challenges, among others. Further, instructors often face institutional barriers to deployment, which can include hesitance of institutions to utilize community developed solutions that often lack a centralized authority. This working group will explore what solutions are currently available, what instructors need, and reasons behind the above-mentioned phenomenon. This will be accomplished via a literature review and survey to identify the tools that have been developed by the community; the solutions that are currently available and in use by instructors; what features are needed moving forward for classroom and research use; what support for extensions is needed to support further CS Education research; and what institutional challenges instructors and researchers are currently facing or have faced in the past in developing, deploying or otherwise using community software solutions. Finally, the working group will identify factors that limit adoption of solutions and ways to integrate and improve the accessibility, discoverability, and dissemination of existing community projects, as well as manage and overcome institutional challenges.

Historically, computing instructors and researchers have developed a wide variety of tools to support teaching and educational research, including exam and code testing suites and data collection solutions. However, these tools often find limited adoption beyond their creators. As a result, it is common for many of the same functionalities to be re-implemented by different instructional groups within the Computing Education community. We hypothesise that this is due in part to discoverability, availability, and adaptability challenges. Further, instructors often face institutional barriers to deployment, which can include hesitance of institutions to rely on community developed solutions that often lack a centralised authority and may be community or individually maintained. To this end, our working group explored what solutions are currently available, what instructors needed, and the reasons behind the above-mentioned phenomenon. To do so, we reviewed existing literature and surveyed the community to identify the tools that have been developed by the community; the solutions that are currently available and in use by instructors; what features are needed moving forward for classroom and research use; what support for extensions is needed to support further Computing Education research; and what institutional challenges instructors and researchers are currently facing or have faced in using community software solutions. Finally, the working group identified factors that limited adoption of solutions. This work proposes ways to integrate and improve the availability, discoverability, and dissemination of existing community projects, as well as ways to manage and overcome institutional challenges.

Trust in a recommendation system (RS) is often algorithmically incorporated using implicit or explicit feedback of user-perceived trustworthy social neighbors, and evaluated using user-reported trustworthiness of recommended items. However, real-life recommendation settings can feature group disparities in trust, power, and prerogatives. Our study examines a complementary view of trust which relies on the editorial power relationships and attitudes of all stakeholders in the RS application domain. We devise a simple, first-principles metric of editorial authority, i.e., user preferences for recommendation sourcing, veto power, and incorporating user feedback, such that one RS user group confers trust upon another by ceding or assigning editorial authority. In a mixed-methods study at Virginia Tech, we surveyed faculty, teaching assistants, and students about their preferences of editorial authority, and hypothesis-tested its relationship with trust in algorithms for a hypothetical ‘Suggested Readings’ RS. We discover that higher RS editorial authority assigned to students is linked to the relative trust the course staff allocates to RS algorithm and students. We also observe that course staff favors higher control for the RS algorithm in sourcing and updating the recommendations long-term. Using content analysis, we discuss frequent staff-recommended student editorial roles and highlight their frequent rationales, such as perceived expertise, scaling the learning environment, professional curriculum needs, and learner disengagement. We argue that our analyses highlight critical user preferences to help detect editorial power asymmetry and identify RS use-cases for supporting teaching and research.

Automated grading systems provide feedback to students in a variety of ways, but they typically focus on identifying incorrect program behaviors. Such systems provide notices of test case failures or runtime errors, but without debugging skills, students often become frustrated when they don't know where to start addressing these defects. Borrowing from work in software engineering research related to automated defect location, we leverage previous research describes using statistical fault localization (SFL) techniques to identify the probable locations of defects in student coding assignments. The goal is to use these SFL techniques to provide a scaffold for students, to direct their debugging efforts without giving too much guidance, and thus minimizing the learning associated with investigating the defects. After determining the "suspiciousness" for each line of code involved in the defect, we create a "heat map" visualization overlay onto their source code of the "suspiciousness" scores to visually guide a student's attention to parts of their code that are most likely to contain problems. This paper describes the results of an analysis comparing two semesters of CS 1114: Introduction to Software Design (CS1) students, one which had access to the heat map feedback (Fall 2017: n~170), and one that did not (Fall 2015: n~270). The results show that when the heat maps were available, students found it easier to make improvements on their code from submission to submission, as well as spending less time overall achieving the maximum score on the instructor testing assessments. In fact, while we were optimistic in the impact of the heat map feedback, the results we observed were unexpectedly strong. To address this concern, we conducted an additional study, using student submissions from our Spring 2019 CS 1 students (n~230). The results of the second analysis confirmed the results of the first.

Learning management system (LMS) tools are increasingly relevant to scaling computing pedagogies. Measuring their utilization and impact at scale, however, remains computationally expensive. We examine the problem of estimating the utilization of a department-wide LMS, and its impact on the design, management and outcomes of Computer Science courses. We introduce 'depth-of-use' (DOU): a first-principles, resource-specific metric of LMS utilization. We then hypothesis-test the relationship between DOU and course attributes like modality (course level, mode-of-delivery, third-party app use), participation (enrollment, viewership), logistics (teaching support, digital skills training) and outcomes (average GPA, DFW rate). Experiments on metadata from over 1300 Computer Science courses taught at Virginia Tech between 2015 and 2019 suggest that our framing of DOU helps identify resource-level preferences of micro-cohorts of courses, linked to their content, logistics and pedagogies. We discover that, across the Computer Science department at Virginia Tech, overall LMS use is consistently linked to favorable learning outcomes. We also discover that a complex interaction between the needs for scale, ubiquitous access and interoperability drives strong LMS utilization, with graduate and online-only courses faring highest in their aggregate use of LMS services. Finally, we describe two key applications of our analyses. One, we demonstrate how DOU can help CS faculty identify the relative impact of transition from legacy apps to LMS services. Two, we describe how DOU can help instructional designers evaluate and improve their design interventions.

Packaging materials is a generalized term to capture a broad array of tasks (creating, revising, sharing, finding, crediting, etc.) for materials such as assignments, teacher notes, and evaluation data. Substantial effort has gone into creating materials over the years, but the community still struggles to find ways to effectively manage these. This BoF provides an opportunity to identify needs, concerns, prior efforts, and future plans. A primary goal is the formation of a Working Group tasked to develop a standard for curricular material creation and sharing, joining with broader efforts of standardization (e.g., CSSPLICE) and existing initiatives for creating repositories, tools, and materials.

University students routinely use the tools provided by online course ranking forums to share and discuss their satisfaction with the quality of instruction and content in a wide variety of courses. Student perception of the efficacy of pedagogies employed in a course is a reflection of a multitude of decisions by professors, instructional designers and university administrators. This complexity has motivated a large body of research on the utility, reliability, and behavioral correlates of course rankings. There is, however, little investigation of the (potential) implicit student bias on these forums towards desirable course outcomes at the institution level. To that end, we examine the connection between course outcomes (student-reported GPA) and the overall ranking of the primary course instructor, as well as rating disparity by nature of course outcomes, based on data from two popular academic rating forums. Our experiments with ranking data about over ten thousand courses taught at Virginia Tech and its 25 SCHEV-approved peer institutions indicate that there is a discernible albeit complex bias towards course outcomes in the professor ratings registered by students.

Automated grading systems provide feedback to students in a variety of ways, but usually focus on identifying incorrect program behaviors. Such systems provide notices of test case failures or runtime errors, but without debugging skills, students often become frustrated when they don't know where to start. They know their code has defects, but finding the problem may be beyond their experience, especially for beginners. An additional concern is balancing the need to provide enough direction to be useful, without giving the student so much direction that you effectively give them the answer. This paper presents our experience using heat maps to visually guide student attention to parts of their code that are most likely to contain problems. These visualizations are generated using existing tools that capture execution traces from instructor-written tests to identify which portions of the code are executed during tests that pass, and which portions are executed during tests that fail. Superimposing execution footprints allows statistical identification of locations in the student's code that are most likely to contain faults. This paper describes the results of using this feedback approach to help guide student attention with heat map visualizations over two semesters of CS1 involving over 700 students. Based on this experience, we analyze the utility of the heat maps, describe student perceptions of their helpfulness, and describe the unexpected challenges arising from students attempts to understand and apply this style of feedback. We conclude with concrete solutions proposed to improve how guiding feedback is presented to students.

Automated grading systems are useful for conveying debugging feedback to students, but the manner in which this feedback is displayed can be problematic for students. Automated systems usually report failures in student code, whether it be the results of test case failures or runtime errors. However, since most students are not explicitly taught how to debug, many get stuck. They know the defects exist, but they don't have to experience to know how to find them. Much work has been done in software engineering research using automatic fault localization to programmatically locate bugs within a piece of code under test, as well as research into validating those detection models when applied to student code. However, determining how to present that information back to the student has not been addressed. There is a balance between giving hints and telling the student exactly where the error occurred. The goal of this paper is to present a technique to present the student's code with suggestions to the student of where to investigate, based on the results of the automatic fault location. Taking results from the GZoltar statistical fault localization library, we have developed a method of expressing the suggestions to the students by visualizing the potential defect locations within the context of the student's source code using heatmaps.

While automated grading systems are useful for generating real-time feedback for students, they also highlight some obstacles. In particular, when a student's program contains bugs, automated systems report this fact, but not how to locate or correct them. This paper reports on a study inspired by current research on automatic fault localization - techniques to programmatically locate bugs within a piece of code under test. The goal is not to "identify" or diagnose the bug, but simply to locate where it is in the code, to give the student a direction to investigate. Before employing this technique live with students, this study validates that automated fault localization can be performed with the level of accuracy needed to give meaningful guidance. We used the GZoltar statistical fault localization library for Java to analyze 135 CS2-level student programs, and then manually debugged the programs to find the locations of their faults. This paper provides the analysis of this information, which produced a feasible strategy for providing accurate, automated suggestions to students for "where to look" in order to fix their own programs.