As undergraduate computer science classes grow in size, institutions increasingly rely on asynchronous computer-based assessments. To investigate whether exam timing reveals evidence of cheating, we analyze 21,403 submissions from 51 asynchronous exams across two undergraduate courses in this retroactive study. We extend prior research on proctored multiday exams by introducing a comparison in student performance trends between two distinct assessment modes: on-site proctored and off-site unproctored. We find that performance declines throughout the exam window in both modes. We observe a weak negative correlation between start time and performance, with standardized scores decreasing by 0.14 points per hour (on-site proctored) and 0.61 points per hour (off-site unproctored). In addition, start-time distributions and student surveys reveal behavioral differences. On-site proctored exams follow a centered start-time distribution, likely influenced by a reserved lecture hour. In contrast, off-site unproctored exams show a left-tailed distribution, with most students starting later than intended. This pattern suggests that greater scheduling flexibility leads to later exam starts, potentially exacerbating performance declines due to academic procrastination.

The ACM Student Research Competition (SRC) began at the SIGCSE Technical Symposium back in 2003. At that time, it was the only SRC, and it made sense to welcome every student's work at SIGCSE. Shortly after that, other conferences joined with their own SRCs. As of today, the SIGCSE Technical Symposium still accepts submissions in any area at its SRC, in large part because of this history.

The rise of online assessments has motivated the development of randomized question banks, with randomization referring to the generation of different variants of a question. Although not all randomization efforts are equally effective in generating question isomorphs, the current classification of questions solely as randomized or not fails to address the varying degrees of randomization. To address this limitation in describing the diversity of randomization designs, we introduce a framework that outlines six distinct randomization levels. Additionally, we designed practical guides to assist educators in effectively using the framework, aligning with their pedagogical objectives. Through our application of this framework to classify around 200 questions from two courses, we further highlight the generalizability of the framework and reveal insights into the considerations and challenges associated with incorporating question randomization into computer science curricula.

The popularity of autograding has grown due to increasing class sizes and the need to reduce grading load while ensuring quality. Autograding has conventionally been used for multiple choice and fill in the blank questions, or to check code correctness. In this work, we discuss the use of autograders at UBC and some non-conventional autograding implementations in our curricula. We reflect upon our autograder use in our courses and discuss the benefits, implications, and considerations of this pedagogical choice.

PrairieLearn is an open source, extensible online assessment platform built on modern web technologies. In this workshop, we will focus on how PrairieLearn can be used to improve student learning in undergraduate computer science classes. However, the platform is also more than suitable for use as an assessment engine in a variety of courses including the humanities, social, physical, and life sciences. In the first part of the workshop, we will showcase multiple question styles that highlight PrairieLearn's abilities as an online platform, including deploying automatically and manually graded questions at scale in large classes. In the second part of the workshop, we will discuss the anatomy of a PrairieLearn question, create several custom questions, and design assessments in PrairieLearn. In the third part, we will share strategies on adopting PrairieLearn at your institution. In particular, how algorithmically generated questions can be used in support of alternative grading schemes such as Mastery- or Specifications-Grading. Finally, we will share how PrairieLearn can be extended to support other coding languages and paradigms with custom and external autograders. There will be plenty of opportunities for questions throughout the workshop, and we intend to leave plenty of time for additional 1:1 support and training. Attendees will be able to attend the session virtually and are recommended to bring a web-connected computing device. By the end of the session, attendees will know enough to run a whole class on PrairieLearn including designing questions appropriate for homework, labs, and tests.

The identification of student struggles has drawn increasing interests from computing education and learning analytics communities in recent years, considering the high failure rate and fast enrollment growth of computer science courses. Prior studies on this topic employed a multitude of data sources and methodologies with varying degrees of success. Nearly all studies attempted to predict low overall course performance to identify struggling students, risking oversimplifying student learning and struggles. Additionally, many studies utilize data sources that are limited to their original contexts or local student demographics, making it difficult to replicate or put the findings into practice. To address these gaps, we studied the feasibility of identifying student struggles at the topic level using features that are agnostic to courses and contexts. Our results show that it is possible to identify student struggles at a more fine-grained level within days. Our findings contribute new insights into automatic identification of student struggles at the topic level on a large scale, which can be used to guide meaningful interventions on student learning.

Cloud-based JupyterHub installations support easy access to computing environments for intro computing using Jupyter Notebooks. We propose an interface and technical design to smoothly integrate JupyterHub with Canvas for intro students.

We propose a novel, inverted two-stage exam format that encourages anticipation of transfer problems. We report on its design, use, and initial assessment for low-stakes quizzes in an algorithms course. A typical two-stage exam, where the group stage comes after the individual stage, emphasizes retrospective learning: reflecting on already-solved problems. Our inverted two-stage format places the group stage first, and incentivizes prospective learning: preparing for transfer to novel problems that will appear on the individual stage and subsequent assessments. TAs reported the new format leads to reliable engagement. In surveys, most students preferred inverted two-stage quizzes to individual quizzes plus a TA walkthrough. Students who preferred this format cited the value of learning from peers, brainstorming in the problem domain, and working out ambiguities in the domain and problems.

We describe key factors in adapting to a computing context the Appreciative Inquiry methodology, which emphasizes participatory exploration of a design (e.g., course or curricular design) and encourages and supports diverse perspectives via a story-driven, strengths-based approach. Appreciative Inquiry is a stakeholder-driven, qualitative methodology for exploring a design or organization that focuses on what is working well and how to preserve and build on that success. As a case study, we share our experience using Appreciative Inquiry to evaluate and improve a non-majors' first-year computer science course that is intended to support diverse students. We describe the methods that we used to give context to our adaptation of Appreciative Inquiry. We then highlight our reflections on our evaluation and the strategies that we learned from this context to effectively apply Appreciative Inquiry. We believe that our approach yielded different and deeper results than we would have found had we not used Appreciative Inquiry. Further, we believe that its focus on strengths may have attracted participants who would not have otherwise participated; its participatory nature is a practical way to include students and teaching assistants as co-evaluators and give power to their voices. We argue that other computer science educators should consider trying this approach to tap into perspectives and input overlooked by more common research methods which focus instead on quantitative data collection or addressing deficiencies and problems.

In this workshop, adapted from our SIGCSE 2018 workshop [1], we as a group will use Appreciative Inquiry [2] (AI) techniques to explore and develop our strengths as computer science educators. Participants will gain appreciation for their strengths as an educator, with concrete plans for building on these strengths. They will also learn about Appreciative Inquiry as a qualitative research methodology that is complementary to more common computer science research methodologies, and that they can apply to evaluate and improve their own educational practice. Appreciative Inquiry drives change by building on what's already working well in an organization. Similarly to other qualitative methods, AI generates rich, deep feedback that is grounded in stakeholders' experiences, but in contrast to other methods its focus on strengths and positives surface unique, strength-based findings and make it an energizing and fulfilling approach to professional development and the scholarship of teaching and learning. We will share our materials and key tips to enable participants to apply Appreciative Inquiry in their own work. Participants may wish to run Appreciative Inquiry workshops with students as an evaluation method, or run them with colleagues for professional development or for promoting positive change in their unit or program, or take smaller steps integrating the appreciative mindset into their teaching or other professional work.

In a session with live music and collaborative parody, we celebrate the long tradition of computing parody songs ("filks") and their potential to contribute to education and a fun environment in CS courses. We perform several new and classic CS filks, such as "Like it Called on Me (QuickSort)" [1], interspersing discussion of how and why these parodies were written. We also propose a song for the audience to parody and walk them through a structured small-group activity to help them brainstorm topics and lyrical phrases and fit them to the existing lyrics and music. Attendees should expect to laugh (and possibly cry) with the singing and to leave inspired to incorporate filks and other creative activities into their computing education practice: playing, performing, or writing songs themselves, and encouraging their students to do so as well.

A Concept Inventory (CI) is a set of multiple choice questions used to reveal student's misconceptions related to some topic. Each available choice (besides the correct choice) is a distractor that is carefully developed to address a specific misunderstanding, a student wrong thought. In computer science introductory programming courses, the development of CIs is still beginning, with many topics requiring further study and analysis. We identify, through analysis of open-ended exams and instructor interviews, introductory programming course misconceptions related to function parameter use and scope, variables, recursion, iteration, structures, pointers and boolean expressions. We categorize these misconceptions and define high-quality distractors founded in words used by students in their responses to exam questions. We discuss the difficulty of assessing introductory programming misconceptions independent of the syntax of a language and we present a detailed discussion of two pilot CIs related to parameters: an open-ended question (to help identify new misunderstandings) and a multiple choice question with suggested distractors that we identified.

This session is a chance for researchers studying concept inventories (CIs)--low-cost assessments highlighting student misconceptions in a field--and CS education practitioners to communicate about advances in concept inventories in an engaging and utterly ridiculous way. We use a "quiz show" format to present CI items from various authors' work across the computing curriculum. On each question, audience members and volunteer contestants consider their own response and guess students' common responses. Then, they see how authentic student data illustrate the misconceptions these items probe. The session's goal is three-fold: educate practitioners about recent results in concept inventory research that may suggest surprising trends in student learning, popularize concept inventories as a tool in research and practice, and collect the audience's expert responses to concept inventory items.

Too many students in introductory programming classes fail to understand the significance and utility of the concepts being taught. Their low motivation impacts their learning. One contributing factor is pedagogy that emphasizes computing for its own sake and assignments that are abstract, such as computing the factorial function.Many educators have improved on such traditional approaches by teaching concepts in contexts that students find more relevant, such as games, robots, and media. Now, it is time to take the next step.In this special session, participants will develop and discuss ways to teach introductory programming by means of real-world data analysis problems from science, engineering, business, and the humanities. Students can be motivated to learn programming in order to analyze DNA, predict the outcome of elections, detect fraudulent data, suggest friends in a social network, determine the authorship of texts, and more (see Section 3.4 for more examples).The approach is more than just a collection of "nifty assignments": rather, it affects the choice of topics and pedagogy, all of which together lead to greater student satisfaction. The approach has been successfully used at 4 colleges and universities. The classes were effective for both CS and non-CS majors. Neither the computing material nor the problems need to be "dumbed down". At the end of the term students were amazed and delighted at the real data analysis that they could perform. They were excited about applying computation in their work and about learning more.The special session contains a mix of activities, including comparative analysis of introductory classes; group discussion of curriculum design; a mini-panel discussing how the approach has worked in practice; and brainstorming about example assignments and curriculum revision.

As guest editor, I’m excited to bring you this issue in which the authors of one invited editorial and three peer-reviewed papers summarize and push forward research on concept inventories in Computer Science. Our guest editorial introduces concept inventories, our survey paper summarizes the state of the art in concept inventories in CS, and finally two papers explore the refinement and use of concept inventories in very different ways.

In this paper, we triangulate evidence for five misconceptions concerning binary search trees and hash tables. In addition, we design and validate multiple-choice concept inventory questions to measure the prevalence of four of these misconceptions. We support our conclusions with quantitative analysis of grade data and closed-ended problems, and qualitative analysis of interview data and open-ended problems. Instructors and researchers can inexpensively measure the impact of pedagogical changes on these misconceptions by using these questions in a larger concept inventory.

We present a highly reusable "inverted" project in which students learn asymptotic and practical behaviour of dictionary data structures--linked-lists, arrays, balanced trees, and hash tables--in an atmosphere of mild competition. Much like David Levine's Nifty Assignment "Sort Detective", rather than implementing the dictionaries, students' programs generate input to our (unlabeled) implementations, and students use timing data to label the implementations. Much like Bryant and O'Halloran's computer architecture labs, students also compete to "convince" a web-based, automated system that their input generators distinguish the dictionaries based on trend-line behaviour. Initial assessment results suggest the project makes substantially improves students' understanding of practical performance of various dictionary data structures, particularly hash tables. UBC has used the project in three terms, and we plan to use it at UBC and U Toronto in coming terms.

We report a study of how dual display screens were used in classroom lectures for university-level courses across a variety of disciplines during five academic terms over a 2-year period. Our goal was to understand the pedagogical consequences of using more than a single electronic display screen to support classroom lectures. We deployed an in-house software system (MultiPresenter) in real classrooms. We examined the use of MultiPresenter by 8 university instructors who taught 15 courses with a total of 1,147 students during 13-week regular terms or 6-week summer terms. We observed classroom lectures, interviewed instructors, collected screen images and log files of MultiPresenter usage, and administered questionnaires to students about their subjective impressions. Based on these data, we analyzed how instructors used MultiPresenter in order to identify examples of how multiple display screens might best be used for educational purposes. The analysis revealed that the following practices are beneficial: the ability to keep information persistent for extended periods, the increased flexibility in where and when information is shown, capability for side-by-side comparison of full screens of information, simultaneous visibility of both overview (“roadmap”) and detailed (“content”) information, and extra space to annotate information. Possible hazards include difficulty focusing on specific information amidst a large amount of information and too much information changing too quickly without proper indication of the changes.

We report on best practices we have established to teach first-year computer science students in closed laboratories, founded on over three years of action research in a large introductory discrete mathematics and digital logic course. Our practices have resulted in statistically significant improvements in student and teaching assistant perception of the labs. Specifically, we discuss our practices of streamlining labs to reduce load on students that is extraneous to the lab's learning goals; establishing a positive first impression for students and TAs in the early weeks of the term; and effectively managing the teaching staff, including weekly preparation meetings for TAs using and a gradual, iterative curriculum development cycle that engages all stakeholders in the course.