The retrospective is a crucial component of the agile software development process. In previous studies of retrospectives in undergraduate team software development projects, students exhibited limited and shallow reflection. We speculate that this is due to students' limited experience with reflection and the absence of clear guidance for engaging in deep reflection during agile retrospectives. To explore the potential for a pedagogical intervention to foster deeper reflection in retrospectives, we present an empirical comparison of a standard retrospective model against an enhanced retrospective model that scaffolds deeper levels of reflection by prompting students to justify and critique their practices and weigh alternative approaches. Through a systematic classification of the reflection level of statements made during individual brainstorming and team discussion phases of retrospectives, our study found that the enhanced model led to individuals and teams engaging in significantly higher levels of reflection. Our findings contribute to improving software engineering education by demonstrating the efficacy of an enhanced pedagogical model for team retrospectives.

The current hiring landscape for teaching faculty is a perfect storm, where two issues - (1) increasing demand but a lack of trained candidates and (2) highly diverse job expectations - combine to create a job search that can be overwhelming for teaching faculty candidates and disappointing for departments of computing. As departments independently navigate this new reality, they have each come up with their own practices for interviewing and screening teaching faculty candidates. This proliferation of hiring practices has made it increasingly difficult for teaching faculty candidates to identify positions that are a good fit and prepare adequately for the diversity of hiring practices. In response to these challenges, the Computing Research Association - Education committee surveyed faculty across Canada and the USA about their hiring practices. This work culminated in a recent white paper "Best Practices for Hiring Teaching Faculty in Research Computing Departments." Panelists will engage with the recommendations from this recent paper, discuss principles underlying some best practices for the hiring process, and field questions from the audience.

The retrospective, or retro, is a fundamental component of the Agile process, widely used in both software engineering courses and industry. In a retro, teams come together at the end of a sprint to reflect on their team's performance. We conducted an empirical study to explore three research questions concerning retros in undergraduate team projects: (1) What do students reflect on? (2) What is the quality of their reflections? (3) How do teams' retros vary in terms of content and quality? Our study analyzed a corpus of 963 statements documented in the retros of 32 undergraduate software teams (n = 182 students) enrolled in four software engineering courses at two North American universities. A content analysis revealed that teams reflected most often on their work, communication, and collaboration practices. Nearly a third of teams' reflections focused on their general work practices, while nearly half focused on specific areas of the software development lifecycle-most prominently, pull requests, issues, and coding/testing/debugging. An analysis of the quality of teams' retro reflections showed that only 13% provided justification for a strategy to be stopped, continued, or started. An analysis of team-by-team results indicated significant differences in teams' retro content and quality. We compare these results to previous studies of retros in academia and industry, and consider their implications for software engineering education.

Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams ( n = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’ contributions to the team, and the equality of team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.

Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students' use of reflection--one aspect of metacognition--in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams' online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams' ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams' use of reflection in team software projects.

Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students' use of reflection-one aspect of metacognition-in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams' online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams' ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams' use of reflection in team software projects.

Background and Context: Assessing team members' indivdival contributions to software development projects poses a key problem for computing instructors. While instructors typically rely on subjective assessments, objective assessments could provide a more robust picture. To explore this possibility, In a 2020 paper, Buffardi presented a correlational analysis of objective metrics and subjective metrics in an advanced software engineering project course (n= 41 students and 10 teams), finding only two significant correlations. Objective: To explore the robustness of Buffardi's findings and gain further insight, we conducted a larger scale replication of the Buffardi study (n = 118 students and 25 teams) in three courses at three institutions. Method: We collected the same data as in the Buffardi study and computed the same measures from those data. We replicated Buffardi's exploratory, correlational and regression analyses of objective and subjective measures. Findings: While replicating four of Buffardi's five significant correlational findings and partially replicating the findings of Buffardi's regression analyses, our results go beyond those of Buffardi by identifying eight additional significant correlations.

Team software development projects are a crucial component of undergraduate computing education. Studies of software developers in industry underscore the need for undergraduates to acquire authentic software development experiences in which they contribute to legacy code bases within collaborative software development contexts. Despite their clear educational value, team software development projects are notoriously challenging to implement and assess. Drawing on experience and relevant research results, this Research in Practice Project Activity (RIPPA) brings together computing educators and researchers to discuss, research, and innovate approaches to designing and assessing authentic software development projects. Participants are invited to run empirical studies in their own courses in which they (a) collect subjective and objective measures of team and individual progress in software projects, and (b) explore alternative pedagogical and assessment approaches, including those supported by the use of custom learning management and data analytics tools. By the end of the RIPPA, we aim to publish one or more research papers that advance the state-of-the-art in team software project pedagogy, supporting technology, and assessment based on the results of multi-institutional research studies.

Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams' processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.

Students' experience with software testing in undergraduate computing courses is often relatively shallow, as compared to the importance of the topic. This experience report describes introducing industrial-strength testing into CMPSC 156, an upper division course in software engineering at UC Santa Barbara. We describe our efforts to modify our software engineering course to introduce rigorous test-coverage requirements into full-stack web development projects, requirements similar to those the authors had experienced in a professional software development setting. We present student feedback on the course and coverage metrics for the projects. We reflect on what about these changes worked (or didn't), and provide suggestions for other instructors that would like to give their students a deeper experience with software testing in their software engineering courses.

Undergraduates have been an important part of the teaching staff at many universities for decades. Recent work such as the Peer Teaching Summit at SIGCSE 2019 [1] and a systematic literature review [2] have focused more attention on issues related to the use of undergraduates in teaching assistant roles. This BOF provides a forum to discuss open research questions about undergrad TA/mentor programs at various stages in their evolution. Attendees will have an opportunity to discuss research questions, research methods, and explore possible collaborations. Discussion Leader(s): Diba Mirza will open the session by summarizing the key findings of literature review on UTAs from ICER'19 [1]-in particular, the fact that while there is widespread consensus that using UTAs is a good idea, the evidence backing up this consensus is mostly anecdotal. This creates many opportunities to establish the effectiveness of current practices, and the claimed benefits of the use of UTAs through more rigorous research and to discuss innovative ways to incorporate UTAs in teaching outside of what has been reported in the literature. This will be followed by a themed discussion to brainstorm about research within each of these areas. Based on the large turnout at the BOF on Undergraduate Teaching Assistants in SIGCSE'19, we plan to organize the discussion in smaller groups. Diba Mirza, Phillip Conrad, and Cynthia Lee will lead the discussion within each group. The leads will also document the discussions and share it with the participants.

In this interactive special session, participants will learn to facilitate a workshop for training teaching assistants (TAs) based on a card game. The game invites players to respond to challenging scenarios that TAs may face in the course of their duties. Each scenario appears on a card, and players discuss how they would respond. Session participants will receive a deck of 52 of these cards created with funding from a SIGCSE Special Projects Grant. This session uses the format of two previous well attended SIGCSE special sessions that focused on bias and had a target audience of faculty. While the intended audience is those interested in TA training, everyone interested in inclusive teaching practices is welcome to join in the rich small and large group discussions that we expect to take place!

The goal of the session is to help attendees who are committed to diversity and inclusion learn to talk about different dimensions of identity (e.g., race, class, gender, sex, sexuality, etc.). The landscape of terms is always changing and we want SIGCSE attendees to feel more comfortable using current language to talk about issues related to diversity and inclusion. This special session will include six lightning talks, individual reflection, small-group discussion, and a Q&A with presenters. This is a reprise of a special session held at the NCWIT Summit in 2018; it was well received and we hope to offer it to the larger SIGCSE audience.

We believe that more Computing faculty should spend some time in industry in order to experience for themselves the types of jobs their students are likely to take at graduation and throughout their careers. Each panelist has spent at least one sabbatical in industry and found the experience to be far more beneficial than anticipated. We all found that our sabbatical experiences helped us to (1) develop our skills, (2) discover and practice the skills that are currently being used in industry, (3) gain credibility with our students, (4) improve our understanding of the computing profession beyond software development, and (5) build relationships with local businesses. Many of us returned to the classroom thinking "I can't believe they let me teach college without any industry experience".

Undergraduates have been an important part of the teaching staff at many universities for decades, but this is not a universal practice. This BOF provides a forum to discuss undergraduate TA/mentor programs at various stages in their evolution. Attendees will have an opportunity to discuss the benefits, best practices, and research questions related to the use of undergraduates as teaching assistants and/or mentors. The audience is expected to consist of faculty that have already implemented such programs, and those who might be considering doing so, and want to share information about successes and challenges. Discussion Leader(s): Phill Conrad will facilitate the discussion and invite Colleen Lewis, Cynthia Lee and Diba Mirza to briefly speak about the Undergraduate TA/Mentor programs at their respective institutions. During the balance of the time participants will be invited to contribute ideas or ask questions related to the use of undergraduate TAs and mentors. Expertise of Discussion Leader(s): Diba Mirza and Phill Conrad are faculty in the Computer Science department at UC Santa Barbara, where they have created an undergraduate mentor program in the past two years. Colleen Lewis is an Associate Professor of CS at Harvey Mudd College, and is the lead PI for the csteachingtips.org project, a project to collect and document CS pedagogical content knowledge. Cynthia Lee is a Lecturer in CS at Stanford University, and works closely with the longstanding undergraduate TA program in the department.

Packaging materials is a generalized term to capture a broad array of tasks (creating, revising, sharing, finding, crediting, etc.) for materials such as assignments, teacher notes, and evaluation data. Substantial effort has gone into creating materials over the years, but the community still struggles to find ways to effectively manage these. This BoF provides an opportunity to identify needs, concerns, prior efforts, and future plans. A primary goal is the formation of a Working Group tasked to develop a standard for curricular material creation and sharing, joining with broader efforts of standardization (e.g., CSSPLICE) and existing initiatives for creating repositories, tools, and materials.

We present a systematic literature review of the prior work on Undergraduate Teaching Assistants (UTAs) in Computer Science with two goals: (1) to create a taxonomy of practices that relate to the design and implementation of UTA programs, (2) to identify the benefits of using UTAs as claimed by the literature and characterize the level of evidence for those claims. We analyze 336 excerpts from 40 papers related to these goals. We use content analysis on excerpts describing practices to extract high-level themes that include recruiting, UTA and program coordinator duties, training, evaluation and organization of UTA programs. We perform a more fine-grained analysis within each theme to identify specific questions about UTA programs and the answers provided by the literature. Using a similar technique, we report on the claimed benefits of UTA programs to students, UTAs, instructors and institutions. Our analysis follows well-defined protocols involving multiple reviewers and we report on the inter-rater reliability. The results provided in this paper lay the groundwork for developing evidence-based best practices in UTA programs and inform practice and policy related to the use of UTAs at tertiary institutions. As such, it is relevant to educators establishing a new UTA program, expanding an existing program, or continuously improving an established program, as well as those designing research studies of such programs.

Replication research is rare in CS education. For this reason, it is often unclear to what extent our findings generalize beyond the context of their generation. The present paper is a replication and extension of Achievement Goal Theory research on CS1 students. Achievement goals are cognitive representations of desired competence (e.g., topic mastery, outperforming peers) in achievement settings, and can predict outcomes such as grades and interest. We study achievement goals and their effects on CS1 students at six institutions in four countries. Broad patterns are maintained --- mastery goals are beneficial while appearance goals are not --- but our data additionally admits fine-grained analyses that nuance these findings. In particular, students' motivations for goal pursuit can clarify relationships between performance goals and outcomes.

Previous work finds that recent college graduates entering the software development industry encounter difficulties early in their careers, due to significant differences between their coding experiences in academia vs. what is expected of them on the job. To explore the gap between academic and industry software development, we conducted interviews with twenty early career software developers with four-year degrees in CS. We present an analysis of these interviews, including excerpts in the developers' own words, organized around six themes. We conclude with thoughts on how to bridge this gap so that our students may be better prepared when we launch them into their careers.