Critical thinking is a core educational goal and an essential competency in higher education. The rapid rise of generative artificial intelligence (GenAI) has redefined the context in which critical thinking develops. While GenAI can act as a cognitive partner that fosters engagement, it can also function as a cognitive shortcut, encouraging reliance and surface-level thinking. Emerging studies report mixed findings on whether GenAI use strengthens or diminishes critical thinking. This study aims to investigate how GenAI influences students' critical thinking in computing education, focusing on the mechanisms and conditions that determine when it supports or hinders critical thinking. The proposed research seeks to inform the design of AI-supported learning environments that sustain critical engagement rather than replace it.
Background and Context: Computer science students must learn the skill of selecting an appropriate algorithm design technique for a given algorithm design problem. Despite the topic's significance, there is little research to improve student performance. Objective: We are studying the student thinking process while selecting an appropriate algorithm for a given algorithm design problem. In this paper, we studied student behaviors while finding a solution to an algorithm design problem. Method: We conducted an hour-long think-aloud interview with students who had taken an Advanced Algorithms course and observed their behavior while generating a solution for a given problem. Findings: Analyzing the interviews, we found twelve common behaviors students engaged in while selecting an algorithm for a given problem. Our results show that most students experimented with given examples and thought of existing techniques for given questions. The behaviors most strongly associated with success were identifying sub-problems and connecting to similar problems, but students did not engage in these behaviors as often as they should. Implications: We proposed the three steps that could be useful for students to follow, which could be used by the instructor to teach students in a systematic way to identify the correct algorithm. Our findings show which behaviors are mostly strongly associated with success, and which seem useful are insufficient on their own for success.
Introductory programming (i.e., CS1) is a key subject in any computing degree, as students who do not succeed are significantly more likely to discontinue their studies. As such, maintaining motivation and engagement is critical to increasing student's chance of success. In this paper we explore the potential for increasing introductory programming student's motivation and engagement by offering multiple assessment options. These options are designed to cover the same concepts, and be of equivalent difficulty, but are themed differently. Our goal is to increase the likelihood that each student will find at least one option personally relatable and motivating. In order to achieve this, we also investigate methods for estimating the difficulty of a programming task based on an exemplar solution, so that we can ensure the assessment options are equivalent. We found that students reacted positively to being given assessment options, and that there were no negative impacts to student's learning outcomes based on the assessment theme they chose. We also demonstrate the potential of several static code metrics in estimating the difficulty of introductory programming tasks.
Randomisation in programming assessment is a popular way to create many versions of the same question or exam. However, the technique of question pooling, writing multiple distinct questions and presenting students a random one, raises issues of fairness, as it is challenging for experienced programmers, such as tertiary assessors, to determine if two questions are actually of a similar difficulty. Some randomisation techniques, such as simple substitution, only require the question author to write a template, and provide a list of keywords that they want substituted into each part of the question. As these questions are generated by programmatically substituting flavour text, and do not actually change how the student should write their solution at an algorithmic level, it seems reasonable to assume that this type of randomisation is perfectly fair. This paper outlines our first steps in researching whether or not this type of randomisation is actually fair, by analysing 22 questions, with over 17,000 real student responses from an introductory programming course. A cursory analysis, looking solely at student marks, reveals that in some assessments, students perform differently based on what question variant they get. However, a further statistical analysis using one-way ANOVA showed that these differences were not practically meaningful, with no statistical significance between question variant difficulty. Overall, the results suggest that variant randomization did not materially affect student outcomes and that the assessment design was robust and fair across versions. This indicates to us that this form of randomisation is not unfairly discriminating against students, and gives confidence in our continued use of the technique.
In computer science, assessment often takes the form of asynchronous programming assignments. Generative AI is pushing assessment into the classroom. As a result, handwritten study aids such as cheat sheets are being revisited as a tool to increase learning and reduce student anxiety. In this study, we research how preparing and using cheat sheets affects students' performance on weekly, short, paper-and-pencil quizzes in a computing course. Across the semester, we examine three versions of cheat sheet preparation and accessibility. For the first three quizzes, students were invited to create and use handwritten cheat sheets. For the next three, they received extra credit for creating a cheat sheet, but it was withheld from use during the quiz. For the remaining five, no incentives were provided and cheat sheets could not be used. We immediately surveyed students after each quiz to understand how they prepared and to gauge their perceived quiz anxiety. We assigned each cheat sheet scores for organization and density, correlated these scores vs. quiz grades, and analyzed student-reported anxiety levels vs. cheat sheet preparation. For the 35 participants, we found that after the first quiz, individual quiz grades are positively correlated with cheat sheet organization and that having a cheat sheet reduced anxiety. The effect held even when cheat sheets were withheld.
In group software projects, individual contributions often vary significantly, but detecting imbalances is time-consuming and complex. This research project examines how Large Language Models (LLMs) can aid in validating student contributions by analysing self-reports and Git histories. We developed a prototype tool that automatically extracts coding-related claims, locates corresponding commits, and verifies supporting evidence. Evaluation across two course cohorts shows the tool achieves high claim validation precision (0.94); however, it is stricter than human reviewers in its requirement for explicit evidence.
Sorting algorithms are a corner stone of algorithmic education. The computational problem of sorting objects is simple and accessible, with a wide variety of algorithms with different approaches and characteristics available. Introducing sorting algorithms in an explorative manner, however, faces a major challenge: students tend to sort objects intuitively and holistically, bypassing the step of comparing two objects. This renders algorithmic differences and advantages moot. In order to address this, we expand on an idea from the CS Unplugged programme and present a set of weights and balances we developed for hands-on exploration. The entire set is easily 3D-printable, making it scalable and thus possible to provide enough material for entire classes with no additional supplies needed.
A widely used pedagogy in introductory programming courses involves students solving small code-writing exercises with well-defined problem statements. Although such exercises help students practice basic programming skills and become familiar with syntax, they offer little opportunity to interpret or clarify ambiguous requirements. However, real-world programming rarely provides such clarity. Developers must interpret incomplete specifications, ask questions, and resolve inconsistencies before writing code. At the same time, advances in large language models (LLMs) have made well-specified programming problems trivial to solve, allowing students to obtain correct solutions with little effort when task requirements are explicit. In this paper, we present a web-based tool that delivers 'Probeable Problems' which are programming tasks with deliberately ambiguous specifications that require students to clarify requirements before coding. Our tool extends prior implementations by enabling students to engage in a natural language dialogue with an AI "client" to uncover and resolve ambiguities, alongside a traditional mechanism for probing behaviour using code inputs. We deployed the tool in a large introductory programming course and examined howstudents engaged with the conversational interface. Students reported that the tool helped them appreciate the importance of asking specific questions and valued the realism of interacting with a simulated client, even when they found the tasks challenging. We also found that greater exploration of ambiguities was associated with fewer failed attempts, indicating that deliberate inquiry before coding supports more effective problem solving.
One promising application of Generative Artificial Intelligence (GenAI) in large introductory computing courses is to power automated tools for responding to student queries on online course forums, which can greatly reduce instructor workload and improve the timeliness of responses to students. However, there is a need to carefully evaluate the suitability and effectiveness of AI-generated responses. We address this by comparing bot responses, which are automatically generated using GenAI, and instructor responses, which are manually written by instructors, to student queries on an online forum for a large introductory programming course. Using a rubric derived from previously identified characteristics for effective feedback in programming education, we assess responses on qualities such as helpfulness and conciseness. We find that while bot responses excel in some areas, such as timeliness and comprehensiveness, they are often verbose and not always effective in handling administrative queries or in capturing the interpersonal aspects of human communication. Our findings suggest that AI-generated responses could be immediately published to students, but only as constructive starting points that instructors should review, comment on, or revise to ensure their accuracy and usefulness. This hybrid approach has the potential to create more effective learning environments that leverage the strengths of both automated and human feedback.
Introductory programming courses often rely on small code-writing tasks with clear and complete problem specifications. Such tasks limit opportunities for students to practice clarifying ambiguous requirements - a critical skill in real-world programming. Given the ubiquity of large language models (LLMs) that can produce accurate solutions for such well-specified tasks, students may question the relevance of learning to write code for such tasks. Probeable Problems, by contrast, deliberately omit essential details, encouraging students to think critically, identify ambiguities, and seek clarifications. This study builds on theories of metacognition, cognitive flexibility, and design thinking to explore the strategies used by human students when tackling ambiguous programming tasks. For three Probeable Problems, we analyze the contents of 40,000 student-written probes to investigate how thoroughly and efficiently students are able to explore ambiguities. Our results offer insights into the effective design of feedback for Probeable Problems, and emphasize the benefits of engaging learners with authentic, ill-structured problems to enhance critical thinking and metacognitive skills.
AI-powered chatbots and digital teaching assistants (AI TAs) are gaining popularity in programming education, offering students timely and personalized feedback. Despite their potential benefits, concerns about student over-reliance and academic misconduct have prompted the introduction of "guardrails" into AI TAs-features that provide scaffolded support rather than direct solutions. However, overly restrictive guardrails may lead students to bypass these tools and use unconstrained AI models, where interactions are not observable, thus limiting our understanding of students' help-seeking behaviors. To investigate this, we deployed a novel AI TA tool with optional guardrails in one lab of a large introductory programming course. As students completed three code writing and debugging tasks, they had the option to receive guardrailed help or use a "See Solution" feature which disabled the guardrails and generated a verbatim response from the underlying model. We investigate students' motivations and use of this feature and examine the association between use and their course performance. We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems. Additionally, low-performing students were more likely to use this feature and use it close to the deadline as they started assignments later. The predominant factors that motivated students to disable the guardrails were assistance in solving problems, time pressure, and lack of self-regulation. Our work provides insights into students' solution-seeking motivations and behaviors, which has implications for the design of AI TAs that balance pedagogical goals with student preferences.
Generative AI tools such as GitHub Copilot are very effective at producing functional code for many introductory programming tasks, which in turn is impacting how students learn to program. While such tools provide some clear benefits, including offering on-demand explanations, feedback, and debugging support, they also risk reducing students' engagement in critical evaluation and problem-solving, particularly when AI solutions can be accepted uncritically. Appropriate scaffolding may be effective in slowing down this process and promoting deliberate reasoning. Drawing inspiration from structured activities such as Parsons Problems, and building on recent ideas for quizzing students in real-time as code solutions are being built, we introduce AI Assist Problems, a tool that supports reflective code evaluation by requiring students to generate, review, and accept AI-suggested code one line at a time. While the tool constrains how code is built, it does not constrain what code can be written, allowing students to explore multiple valid pathways and produce diverse final solutions. We deployed this tool in a large first-year programming course (n = 996) and examined student interactions and perceptions, and the diversity of resulting code solutions. We found that greater exploration of AI suggestions predicts higher decision accuracy, and students' belief in the need to carefully review AI-generated code increased after use. We discuss how structured AI assistance can encourage reflection and critical evaluation as code-generation tools become an integral part of programming education.
Learning the process of writing a computer program is an important part of introductory computer programming education. Many educators attempt to teach programming process implicitly through live coding. One major obstacle preventing more direct instruction on process is that we have no empirically derived terms to describe it. In this paper, we begin to build a set of programming process descriptors by classifying topics and concepts that students pay attention to as they watch replays of other students writing code. We conducted think-aloud interviews with students in a CS1 course while they watched reconstructed playbacks of peers completing a programming task. We observe that, surprisingly, students talk about the temporal aspects of writing code almost twice as often as the code itself. Some participants did not recognize their own code-writing process when played back to them. We also find that students are surprisingly interested in what the programmer is thinking while writing. We identify clusters of similar phrases that students use to describe the evolution of code, such as being written "top to bottom" or "jumping back and forth". These descriptive features will, in future work, be used to develop and validate quantifiable measures of programming process that are grounded in empirical measures of student behavior.
The topic of Artificial Intelligence is gaining in importance and so is the need to learn about the core principles of AI. In this tool paper, we present Jumpy, an educational jump 'n' run game designed to provide hands-on experience with reinforcement learning. The students take the AI's point of view and act as an AI system themselves. While playing and thereby using reinforcement learning, they learn about key concepts like agents, reward functions, and policies. The tool is intended for use in classroom settings, with students working in pairs and participating in guided discussions. We evaluated Jumpy with more than 120 students aged 11-12. One third formed an experimental group while the remaining students served as a control group and used the tool after the formal evaluation. Both groups completed a questionnaire before and after the intervention. The results indicate a shift in students' attitudes towards AI and an increased familiarity with AI-related concepts.
This study examines the sense of belonging (SoB) among students in the Faculty of Science and Engineering at a regional Australian university. It explores relationships between SoB, trust in instructors, and academic anxiety, recognising their interconnections. The results suggest that distance from campus is not a significant factor in student belonging, trust, or anxiety. Computing students reported significantly higher belonging and lower anxiety than Science students, with Engineering students between. The most striking finding is the disconnect between quantitative and qualitative data: while online and on-campus students showed no statistical belonging differences, online students strongly articulated isolation concerns, suggesting that traditional belonging measures may not fully capture the online experience. For the growing population of online, part-time and mature-age STEM students, institutions must move beyond traditional campus-based belonging interventions to create meaningful connection opportunities that accommodate diverse life circumstances.
This paper presents a brief overview of a recent international competition focused on the creation of fictional historical characters using modern LLM technology. Competition entrants were able to design and create their own LLM-based personas, without the need for any prior programming skills or knowledge. This competition emphasised several contemporary LLM concepts, such as prompt engineering, hallucinations and jailbreaking, allowing entrants to explore the strengths and weaknesses of current LLM systems. The competition received submissions from over 200 prospective students, demonstrating its wide-ranging appeal and future potential.
First-year university students often experience high levels of stress and disengagement when transitioning into programming courses, contributing to high attrition rates and reduced confidence. These challenges are often linked to the abrupt shift from structured, guided exercises to open-ended programming assignments requiring the integration of multiple concepts. To support students during this transition, we designed and implemented a warm-up assignment that mirrors the structure of a major programming task in a simplified, low-stakes format. The warm-up aimed to reduce cognitive load, build familiarity with technical tools, and scaffold conceptual understanding. Drawing on thematic analysis of 220 student responses from a cohort of 800, we found that the intervention improved students' preparedness, reduced anxiety, and supported the transfer of skills to the main assignment. While the evaluation is qualitative, the results indicate that structured preparatory tasks can enhance early engagement and build confidence in novice programmers. This approach may inform scalable interventions in computing education and similar domains.
Large language models (LLMs) have broad implications for education in general, impacting the foundations of what we teach and how we assess. This is especially true in computing, where LLMs tuned for coding have demonstrated shockingly good performance on the types of assignments historically used in introductory CS (CS1) courses. As a result, CS1 courses will need to change in terms of the skills that are taught and how they are assessed. Computing education researchers have begun to study student use of LLMs, but there remains much to be understood about the ways that these tools affect student outcomes. In this paper, we present the design and evaluation of a new CS1 course at a large research-intensive university that integrates the use of LLMs for student learning. We describe the design principles used to create our course, our new course objectives, and evaluation of student outcomes and perceptions throughout the course as measured by assessment scores and surveys. Our findings suggest that 1) student exam performance outcomes, including differences among demographic groups, are largely similar to historical outcomes for courses without integration of LLM tools, 2) large, open-ended projects may be particularly valuable in an LLM context, and 3) students predominantly found the LLM tools helpful, although some had concerns regarding over-reliance on the tools.
