Background and ContextComputational thinking is a key part of primary school education, often taught with programmable robots. These robots are typically controlled using one of two modes: a "plugged-in" block-based software environment or an "unplugged" mode relying on direct physical interaction without screens.ObjectiveThis paper investigates whether an initial unplugged robot experience (using Ozobot robots) enhances primary school children's learning of concepts in a subsequent block-based programming environment (using Codey Rocky robots), and how this affects their overall enjoyment.MethodA quasi-experiment with 134 children from 8 school classes compares a treatment group (Ozobot then Codey Rocky workshop) with a control group (Codey Rocky only). Learning was assessed via pre- and post-test questionnaires on programming concepts; meanwhile, enjoyment and experiences were evaluated through surveys and observation.FindingsChildren enjoyed both approaches. While the unplugged Ozobot experience did not yield immediate conceptual gains, the results suggest it may have supported learning outcomes during the subsequent Codey Rocky workshop.ImplicationsWithin the scope of this intervention, preceding block-based coding with unplugged activities shows potential as a pedagogical strategy for priming students for abstract programming. Future research should investigate how to optimally combine these approaches.

Many popular Python libraries use C-extensions for performance-critical operations allowing users to combine the best of the two worlds: The simplicity and versatility of Python and the performance of C. A drawback of this approach is that exceptions raised in C can bypass Python's exception handling and cause the entire interpreter to crash. These crashes are real faults if they occur when calling a public API. While automated test generation should, in principle, detect such faults, crashes in native code can halt the test process entirely, preventing detection or reproduction of the underlying errors and inhibiting coverage of non-crashing parts of the code. To overcome this problem, we propose separating the generation and execution stages of the test-generation process. We therefore adapt Pynguin, an automated test case generation tool for Python, to use subprocess-execution. Executing each generated test in an isolated subprocess prevents a crash from halting the test generation process itself. This allows us to (1) detect such faults, (2) generate reproducible crash-revealing test cases for them, (3) allow studying the underlying faults, and (4) enable test generation for non-crashing parts of the code. To evaluate our approach, we created a dataset consisting of 1648 modules from 21 popular Python libraries with C-extensions. Subprocess-execution allowed automated testing of up to 56.5

Context: Code coverage is widely used as a software quality assurance measure. However, its effect, and specifically the advisable dose, are disputed in both the research and engineering communities. Prior work reports only correlational associations, leaving results vulnerable to confounding factors. Objective: We aim to quantify the causal effect of code coverage (exposure) on bug introduction (outcome) in the context of mature JavaScript and TypeScript open source projects, addressing both the overall effect and its variance across coverage levels. Method: We construct a causal directed acyclic graph to identify confounders within the software engineering process, modeling key variables from the source code, issue- and review systems, and continuous integration. Using generalized propensity score adjustment, we will apply doubly robust regression-based causal inference for continuous exposure to a novel dataset of bug-introducing and non-bug-introducing changes. We estimate the average treatment effect and dose-response relationship to examine potential non-linear patterns (e.g., thresholds or diminishing returns) within the projects of our dataset.

Machine learning (ML) libraries such as PyTorch and TensorFlow are essential for a wide range of modern applications. Ensuring the correctness of ML libraries through testing is crucial. However, ML APIs often impose strict input constraints involving complex data structures such as tensors. Automated test generation tools such as Pynguin are not aware of these constraints and often create non-compliant inputs. This leads to early test failures and limited code coverage. Prior work has investigated extracting constraints from official API documentation. In this paper, we present PynguinML, an approach that improves the Pynguin test generator to leverage these constraints to generate compliant inputs for ML APIs, enabling more thorough testing and higher code coverage. Our evaluation is based on 165 modules from PyTorch and TensorFlow, comparing PynguinML against Pynguin. The results show that PynguinML significantly improves test effectiveness, achieving up to 63.9

Developers create bug-reproducing tests that support debugging by failing as long as the bug is present, and passing once the bug has been fixed. These tests are usually integrated into existing test suites and executed regularly alongside all other tests to ensure that future regressions are caught. Despite this co-existence with other types of tests, the properties of bug-reproducing tests are scarcely researched, and it remains unclear whether they differ fundamentally. In this short paper, we provide an initial empirical study to understand bug-reproducing tests better. We analyze 642 bug-reproducing tests of 15 real-world Python systems. Overall, we find that bug-reproducing tests are not (statistically significantly) different from other tests regarding LOC, number of assertions, and complexity. However, bug-reproducing tests contain slightly more try/except blocks and “weak assertions” (e.g., ). Lastly, we detect that the majority (95

Although automated test generation is common in many programming domains, games still challenge test generators due to their heavy randomisation and hard-to-reach program states. Neuroevolution combined with search-based software testing principles has been shown to be a promising approach for testing games, but the co-evolutionary search for optimal network topologies and weights involves unreasonably long search durations. In this paper, we aim to improve the evolutionary search for game input generators by integrating knowledge about human gameplay behaviour. To this end, we propose a novel way of systematically recording human gameplay traces, and integrating these traces into the evolutionary search for networks using traditional gradient descent as a mutation operator. Experiments conducted on eight diverse Scratch games demonstrate that the proposed approach reduces the required search time from five hours down to only 52 minutes.

Creative approaches to attract students to software engineering at an early age are emerging, yet their differential impact on gender remains unclear. This study investigates whether design thinking's empathy-driven approach addresses the documented gender gap in interest in software engineering. In a 10-week curriculum-integrated design thinking software development course with 55 secondary school students aged 13-15 from two schools in Canada, we examined gendered differences in perceived gains in knowledge and interest, as well as in social-emotional experiences. Our results show that both girls and boys gained perceived knowledge in software development. However, girls showed significant improvements in self-efficacy, interest, engagement with sustainability topics, and well-being, including optimism, sense of usefulness, and social connectedness. Positive emotions were strongest during creative, collaborative phases, while technical tasks led to some boredom, especially among boys, though they still benefited overall. This suggests that human-centred design thinking might be one effective way to address gender equity challenges, though we need more differentiated technical implementations.

Test cases are essential for software development and maintenance. In practice, developers derive multiple test cases from an implicit pattern based on their understanding of requirements and inference of diverse test scenarios, each validating a specific behavior of the focal method. However, producing comprehensive tests is time-consuming and error-prone: many important tests that should have accompanied the initial test are added only after a significant delay, sometimes only after bugs are triggered. Existing automated test generation techniques largely focus on code coverage. Yet in real projects, practical tests are seldom driven by code coverage alone, since test scenarios do not necessarily align with control-flow branches. Instead, test scenarios originate from requirements, which are often undocumented and implicitly embedded in a project's design and implementation. However, developer-written tests are frequently treated as executable specifications; thus, even a single initial test that reflects the developer's intent can reveal the underlying requirement and the diverse scenarios that should be validated. In this work, we propose TestGeneralizer, a framework for generalizing test cases to comprehensively cover test scenarios. TestGeneralizer orchestrates three stages: (1) enhancing the understanding of the requirement and scenario behind the focal method and initial test; (2) generating a test scenario template and crystallizing it into various test scenario instances; and (3) generating and refining executable test cases from these instances. We evaluate TestGeneralizer against three state-of-the-art baselines on 12 open-source Java projects. TestGeneralizer achieves significant improvements: +31.66

The trend of embedding source code for machine learning applications also enables new opportunities in learning analytics in programming education, but which code embedding approach is most suitable for learning analytics remains an open question. A common approach to embedding source code lies in treating the code as a token sequence similar to natural language when training large language models (LLMs). However, in case of visual block-based programming languages like Scratch, this approach cannot be applied directly. While text-based representations of block-based code can be created to apply LLMs to this problem, other dedicated embedding models could potentially exhibit improved performance by capturing additional structural information. In this paper, we therefore instantiate four LLMs and five different popular embedding approaches for Scratch programs, create a token-prediction and two different classification tasks with corresponding datasets, and empirically evaluate the models on them. Our experiments demonstrate that a transfer of code embeddings to the educational environment of Scratch is feasible. The embedding models trained on large open Scratch datasets capture relevant structural and semantic information about the code to enable learning analytics like predicting functional correctness of student programs, in the typically small classroom setting without requiring further task-specific model fine-tuning.

Most traditional benchmarks for evaluating Large Language Models (LLMs) in software development suffer from a narrow focus, high risks of data contamination, and static difficulty levels that fail to keep pace with rapid model evolution. To address these limitations, we introduce CDBench, a novel zero-sum benchmark based on the Code Defenders mutation testing game. By pitting models against each other in a competitive environment—where “attackers” introduce code mutations and “defenders” create tests to detect them—CDBench establishes a dynamic difficulty curve that scales naturally without human intervention. Our experiments reveal that while LLMs can generate diverse mutations, they often struggle with code validity; nevertheless, the framework effectively distinguishes model capabilities, highlighting the superior test generation of models like Gemini 2.5 Pro while exposing the instruction-following limitations of reasoning models. These findings demonstrate that zero-sum games offer a viable, contamination-resistant solution to the stagnation of current evaluation methodologies.

Block-based programming environments like Scratch have become widely adopted in Computer Science Education, but the mouse-based drag-and-drop interface can challenge users with disabilities. While prior work has provided solutions supporting children with visual impairment, these solutions tend to focus on making content perceivable and do not address the physical interaction barriers faced by users with motor disabilities. To bridge this gap, we introduce MeowCrophone, an approach that uses voice control to allow editing code in Scratch. MeowCrophone supports clicking elements, placing blocks, and navigating the workspace via a multi-modal voice user interface that uses numerical overlays and label reading to bypass physical input entirely. As imperfect speech recognition is common in classrooms and for children with dysarthria, MeowCrophone employs a multi-stage matching pipeline using regular expressions, phonetic matching, and a custom grammar. Evaluation shows that while free speech recognition systems achieved a baseline success rate of only 46.4

Addressing learner heterogeneity in programming education is challenging due to variations in student speed, prior knowledge, and motivation. While differentiated instruction, such as tiered sequences, allows students to engage at appropriate difficulty levels, manually creating these resources is labour-intensive. This paper introduces CodeOwl, an AI-driven tool that automates the generation of tiered Parsons problems. Starting from a sample task or specific programming concepts, CodeOwl produces tiered sequences of Parsons problems automatically. We evaluated CodeOwl with a mixed-method framework comprising complexity analysis, expert ratings, and user studies. Analysis of 297 tiered sequences (three tiers each) revealed that 98.7

The recent hype around machine learning has fully captured software engineering research. Correspondingly, a variety of different ways to represent code as input to deep learning models have been proposed. These code embedding models are usually evaluated in terms of common metrics such as accuracy or bleu scores, on benchmark tasks such as predicting method names from their body. Although this evaluation approach is well established in research, it leaves open challenges for the deployment of these models in practice: First, comparing accuracy on standardised benchmark results conveniently avoids some of the challenges of actually running different prototype model implementations, which, however is necessary to apply the models in practice. Second, the models are usually trained and evaluated on abundantly available open-source training data, which may be very different from closed-source industrial code. Third, the deployment of machine learning models in an industrial environment does not only entail technical but also organisational challenges. Finally, while competitive accuracy or bleu scores may be indicative of relative model performance, they may not reflect to what extent the models are suitable for being used by developers. In this paper we describe our experience of evaluating and deploying state-of-research code embedding models in an industrial environment, and present lessons learned from our struggles with each of these questions.

Evaluating the quality of test suites for Deep Learning (DL) systems generally relies either on structural coverage criteria (e.g., Neuron Coverage), distribution-aware metrics (e.g., Surprise Adequacy), or mutation testing. While mutation testing offers the most reliable measure of fault-detection, it is often prohibitively expensive, as each mutant configuration may require retraining the model from scratch multiple times to establish detectability. This paper investigates to what degree the computationally cheaper coverage and distribution metrics can serve as reliable proxies for mutation testing. We measured the correlation (Spearman and Pearson) between nine adequacy metrics and the DeepCrime mutation score across four datasets (two classification, two regression) at ten test set size proportions. For classification tasks, all metrics except NeuraL Coverage (NLC) demonstrate moderate-to-strong, statistically significant correlations (Spearman 0.53 to 0.69) with the mutation score. For regression tasks, most metrics yield moderate, statistically significant positive Spearman correlations with growing test sets, though partial correlation analysis suggests these are largely mediated by test set size. Our findings suggest that standard adequacy metrics can nevertheless serve as proxies for mutation testing, particularly in classification DL systems.

Test flakiness disrupts Continuous Integration (CI) pipelines in monorepos, where a single flaky failure can block merges, trigger cascading pipeline resets, and waste substantial compute resources. We present the stage-aware flakiness mitigation strategy for BMW’s monorepo, spanning over 80 million lines of code and serving more than ten thousand engineers. Our strategy comprises three complementary systems: AutoSheriff detects and manages flaky tests in our pre-merge and merge pipelines through lightweight heuristics derived from merge-pipeline failure patterns; CTRA (CI Test Result Analysis) visualizes test health in post-merge pipelines and identifies culprit PRs; and TnF (Track&Fix) notifies responsible engineers and escalates unresolved failures. We evaluate AutoSheriff on 1.4 million test execution batches collected over one month, showing it effectively maintains the flakiness rate below 1% in our (pre-)merge stages.

A broken mainline in large-scale, fast-paced monolithic repositories disrupts thousands of developers and greatly affects overall productivity. Merge pipelines can help to protect the main branch from such breakages by validating Pull Requests (PRs) on speculative integration states (i.e., exactly the state resulting from merging the PR). When applied at scale, however, merge pipelines become bottlenecks as PR submission rates outpace CI capacity. Naive optimizations rely heavily on parallelization and, while effective, remain insufficient for fully resolving these bottlenecks. More sophisticated approaches have demonstrated the capability for further improvements; however, they require deep system integration, which can prevent widespread adoption. We present PREDIBATCH, a lightweight approach that uses machine learning to group low-risk PRs into batches, which reduces the required number of speculative integration states. predibatch relies solely on language-agnostic features from PR metadata and CI history, making it applicable to many CI systems. Our simulation-based evaluation on real-world data from BMW’s merge pipeline setup shows that PREDIBATCH significantly reduces average turnaround times by 8.3% and saves 13.3 % of compute resources compared to the current setup.

Search-based test-generation algorithms have countless configuration options. Users rarely adjust these options and usually stick to the default values, which may not lead to the best possible results. Tuning an algorithm’s hyperparameters is a method to find better hyperparameter values, but it typically comes with a high demand of resources. Meta-heuristic search algorithms—that effectively solve the test-generation problem—have been proposed as a solution to also efficiently tune parameters. In this work we explore the use of differential evolution as a means for tuning the hyperparameters of the DynaMOSA and MIO many-objective search algorithms as implemented in the Pynguin framework. Our results show that significant improvement of the resulting test suite’s coverage is possible with the tuned DynaMOSA algorithm and that differential evolution is more efficient than basic grid search.

Games are designed to challenge human players, but this also makes it challenging to generate software tests for computer games automatically. Neural networks have therefore been proposed to serve as dynamic test cases trained to reach statements in the underlying code, similar to how static test cases consisting of event sequences would do in traditional software. The Neatest approach combines search-based software testing principles with neuroevolution to generate such dynamic test cases. However, it may take long or even be impossible to evolve a network that can cover individual program statements, and since Neatest is a single-objective algorithm, it will have to be sequentially invoked for a potentially large number of coverage goals. In this paper, we therefore propose to treat the neuroevolution of dynamic test cases as a many-objective search problem. By targeting all coverage goals at the same time, easy goals are covered quickly, and the search can focus on more challenging ones. We extend the state-of-the-art many-objective test generation algorithms MIO and MOSA as well as the state-of-the-art many-objective neuroevolution algorithm NEWS/D to generate dynamic test cases. Experiments on 20 Scratch games show that targeting several objectives simultaneously increases Neatest's average branch coverage from 75.88% to 81.33% while reducing the search time by 93.28%.