Mutation analysis is a fault-based test criterion that evaluates test suite quality by introducing artificial faults (mutants) into the program and checking that the tests detect those mutants. Although effective, it is computationally expensive because it generates many mutants, even for small programs, making the approach costly. Various static mutant selection techniques (i.e., techniques that do not require test executions) have been proposed to reduce the high computational cost of mutation analysis by selecting subsets of mutants. The effectiveness of these techniques has mostly been evaluated by their capacity to select mutants that have a similar mutation score (the proportion of mutants detected by the tests) to the whole set. However, this metric alone provides only a partial assessment of each technique’s advantages and optimal use cases. We evaluate 10 static mutant selection techniques when using different proportions of mutants and assess their effectiveness using four evaluation metrics, including minimal mutants coverage, subsumption tree coverage and minimum test size in addition to mutation score. We use the MediumDarwin mutation tool to conduct experiments on 14 open source Java projects ranging from 41 to 17KLoC. We also examine how sampling rate and program size influence the performance of the techniques and the extent to which the four evaluation metrics are correlated.

Solving topological grid puzzles requires reasoning over global spatial invariants such as connectivity, loop closure, and region symmetry and remains challenging for even the most powerful large language models (LLMs). To study these abilities under controlled settings, we introduce TopoBench, a benchmark of six puzzle families across three difficulty levels. We evaluate strong reasoning LLMs on TopoBench and find that even frontier models solve fewer than one quarter of hard instances, with two families nearly unsolved. To investigate whether these failures stem from reasoning limitations or from difficulty extracting and maintaining spatial constraints, we annotate 750 chain of thought traces with an error taxonomy that surfaces four candidate causal failure modes, then test them with targeted interventions simulating each error type. These interventions show that certain error patterns like premature commitment and constraint forgetting have a direct impact on the ability to solve the puzzle, while repeated reasoning is a benign effect of search. Finally we study mitigation strategies including prompt guidance, cell-aligned grid representations and tool-based constraint checking, finding that the bottleneck lies in extracting constraints from spatial representations and not in reasoning over them. Code and data are available at github.com/mayug/topobench-benchmark.

Software testing is essential to ensure the reliability and correctness of software systems. However, the effectiveness of testing is highly dependent on the quality of the test suites themselves. Mutation analysis, a powerful technique for evaluating the quality of tests, introduces small changes into the code and checks whether the tests detect them. Despite its strengths, mutation analysis faces challenges in scalability due to the high computational cost of compiling and running tests against mutants. This paper presents MediumDarwin, a substantially upgraded version of the original LittleDarwin, initially introduced as a research prototype. Our enhanced version retains the original foundational architecture but introduces significant new capabilities and performance optimisations that transform it into a robust platform for both industrial use and advanced research. The enhancements made to LittleDarwin include: (1) persistent storage of mutation results in a relational database to facilitate advanced analysis, (2) coverage-based test selection optimisation to minimise test executions, (3) implementation of mutant schemata to reduce compilation overhead, (4) enhanced mutation operators alongside safeguards against non-compilable mutants, and (5) dynamic subsumption graph computation for efficient mutant analysis. These innovations collectively improve the tool's scalability and practical utility in software quality assurance in both industrial and research contexts. A screencast demonstrating the use of MediumDarwin is available at https://www.youtube.com/watch?v=Zsd3pZt63AE.

We propose RocketScience, an open-source contrastive VLM benchmark that tests for spatial relation understanding. It is comprised of entirely new real-world image-text pairs covering mostly relative spatial understanding and the order of objects. The benchmark is designed to be very easy for humans and hard for the current generation of VLMs, and this is empirically verified. Our results show a striking lack of spatial relation understanding in open source and frontier commercial VLMs and a surprisingly high performance of reasoning models. Additionally, we perform a disentanglement analysis to separate the contributions of object localization and spatial reasoning in chain-of-thought-based models and find that the performance on the benchmark is bottlenecked by spatial reasoning and not object localization capabilities. We release the dataset with a CC-BY-4.0 license and make the evaluation code available at: https://github.com/nilshoehing/rocketscience

Pose estimation systems are used in a variety of fields, from sports analytics to livestock care. Given their potential impact, it is paramount to systematically test their behaviour and potential for failure. This is a complex task due to the oracle problem and the high cost of manual labelling necessary to build ground truth keypoints. This problem is exacerbated by the fact that different applications require systems to focus on different subjects (e.g., human versus animal) or landmarks (e.g., only extremities versus whole body and face), which makes labelled test data rarely reusable. To combat these problems we propose MET-POSE, a metamorphic testing framework for pose estimation systems that bypasses the need for manual annotation while assessing the performance of these systems under different circumstances. MET-POSE thus allows users of pose estimation systems to assess the systems in conditions that more closely relate to their application without having to label an ad-hoc test dataset or rely only on available datasets, which may not be adapted to their application domain. While we define MET-POSE in general terms, we also present a non-exhaustive list of metamorphic rules that represent common challenges in computer vision applications, as well as a specific way to evaluate these rules. We then experimentally show the effectiveness of MET-POSE by applying it to Mediapipe Holistic, a state of the art human pose estimation system, with the FLIC and PHOENIX datasets. With these experiments, we outline numerous ways in which the outputs of MET-POSE can uncover faults in pose estimation systems at a similar or higher rate than classic testing using hand labelled data, and show that users can tailor the rule set they use to the faults and level of accuracy relevant to their application.

Analysis of rugby match and training footage is particularly useful for coaches and players to understand and improve their tackling technique, and potentially lower the rate of injuries. Machine learning models (in particular for pose estimation) promise to streamline rugby analysis. However models trained for “general purpose” computer vision tasks, such as pose estimation and object detection, frequently fail as a result of the challenging conditions and significant domain shift that rugby footage presents: high-impact, close-contact play causes problems such as occlusions, motion blur, and unconventional body orientations. It is therefore crucial to understand the specific conditions which cause these systems to fail so they can be prioritised during pre-processing and expensive manual data collection. In this paper we leverage Met-Pose, a metamorphic testing system to understand the specific conditions that cause pose estimation systems to fail. Metamorphic testing is particularly advantageous as this approach side-steps the need for costly, manually labelled data. Our ongoing project on applying pose estimation for rugby analysis employs MediaPipe, a popular, widely used pose estimation system, on rugby broadcast footage. We show how applying metamorphic testing to a sport analytics application can reveal situations that challenge the model without the need for any manual data labelling. For example, our results show that in this context, MediaPipe is particularly sensitive to motion blur and colour loss, but less so to lighting and resolution changes. Furthermore, we show how this process can be adapted to focus on particular aspects of an application by proposing a new metamorphic rule exploring the effect of including or excluding context on MediaPipe’s results. Our results show where MediaPipe struggles in complex, real-world sporting scenarios and also offer concrete insights for improving data augmentation, data collection and system design in sports analytics.

Program synthesis is an important challenge that has attracted significant research interest, especially in recent years with advancements in Large Language Models (LLMs). Although LLMs have demonstrated success in program synthesis, there remains a lack of trust in the generated code due to documented risks (e.g., code with known and risky vulnerabilities). Therefore, it is important to restrict the search space and avoid bad programs. In this work, pre-defined restricted Backus-Naur Form (BNF) grammars are utilised, which are considered 'safe', and the focus is on identifying the most effective technique for grammar-obeying program synthesis, where the generated code must be correct and conform to the predefined grammar. It is shown that while LLMs perform well in generating correct programs, they often fail to produce code that adheres to the grammar. To address this, a novel Similarity-Based Many-Objective Grammar Guided Genetic Programming (SBMaOG3P) approach is proposed, leveraging the programs generated by LLMs in two ways: (i) as seeds following a grammar mapping process and (ii) as targets for similarity measure objectives. Experiments on a well-known and widely used program synthesis dataset indicate that the proposed approach successfully improves the rate of grammar-obeying program synthesis compared to various LLMs and the state-of-theart Grammar-Guided Genetic Programming. Additionally, the proposed approach significantly improved the solution in terms of the best fitness value of each run for 21 out of 28 problems compared to G3P.

Concerns about the cumulative effects of head acceleration events in rugby are growing, but how tackle events lead to direct head contact in women's rugby remains underexplored. This cross-sectional study aimed to develop and evaluate a machine learning model to identify characteristics associated with direct head contact and incorrect tackler head placement in elite women's rugby. Match situational and precontact technical characteristics (n = 31) from 1500 randomly selected tackle events were coded visually and retrospectively analyzed from the 2022-23 Women's Six Nations Championship. A machine learning model was developed and evaluated using a grid search with 5-fold cross-validations and F1 scores (i.e., a measure of predictive performance). The top modifiable characteristics associated with the target outcomes across 100 test sets were identified by mutual importance and decision tree modeling. The top modifiable characteristics linked to direct head contact to the tackler were incorrect head placement, coming to balance, and foot placement. Tackle direction, point of contact on the tackler, and multiplayer tackles were key characteristics for incorrect tackler head placement. Tackler drop height, front/oblique tackle direction, and multiplayer tackles were strongly associated with direct head contact to the ball-carrier. Incorrect tackler head placement, the direction of tackle, tackler drop height, and multiplayer tackles are key characteristics in direct head contact events in elite women's rugby. Addressing these characteristics should be prioritized in contact training practices, education resources, and law enforcement to enhance player safety and direct head contact events in the women's game.

Mutation analysis is known for its effectiveness in assessing the quality of test suites. However, it is a costly approach, as it generates many mutants even for small programs. Generating, compiling, and executing these mutants is a slow and resource-intensive process. Many mutant selection techniques have been proposed to reduce the number of mutants considered and thus lower the cost of mutation analysis. Yet, the effectiveness of all these techniques has not been systematically compared to understand the advantages of each technique and when they should be used. This work focuses on static mutant selection techniques (i.e., those that do not require executing tests against the mutants to select them) and compares their effectiveness in approximating the mutation score of a test suite. Using a dataset of 15 Java projects of different sizes and application domains and the LittleDarwin mutation tool, we compare the performance of ten state of the art static mutant selection techniques under different settings. Results show that no one technique provides better results than the others in all situations, i.e., across all projects and mutants sampling rates. Still, we found that stratification based selection techniques mostly outperform the other techniques (in up to 129 out of 135 of the studied settings). In particular, stratified sampling based on the source file in which the mutants appear provided the best approximation of the mutation score in nearly half the cases considered in our experiments (up to 68/135). Additionally, we found that the quality of a project's test suite had a noticeable influence on the selection techniques' performance. Indeed, for lower quality test suites, the selected mutants performed worse and strongly under-estimated the mutation score.

This cross-sectional study aims to identify the situational characteristics, ball-carrier technical variables, and Tackle Ready recommended techniques associated with performance outcomes in elite women's Rugby Union. Using retrospective video analysis, 43 tackler and ball-carrier technical characteristics for 1500 tackle events in the 2022-23 Women's Six Nations Championship were assessed, considering match situation and performance outcomes. Rate ratio (RR) was determined using propensity rates. Effective tackles were associated with match situations involving two defenders, forwards tackling forwards, defensive teams moving forwards, and tackles initiated closer to attackers at ball reception. Seven out of the 22 coded Tackle Ready techniques were significantly associated with superior performance outcomes. Techniques associated with the greatest likelihood of effective tackle outcome included wrap and clamp (RR 46.8) and ear to body (RR 20.9). Tackles made to the hip and leg of the ball-carrier increased the risk of missed tackles. This study provides the first analysis of tackle characteristics associated with performance outcomes in women's rugby, providing a reference to inform coaching practice and the implementation of tackle education resources and law changes. Further research is warranted to explore techniques associated with injury risk, and interactions between match situations and subsequent tactical/technical tackle actions.

A key ingredient in any successful genetic programming is robust initialisation. Many successful initialisation methods used in genetic programming have been adapted to use with grammatical evolution, to varying levels success. This paper examines the effectiveness of some of the most popular of these initialisation techniques on structured grammatical evolution. Namely, we investigate Sensible Initialisation and Probabilistic Tree Creation 2, as well as the standard initialisation procedure used in structured grammatical evolution, Grow. We also propose a novel procedure called Local Optimised Probabilistic Tree Creation 2, which runs a quick greedy optimisation on the trees created. We do this using using two different grammar specifications, both with and without protected operators, and using an error based and correlation based fitness function. We examine their performance, as well as the diversity of solutions they create, on 8 well-known benchmarks. We observe that Local Optimised Probabilistic Tree Creation 2 created the fittest, or joint fittest, initialisation populations on every benchmark considered, bar one. Local Optimised Probabilistic Tree Creation 2 remained the best initialisation procedure when the grammar specification was changed, confirming it’s robustness. This did not necessarily result in overall better runs, however, and SGE runs with below average initialisation performance were seen to overcome their “bad start”. The diversity of solutions, particularly fitness diversity, at the end of the run was lower for Local Optimised Probabilistic Tree Creation 2 and Probabilistic Tree Creation 2 than for both sensible initialisation and grow. Local Optimised Probabilistic Tree Creation 2 was seen to take between 8 and 20 times longer to create the initial population than the other methods. This article is an extension of a paper which originally appeared at the Grammatical Evolution Workshop held as part of GECCO 2023.

Android automated test input generation has been a highly researched topic for over a decade and has shown promising results with a variety of approaches. Random input generation is commonly used and the easiest to maintain, but ultimately inefficient. Systematic and search-based approaches produce effective tests but require a disproportionally large generation runtime. Model-based approaches have the additional overhead of modelling the application under test (AUT) but they result in a faster test generation. In this paper we present Precise AnDRoid Automated Input Generation (PADRAIG), a model-based test input generation framework that uses a detailed control flow model of the AUT to generate tests that can achieve higher line coverage, with a lower test generation runtime than the state of the art. We compare the line coverage achieved, and the generation runtime of PADRAIG against 3 state of the art tools, each of which uses a different test input generation technique. Our results, using 19 randomly selected Android apps from the F-Droid application store, show that PADRAIG achieves, on average, 16% more coverage of the AUT than the state of the art and it can generate tests with, on average, 84% less runtime.

Primary School education in Ireland is in the process of updating its pedagogical approaches, embracing new teaching methods and desiring new learning outcomes. In particular, the new curriculum emphasizes developing skills necessary for modern-day life. Peer-led interactive group learning is an effective method of teaching those desired skills. One such technology embracing these methods of learning is the “Magical Leaders” programme. However, creating an environment that allows for optimum delivery of the content is difficult as the teacher needs to deal with the new method of content delivery as well as any behavioral or social issues that exist in the classroom. We investigate three factors that affect the successful delivery of the “Magical Leaders” programme: group composition; peer leader selection; and preparation time. We find that classes in which the teacher selects the groups complete the challenges more than 10 min quicker than those classes in which the peers select the groups or they are randomly assigned. The mean time to complete each challenge was approximately the same for groups in which the peer leaders were selected by the teacher and those groups were volunteers. However, the number of volunteer-led groups that finished on or before schedule was higher. Groups with peer leaders who completed less than 30 min of preparation time finished on average 9 min quicker than groups with peer leaders who did more than 30 min of preparation time. However, while the mean time is much less the number of classes finishing on time is higher for those with more preparation time, 60

Object detection often struggles when applied to low-resource, domain-specific datasets. This challenge is exacerbated when dealing with sports-related data such as rugby, where fast-paced gameplay and tackles result in frequent instances of motion blur and occlusion, representing a substantial domain-shift from widely available pre-trained models. Given the high cost of manual labelling, we seek to determine whether we can minimise the number examples needed for fine-tuning by identifying implausible label classifications made by pre-trained object detection models. We do this using a coarse-grained labelling approach in the absence of detailed ground truth bounding boxes, allowing us to determine whether a label is implausible within the context of a rugby pitch. This is done to maximize the information provided by each example used for fine-tuning with the goal of minimizing the number of examples needed. Our results show that using pool-based, single-step uncertainty sampling to select examples from a subset of frames with implausible labels improves the model performance. More specifically, we show that fine-tuning on frames with the lowest confidence scores first can lead to greater performance after roughly 30 examples.

The ability to automatically generate code, i.e., program synthesis, is one of the most important applications of artificial intelligence (AI). Currently, two AI techniques are leading the way: large language models (LLMs) and genetic programming (GP) methods—each with its strengths and weaknesses. While LLMs have shown success in program synthesis from a task description, they often struggle to generate the correct code due to ambiguity in task specifications, complex programming syntax, and lack of reliability in the generated code. Furthermore, their generative nature limits their ability to fix erroneous code with iterative LLM prompting. Grammar-guided genetic programming (G3P, i.e., one of the top GP methods) has been shown capable of evolving programs that fit a defined Backus–Naur-form (BNF) grammar based on a set of input/output tests that help guide the search process while ensuring that the generated code does not include calls to untrustworthy libraries or poorly structured snippets. However, G3P still faces issues generating code for complex tasks. A recent study attempting to combine both approaches (G3P and LLMs) by seeding an LLM-generated program into the initial population of the G3P has shown promising results. However, the approach rapidly loses the seeded information over the evolutionary process, which hinders its performance. In this work, we propose combining an LLM (specifically ChatGPT) with a many-objective G3P (MaOG3P) framework in two parts: (i) provide the LLM-generated code as a seed to the evolutionary process following a grammar-mapping phase that creates an avenue for program evolution and error correction; and (ii) leverage many-objective similarity measures towards the LLM-generated code to guide the search process throughout the evolution. The idea behind using the similarity measures is that the LLM-generated code is likely to be close to the correct fitting code. Our approach compels any generated program to adhere to the BNF grammar, ultimately mitigating security risks and improving code quality. Experiments on a well-known and widely used program synthesis dataset show that our approach successfully improves the synthesis of grammar-fitting code for several tasks.

One of the most significant challenges to sign language recognition (SLR) today is the low resource nature of sign language datasets, with many datasets being extremely low resource. Transfer learning is therefore a promising, and likely indispensable, method of increasing recognition performance. The use of pose estimation models, which are typically trained on a large and diverse population, can also aid generalization for extremely low resource sign languages. However, research on transfer learning for pose estimation keypoints as inputs has been limited. In this work, we explore transfer learning as a means to improve SLR classification performance for the extremely low resource Irish Sign Language (ISL). We show that transfer learning on larger datasets containing secondary sign languages significantly improves performance on our target sign language, ISL. To understand these results and the attributes that make one dataset better than another for pre-training, we analyse the linguistic relationships between these datasets. We find that certain attributes of datasets are associated with better transfer learning performance. We hope that our findings will not only motivate further research into transfer learning for pose keypoint-based SLR but also act as a practical guide to researchers on choosing the most suitable datasets with which to pre-train models.