Page Objects (POs) are a widely adopted design pattern for improving the maintainability and scalability of automated end-to-end web tests. However, creating and maintaining POs is still largely a manual, labor-intensive activity, while automated solutions have seen limited practical adoption. In this context, the potential of Large Language Models (LLMs) for these tasks has remained largely unexplored. This paper presents an empirical study on the feasibility of using LLMs, specifically GPT-4o and DeepSeek Coder, to automatically generate POs for web testing. We evaluate the generated artifacts on an existing benchmark of five web applications for which manually written POs are available (the ground truth), focusing on accuracy (i.e., the proportion of ground truth elements correctly identified) and element recognition rate (i.e., the proportion of ground truth elements correctly identified or marked for modification). Our results show that LLMs can generate syntactically correct and functionally useful POs with accuracy values ranging from 32.6% to 54.0% and element recognition rate exceeding 70% in most cases. Our study contributes the first systematic evaluation of LLMs strengths and open challenges for automated PO generation, and provides directions for further research on integrating LLMs into practical testing workflows.

Web applications are essential in modern society and require thorough testing to ensure reliability and dependability. End-to-End (E2E) testing is key to evaluating both front-end and back-end components of complex web applications.Recent research has focused on improving E2E test suites by reducing maintenance costs, mitigating flakiness, and enhancing test resilience. Yet, a major challenge remains the lack of a publicly available benchmark for comparing techniques. Developing test suites for research is complex and prone to bias, highlighting the need for a shared benchmark to facilitate comparisons and experimental validation.We address this gap by providing a set of test suites designed as a benchmark for E2E testing studies. They come with scripts and automated installers for seamless deployment of the application under test in Docker containers, enhancing their usability. Our benchmark consists of 36 Java Selenium WebDriver-based E2E test suites for 8 different web applications. Each application includes multiple test suites with varying characteristics, such as the use of the Page Object pattern and advanced waiting mechanisms. Additionally, for 4 out of the 8 web applications, we provide test suites for two different versions, enabling studies on test suite evolution. Our benchmark includes 1166 test scripts, featuring 492 Page Objects and 8,284 locators. To manage asynchronous behavior and reduce flakiness, the test suites incorporate 718 thread sleeps and 815 explicit waits. With a total of 51,838 lines of code (LOC) this dataset can potentially be used as a resource, for example, to evaluate test automation strategies, study test suite evolution, or explore flakiness mitigation techniques.

Context: Selenium is a widely used tool for end-to-end (E2E) web testing. However, it is often criticized for brittleness, slowness, and flakiness. In parallel, newer frameworks and Artificial Intelligence (AI) are reshaping the test automation landscape. Objectives: This study aims to investigate current practices, challenges, and emerging trends in Seleniumbased test automation. Methods: We designed and executed a large-scale online survey targeting software professionals who use Selenium. The questionnaire covered technical practices, tooling, AI usage, perceived challenges, and competing tools. Results: A total of 88 complete responses were analyzed using descriptive statistics and thematic coding. The results show that Selenium remains the dominant tool for regression and functional testing, primarily using the Page Object Model (POM) pattern. The most reported challenges are related to assertability, asynchrony, and brittleness. AI tools like ChatGPT are gaining traction for test generation. Playwright is the most prominent alternative. Conclusion: While Selenium is recognized as a cornerstone in many automation workflows, its limited native test-specific features present a significant drawback. The findings indicate an increasing demand for testingfocused improvements within the Selenium ecosystem, as well as for enhanced integration with AI-driven development tools.

Background: The adoption of Artificial Intelligence (AI) techniques in Software Testing (ST) has grown rapidly, particularly in response to the increasing complexity of modern systems. In GUI-based testing, AI is often cited as a promising means to automate repetitive tasks and improve testing efficiency. However, the actual use of AI in this domain remains underexplored through systematic empirical investigation.Objective: This study aims to analyze how AI is adopted in GUI-based testing, identifying the techniques and tools employed, the testing activities they support, and the perceived benefits and limitations.Method: We conducted a large-scale survey involving 107 participants from both academia and industry. The survey focuses on three core testing activities: test case definition, test oracle design, and test case optimization. It extends a prior study based on interviews with 45 industry practitioners.Results:Findings show that AI is primarily used to support test case definition, with techniques such as Natural Language Processing, Optimization, and Large Language Models (LLMs) being the most common. AI also provides support in test oracle design, where image processing and knowledge representation play key roles, and in test suite optimization, through the use of supervised learning, reinforcement learning, and search-based techniques.Conclusion: The paper identifies ongoing challenges and outlines future directions, including the need for transparent AI tools, guidelines for LLM integration, and the deployment of a continuously open survey to monitor trends in AI adoption over time.

Context: Test Automation (TA) techniques are crucial for quality assurance in software engineering but face limitations such as high test suite maintenance costs and the need for extensive programming skills. Artificial Intelligence (AI) offers new opportunities to address these issues through automation and improved practices. Objective: Given the prevalent usage of AI in industry, sources of truth are held in grey literature as well as the minds of professionals, stakeholders, developers, and end-users. To this aim, our study surveys grey literature to explore how AI is adopted in TA, focusing on the problems it solves, its solutions, and the available tools. Additionally, the study is complemented by expert insights. Methods: Over five years, we reviewed over 3,600 grey literature sources, including blogs, white papers, and user manuals, and finally filtered 342 documents to develop taxonomies of TA problems and AI solutions. We also cataloged 100 AI-driven TA tools and interviewed five expert software testers to gain insights into AI’s current and future role in TA. Results: The study found that manual test code development and maintenance are the main challenges in TA. In contrast, automated test generation and self-healing test scripts are the most common AI solutions. We identified 100 AI-based TA tools, with Applitools, Testim, Functionize, AccelQ, and Mabl being the most adopted in practice. Conclusion: This paper offers a detailed overview of AI’s impact on TA through grey literature analysis and expert interviews. It presents new taxonomies of TA problems and AI solutions, provides a catalog of AI-driven tools, and relates solutions to problems and tools to solutions. Interview insights further revealed the state and future potential of AI in TA. Our findings support practitioners in selecting TA tools and guide future research.

End-to-end (E2E) testing is an approach in which an application is automatically tested through scripts that simulate the actions a user would perform. Properly managing asynchronous interactions is crucial in this approach to avoid test failures and flakiness. In the Selenium WebDriver framework, this is typically addressed by using thread sleeps (which pause the test for a fixed time) or explicit waits (function calls that pause the test execution until a specified condition is met). Explicit waits require the selection of both a condition to wait for (e.g., element visibility, element clickability) and an element on which that condition applies. Since thread sleeps are unreliable and replacing them with appropriate explicit waits is a time consuming task, in this work, we leverage a Large Language Model (LLM) to assist testers in selecting the most appropriate explicit waits. We defined a structured procedure (a series of prompts) for engaging with the LLM and validated this approach empirically on three test suites affected by asynchronous waiting issues, as well as on 12 synthetic examples. Additionally, we compared our approach with SleepReplacer, the current state-of-the-art tool for replacing thread sleeps with explicit waits in E2E web test suites. The results show that the LLM-based approach can automatically replace the majority of thread sleeps in a test suite on the first attempt, outperforming SleepReplacer.

Context: While test automation offers numerous benefits, it also introduces significant challenges. Two challenges that developers and testers face on a daily basis, particularly when using Selenium WebDriver to test web applications, are driver management (involving tasks such as version identification, download, installation, and maintenance) and management of test lifecycle phases (using specific test libraries, as for example JUnit, and inserting annotations into the code). These manual tasks make test suite development particularly tedious, error-prone, and expensive. Recently, to ease the burden on developers and testers, some Java libraries have been proposed, called WebDriverManager and Selenium-Jupiter, capable of automatically carrying out the driver management process for Selenium WebDriver and simplifying the development of test suites. These libraries appear to be very promising but until now no one has experimentally evaluated their effectiveness. Objective: To investigate the effectiveness of WebDriverManager and Selenium-Jupiter in reducing driver management times and boilerplate code. Method: We designed and conducted a family of experiments (three for WebDriverManager and two for Selenium-Jupiter) with 104 master student participants from the University of Genoa, Italy (across academic years 2021/2022 and 2022/2023) and nine professional participants. Results: Results indicate that the adoption of Selenium WebDriver with WebDriverManager significantly reduces setup time for multi-browser test suites from 33% to 50% (depending on the tester experience). Additionally, Selenium-Jupiter reduces test suite development time significantly (20% on average). Although it also decreases total code length, the reduction is relatively small compared to overall code length. Conclusion: WebDriverManager and Selenium-Jupiter can be seen as valuable solutions for enhancing testers’ productivity by shortening the time needed to develop test suites and minimizing the amount of code to write.

Web applications quality is commonly assessed by executing End-to-End (E2E) test scripts interacting with those systems as a human tester would. To avoid setting up the web application state for each test script, testers usually create test scripts that may depend on others previously executed. However, the presence of dependencies prevents parallelization, a fundamental technique for speedup the execution of large test suites. In this paper, we present STILE, a tool for parallelizing the execution of E2E web test scripts that generates and executes a set of test schedules satisfying two important constraints: (1) every schedule respects existing test dependencies, and (2) all test scripts in the test suite are executed at least once. Moreover, STILE optimizes the execution by running only once the test scripts that are shared among the schedules. We empirically evaluated STILE on eight E2E test suites by comparing the execution time of STILE both with the sequential execution and with the parallel execution based on Selenium Grid. Our results show that STILE can reduce the execution time up to 80% w.r.t. the sequential execution and up to 50% w.r.t. Grid. Moreover, STILE provides a reduction in the CPUs usage (i.e., overall CPU-time) up to 75%.

Web applications are critical to modern life and require rigorous End-to-End (E2E) testing to ensure reliability across front-end and back-end components. While recent work has improved E2E testing reducing cost, flakiness, and increasing robustness a common benchmark is missing, hindering fair comparison and progress. This work introduces the first E2E benchmark dataset to address that gap: 12 Selenium WebDriver test suites for 8 web applications, packaged in Docker for easy deployment. It supports test evolution, automation, and flakiness studies, offering 389 Gherkin-based test cases, 283 Page Objects, 1,364 locators, and over 19k lines of code. By providing a reproducible, diverse foundation, this benchmark enables consistent evaluation of testing techniques and fosters advancement in E2E testing research.

Ensuring the quality of modern web applications through end-to-end (E2E) testing is crucial, especially for dynamic systems like single-page applications. Managing asynchronous calls effectively is a key challenge, often addressed using thread sleeps or explicit waits. While thread sleeps are simple to use, they cause inefficiencies and flakiness, whereas explicit waits are more efficient but demand careful implementation.This work explores extending SleepReplacer, a tool that automatically replaces thread sleeps with explicit waits in Selenium WebDriver test suites. We aim to enhance its capabilities by integrating it with ChatGPT, enabling intelligent and automated replacement of thread sleeps with optimal explicit waits. This integration aims to improve code quality and reduce flakiness.We developed a structured procedure for interacting with ChatGPT and validated it on three test suites and synthetic examples covering diverse cases.Results show that the LLM-based approach correctly replaces thread sleeps with explicit waits on the first attempt, consistently outperforming SleepReplacer. These findings support integrating ChatGPT with SleepReplacer to create a smarter, more efficient tool for managing asynchronous behavior in test suites.

Web applications play a crucial role in our daily lives, making it essential to employ testing methods that ensure their quality. Typically, Web testing automation frameworks rely on locators to interact with the graphical user interface, acting as connection points to the elements on a Web page. Nevertheless, locators are widely recognized as a major vulnerability in Web testing, as they are highly sensitive to the frequent changes in Web page structures caused by rapid software evolution. The adoption of the Page Object pattern to separate test logic from structural layout - supporting code reuse and maintainability - has generally led to more robust test cases. However, their implementation is a manually intensive task, and even automated support may require manual realignment efforts. Although gamification strategies have recently been integrated into the Web testing process to boost user engagement, using tasks and rewards aligned with testing activities, they have not yet been employed to enhance the robustness of locators and support the implementation of Page Objects. In this paper, we introduce TESTQUEST, a tool designed to improve test robustness by applying gamification to locators and Page Objects, boosting user engagement while guiding them toward the adoption of best practices.

High code quality, particularly in terms of maintainability, is crucial for ensuring that software remains efficient and adaptable over time, while minimizing long-term maintenance costs. As artificial intelligence continues to evolve, its application in software development offers new opportunities to improve code quality. This study investigates the use of Large Language Models (LLMs) to enhance software maintainability through code refactoring. The results indicate that LLMs can be effectively utilized for this purpose, with effectiveness varying depending on the model and the evaluation metric used. Although the study is based on a limited set of Python projects and specific prompting strategies, it provides a meaningful step toward understanding the broader applicability of LLMs in this context.

Automated testing is vital for ensuring the reliability of web applications. This paper presents a preliminary study on leveraging artificial intelligence (AI) models, specifically ChatGPT and Github Copilot, to generate test scripts for web end-to-end testing. Through experimentation, we evaluated the feasibility and effectiveness of AI language models in generating test scripts based on natural language descriptions of user interactions with web applications. Our preliminary results show that AI-based generation generally provides an advantage over fully manual test scripts development. Starting from test cases clearly defined in Gherkin, a reduction in development time is always observable. In some cases, this reduction is statistically significant (e.g., Manual vs. a particular use of ChatGPT). These results are valid provided that the tester has some skills in manual test script development and is therefore able to modify the code produced by the AI-generation tools. This study contributes to the exploration of AI-driven solutions in web test scripts generation and lays the foundation for future research in this domain.

Automated testing is crucial in software development to ensure that applications perform as intended. However, generating automated End-to-End (E2E) tests can be time-consuming and challenging, especially for junior developers. This study investigates the use of ChatGPT, a popular Generative Artificial Intelligence (GenAI) model, as an assistant in developing automated E2E test scripts for Android apps. We present an empirical study that compares the effort required to create E2E test scripts and the resulting reliability of these tests using two treatments: manually and assisted by ChatGPT. We used Gherkin, a domain-specific language that allows non-technical practitioners to define test scenarios using a human-readable syntax. Our findings indicate that using ChatGPT significantly reduces the time required to develop automated test scripts without compromising the reliability of the scripts. Statistical analysis shows a notable reduction in development time for the ChatGPT-assisted group compared to the manual group, with a large effect size. While the reliability of the tests did not show a significant difference between the two groups, the results suggest practical benefits in terms of efficiency.

Browser automation refers to the use of specific software to perform user tasks (such as navigation among web pages, interaction with forms, etc.) automatically in a web browser. It is commonly performed programmatically for automated testing of various types of applications, web-based, cloud-based, or composed of multiple microservices, although other uses are possible (e.g., web scraping or automating repetitive tasks). For many years, Selenium has been considered the defacto tool for browser automation. However, recently, we have witnessed the advent of alternative tools, such as Cypress, Puppeteer, or Playwright. This paper gives a comparative high-level overview and presents their architecture and key features. We expect this information will assist researchers and practitioners in making well-informed decisions regarding the browser automation tool that merits their attention.

Mutation testing is an important technique able to evaluate the bug-detection effectiveness of existing software test suites. Mutation testing tools exist for several languages, e.g., Java and JavaScript, but no solutions are available for managing the mutation testing process for entire web applications, in the context of end-to-end (E2E) web testing. In this paper, we propose Mutta , a novel tool able to automate the entire mutation testing process. Mutta mutates the various server source files of the target web application, runs the E2E test suite against the mutated web applications, and finally collects the test outcomes. To evaluate Mutta , we designed a case study using the mutated versions of the target web application with the aim of comparing the effectiveness of two different approaches to E2E web testing: (1) test cases based on classical assertions and (2) test cases relying on differential testing. In detail, Mutta has been executed on two web applications, each equipped with different test suites to compare assertions with differential testing. In this scenario, Mutta generated a large number of mutants (more than 15k overall), took into account the coverage information to consider only the mutants actually executed, deployed the mutated web app, ran the entire E2E test suites (about 87k tests runs overall), and finally, it correctly saved the test suite results. Thus, results of the case study show that Mutta can be successfully employed to automate the entire mutation testing process of E2E web test suites and, therefore, can be used in practice to evaluate the effectiveness of different test suites (e.g., based on different techniques, E2E frameworks, or composed by a different number of test scripts).

In today's technology-driven world, there is a growing interest in leveraging Artificial Intelligence (AI) to streamline software testing processes. Our research delves into GUI-based testing, a prominent technique for verifying software functionality. Preliminary findings from our industrial survey of 45 respondents provide insights into the use of AI in GUI-based software testing. The survey aims to understand how AI supports GUI-based testing, the AI techniques and tools used, and the perceived advantages and limitations. The collected results suggest a diffuse yet superficial utilization of AI-based mechanisms among GUI-based testers. Practitioners often employ AI techniques in a technology-agnostic way, treating commercial tools as black boxes. These findings underscore the need for additional research aimed at gaining a deeper understanding of the AI techniques and tools employed in industry and their intended purposes.

Web applications have become fundamental to our lives, thus testing methods ensuring their quality are essential. Usually, Web test automation frameworks and tools use locators to interact with the GUI, serving as hooks to the widgets within Web pages. However, locators are known to be one of the most significant points of fragility in Web testing, due to their susceptibility to rapid software evolution impacting the Web pages structure. While gamification strategies have been recently incorporated into the Web testing process to increase testers engagement, by means of tasks and rewards tailored around test activities, they have not yet been applied to locators robustness. In this paper, we introduce TestQuest, a tool under development that adapts gamification to improve locators robustness, by employing evaluation metrics and best practices designed to make testing via locators more engaging for testers, potentially enhancing the overall test quality.

A new advancement in test automation is the use of natural language processing (NLP) to generate test cases (or test scripts) from natural language text. NLP is innovative in this context and promises of reducing test cases creation time and simplifying understanding for "non-developer" software testers as well. Recently, many vendors have launched on the market many proposals of NLP-based tools and testing frameworks but their superiority has never been empirically validated. This paper investigates the adoption of NLP-based test automation in the web context with a series of case studies conducted to compare the costs of the NLP testing approach-measured in terms of test cases development and test cases evolution-with respect to more consolidated approaches, that is, programmable (or script-based) testing and capture & replay testing. The results of our study show that NLP-based test automation appears to be competitive for small- to medium-sized test suites such as those considered in our empirical study. It minimizes the total cumulative cost (development and evolution) and does not require software testers with programming skills.