Objective: This study explores the quality of tests in Dart, the main language for mobile application development with the Flutter framework. Methods: The study begins by using the DNose tool, used to detect 14 types of test smells in code written in the Dart language. Next, we evaluate the tool’s precision, accuracy, recall, and F1-score. Using this tool, we conduct a detailed analysis of tests in open-source projects extracted from the language’s central repository. Results: The study starts with a dataset of 5,410 Dart-language projects, from which we were able to clone 4,154 repositories after processing. Based on the cloned projects, we generated a dataset containing 907,566 occurrences of test smells. Through our analysis, we characterized the specific types of test smells most frequently encountered and identified their causes. We observed the presence of test smells in 74% of test files. Another noticeable characteristic among the analyzed projects was the scarcity of tests, with 1,873 projects having one or no tests, which led us to expand the number of analyzed projects to a broader base. Conclusion: This research makes a significant contribution by providing insights into the quality of tests in projects from Dart’s official repository, as well as by offering an open-source tool for detecting 14 types of test smells.
The main goal of reviews in software development is to grant the quality and correctness of the products. One of those tasks that we can highlight is image review. This process could require more time to process and tends to be less accurate due to human error. Considering that developers who perform this type of task also have other responsibilities, the longer the review takes, the more work time is required, and this can affect the time of deliverables in the industry. Therefore, the proposal of this research is to use Optical Character Recognition (OCR) and Computer Vision (CV), to automate the process of image review, for the interpretation and validation of the content, both textual and objects. To carry out this research a set of 100 images was used in manual and automated review process and was verified that the automated approach reduced the execution time by 96.45%, and reduced the number of errors by 4%, proving that it is feasible in this development context. The application of this tool helped to increase efficiency by reducing the time spent on manual reviews, increased the consistency of image analysis, ensuring better accuracy compared to manual reviews. It also promoted improvements in development and the team, through a more productive work environment.
Context: In this study, we investigated the integration of Large Language Models (LLMs) in test teams, aiming to understand how these tools can improve the test process and improve productivity. The main goal was to determine if the implementation of LLMs can enhance the productivity of test professionals. Methodology: We performed an empirical study with 14 participants, and the results were analysed quantitatively and qualitatively. For the quantitative analysis, we analysed the frequency of activities, and for the qualitative analysis, we analysed the participants’ perception of productivity through a scoring system, in which participants gave a score between 0 to 5 for their perception of learning from before and after the implementation of LLM in the team. Results: The results show a significant improvement in productivity (p-value = 0,0193) after the implementation of LLM. Furthermore, the qualitative analysis provided data on the impact of LLM in learning, use experience, trust perception and productivity. Conclusion: It was noted that the integration of LLMs in test teams can be a valuable tool to improve productivity and efficiency of test professionals.
Background:Website layouts often change with new design trends and front-end frameworks. Quality assurance is necessary during these changes, but manual testing takes much time and money. Manual tests are the standard way to maintain quality, but they are slow and expensive. Changes in graphical interfaces can cause errors or break features, which affects quality.Writing manual tests is the most time-consuming part of the process. Aims: This paper presents a tool that uses ChatGPT to create Natural Language Tests from screenshots and operator instructions. The goal is to reduce the time spent on manual test creation and to maintain quality in both the tests and the application. Method:We used two evaluation methods. First, we conducted a survey with 18 software testing professionals and students to compare tests made by ChatGPT and by humans. Second, we used Natural Language Processing techniques to measure the similarity between ChatGPT-generated tests and human-made tests. Results: The qualitative analysis showed that ChatGPT tests exceed human tests in completeness by a difference of 5.56%, achieving 36.11% acceptance rate. Human tests exceeded ChatGPT tests in clarity by 6.95%, reaching 41.67% acceptance rate. The quantitative analysis found that 66.7% of ChatGPT tests shared over 50% similarity with human tests. Conclusions: Our tool can help automate the creation of software tests. The similarity between AI-generated and human-made tests shows that this approach can save time and reduce costs, while keeping test quality at an acceptable level. This framework can help maintain quality during changes in website layouts and application development.
Software testing through fuzzing has gained widespread adoption for discovering security vulnerabilities, yet questions remain about its effectiveness in detecting subtle behavioral faults. This paper presents an empirical evaluation investigating the intersection of fuzzing and mutation testing, specifically examining how well fuzz targets perform when evaluated through mutation analysis. We conducted a systematic study using Bitcoin Core as our subject system, analyzing 10 different fuzz targets across various modules and evaluating their ability to detect 726 generated mutants. Our methodology involved executing fuzz targets with existing seed corpora and measuring mutation scores both with and without assertion statements to understand the role of explicit oracles in fault detection. Our findings reveal that contrary to previous studies suggesting fuzzing’s limited effectiveness in mutation testing, several fuzz targets achieved high mutation scores, with two targets reaching 100% mutant detection rates. We identified three key design patterns that significantly enhance mutant detection capabilities: (1) round-trip testing approaches that verify data integrity through serialization-deserialization cycles, (2) mathematical oracles that implement exact behavioral verification through redundant calculations, and (3) metamorphic relations that validate expected relationships between inputs and outputs. Our analysis demonstrates a positive correlation between assertion density in fuzz targets and mutation scores, with assertion removal causing substantial drops in detection rates across all targets. The study contributes empirical evidence that well-designed fuzz targets can effectively detect subtle behavioral faults beyond traditional crash-based vulnerabilities. Our results suggest that incorporating explicit oracles, metamorphic properties, and roundtrip verification mechanisms into fuzz target design significantly improves their mutation testing performance. These findings have practical implications for improving fuzzing methodologies and developing more comprehensive automated testing strategies for safety-critical software systems.
Context: In a software development project, the developers’ main task is to understand the requirements to then implement them without bugs. Particularly, this paper is inserted in the context of a test team from a software institute that receives new requirements daily. With the goal of verifying if the requirements were correctly implemented, test cases are created based on these requirements, to then confirm if the software is working. Problem: Each requirement must be associated to one or more test cases, thus allowing tracking and ensuring the validation during the test process. However, manually classifying requirements to their associated test cases demands time and effort. Solution: This work details a study about the viability of using the pre-trained models: BERT, Electra, RoBERTa, DeBERTa and XLNet in the process of multi-label classification of requirements, identifying the test cases associated to each requirement. Method: For this study, we considered requirements received by the test team from 2024 to April 2025, and 38 associated test cases. After collecting and preparing the data, we trained the models and measured their performance through the metrics: Hamming Loss, Jaccard, Precision, Recall and F1-Score. Results: Results showed that all models had low Hamming Loss values, specially the BERT model, with 0.012 Hamming Loss. After performing the Friedman and Conover tests, it was determined that there is a significative discrepany between the models. Conclusions: Therefore, this research’s results show that it is possible to use pre-trained models to classify requirements with associated test cases.
Exploratory Testing (ET) plays a critical role in ensuring software quality by uncovering edge cases and enabling adaptive evaluation of systems. However, ET is often constrained by its reliance on tester expertise, challenges in reproducing failures, and limited testing resources. Generative Artificial Intelligence (GenAI), particularly ChatGPT, presents opportunities to support ET by automating tasks such as test case generation. This paper reports on an empirical study comparing exploratory test suites generated by ChatGPT using six image-based prompts with those created by human testers. We evaluated the quality of the generated suites, identified the most effective prompt formats, and gathered feedback from professional testers. Results show that ChatGPT can produce high-quality test suites, though it may miss complex faults and occasionally generate incoherent cases. The best performance was achieved using sub-prompts focused on specific features. Testers found the AI-generated suites clear and practical but noted gaps requiring complementary manual scenarios. These findings suggest that ChatGPT can serve as a valuable aid in ET, while highlighting areas for further improvement.
Modern software systems are frequently developed and tested across multiple platforms (e.g., Windows, Linux, and macOS). In the software testing context, practitioners adapt the tests to run differently according to the target platform. These tests, which need to identify the platform on which they will be executed, are referred to as OS-specific tests. In this paper, we present an empirical study to evaluate how developers implement OS-specific tests in CPython, which is the reference implementation project for the Python programming language. Then, we mine this project and assess their OS-specific tests quantitatively. For this, we propose three research questions to assess the frequency, location, and issues related to OS-specific tests. Our results show that OS-specific tests are common in the CPython project, and 13% of the analyzed test files are OS-specific tests (RQ1). OS Identification APIs are used more frequently in test code (53.46%), and the test decorator @unittest.skipUnless is the most used to skip tests depending on the platform (RQ2).We also find 170 issues related to OS-specific tests in CPython, and Windows is the most targeted platform (RQ3). Lastly, we discussed practical implications for practitioners and researchers. Based on our findings, we emphasized the importance of testing across multiple platforms and examined the relationship between issues and OS-specific tests, among other insights.
Graphical User Interface (GUI) testing is an important task in mobile application development but remains time-consuming when done manually. With the rise of Large Language Models (LLMs), there is growing interest in their potential to automate software development tasks, including GUI test generation. This study investigates the ability of LLMs to generate GUI test intentions and scripts for Android applications using multimodal inputs, such as screenshots and structured UI data. We present an approach that combines visual and textual input from eight open-source Android apps and evaluate the performance of four LLMs. The results show significant variation in the models’ ability to generate GUI tests: Claude 3 Sonnet produced the most detailed and complete test sequences, GPT-4o generated simpler test scripts with fewer test intentions and user interactions, focusing on more basic user flows, while Gemini 2.5 and Gemma 3 presented moderate and similar results. These findings indicate that while LLMs can aid GUI test automation, their effectiveness varies significantly across models.
In embedded software projects for mobile devices, test requests are critical to ensuring product quality, yet complex dependencies on binaries, version control, and country-specific configurations often cause submission errors. These errors lead to inefficiencies and delays in testing pipelines. This paper reports on the development and evaluation of RAVEN (Request Assessment and Verification Engine), a real-time verification system built through an Action Research approach. RAVEN integrates data from JIRA, version control systems, and binary baselines to automatically verify over 30 criteria as users fill out request forms. Deployed in an industrial environment, RAVEN improved process efficiency—reducing approval lead time by 58%, halving submission iterations—and enhanced reliability, lowering the rejection rate of a key request type by 17 percentage points. It also improved the clarity of verification feedback and fostered greater requester accountability. A TAM-based questionnaire confirmed high user-perceived usefulness and ease of use. While some verification gaps remain, RAVEN demonstrated the value of proactive, educational verification. Future work includes extending its scope, integrating ML-based recommendations, and replicating the approach in other contexts.
Mobile systems are occupying an increasingly larger share of personal technology usage by the worldwide population, with Android being one of the most widely distributed operating systems in the market. With that in mind, testing security flaws in the Android environment has become a task of growing importance, considering that these tests should be conducted in a controlled environment. In this article, we propose a Container-Based solution for a system that creates an emulated Android device with customized kernels and Android versions. A working framework was implemented based on the proposed system, which was used to execute a case study using a vulnerability trigger as a security test example.
Behavior-Driven Development (BDD) has gained widespread adoption as a means to align software behavior with stakeholder expectations, yet maintaining high-quality scenarios remains challenging at scale. Manual review of Gherkin-based steps is often slow, inconsistent, and prone to oversight, leading to structural errors, semantic inconsistencies, and reduced maintainability. To address these issues, this work proposes a hybrid automated analysis framework that combines Natural Language Processing (NLP) and Machine Learning (ML) to improve both the clarity and correctness of BDD artifacts. The framework consists of two complementary components: a rule-based validator that inspects linguistic and structural adherence to established BDD conventions and a supervised classifier that assigns each step to one of three semantic categories: Precondition, Action, or Expected Result regardless of its original Gherkin keyword. Models were trained on a balanced synthetic dataset of 1,500 labeled steps and validated against a large-scale industrial repository from a leading global manufacturer of laptops and mobile devices, ensuring external validity. Performance was measured using macro-averaged accuracy, precision, recall, and F1-score, alongside statistical significance testing to compare algorithms. The best results were achieved by Support Vector Machines and gradient boosting models, which outperformed neural and transformer-based approaches. Designed for near real time operation, the framework can be applied to any Gherkin compatible library and any supported natural language, enabling broad applicability across projects. It integrates seamlessly into development workflows, including pull requests and CI/CD pipelines, to provide continuous, automated feedback on BDD scenarios. Findings suggest that hybrid NLP–ML solutions are effective in scaling quality assurance for agile both Test and DevOps teams, while reducing the manual effort required for review and maintenance.
The co-occurrence of test smells poses a challenge for refactoring test code, as these issues rarely appear in isolation. Consequently, developers often need to apply multiple transformations during refactoring, which increases complexity and effort. Therefore, it is essential to explore efficient strategies that reduce steps and practically allow the simultaneous refactoring of these problems. Although the literature already documents several test smells, the joint refactoring of these problems remains unaddressed. Therefore, this paper investigates the co-occurrence of test smells in test code (both at the class and test method level) and strategies to fix them efficiently, minimizing the number of transformations and preserving the behavior of the tests. To achieve this goal, we identified test smells using an automated tool in twenty-two open-source projects. We evaluated the co-occurrence between different types of test smells. Then, we ranked the thirty most frequent test smells pairs and suggested ways to refactor them in an integrated manner. Our findings can support developers in improving the quality of test cases, as our approach was designed with the industry’s reality in mind, where test smells often appear simultaneously.
Since 2015, a public legal institution in Brazil has relied on a large-scale information system to support its services. As the system's complexity and business relevance increased over the years, functional testing evolved from an ad-hoc, manual approach to a systematic, automated practice integrated into the development workflow. This paper reports on the evolution of the maturity stages, detailing the challenges encountered and key milestones achieved by adopting Selenium IDE automated tool testing. It also presents the main lessons learned by the testing team over this long-term process. Notably, process adjustments, regression testing strategies, and the adoption of Selenium IDE were instrumental in enhancing software quality. By sharing this trajectory, we aim to provide practical insight into other institutions facing similar constraints and demands.
Due to the inconsistency between the professional qualifications offered in academic training and the actual demands of the software industry, entering or qualifying for a career in Software Testing can be challenging for entry-level professionals. Combined with a lack of clarity regarding the desired requirements at different career levels in this field, the industry presents a lengthy list of hard skills that are essential for a career. It emphasizes the search for mid-level or senior-level professionals, which limits the hiring process. Therefore, this study goal to identify the professional requirements expected by the industry in the field of Software Testing. Thus were conducted: (i) a qualitative and quantitative study, based on the analysis of national and international job openings posted on professional social media; and (ii) a survey of professionals with technical experience in the Software Testing sector, to validate the results obtained in the previous study. The results were organized into Junior, Intermediate, and Senior levels, and categorized according to hard skills, soft skills, academic background, and certifications. Regarding technical skills, automation tools and agile methodologies, and interpersonal skills, communication, teamwork, and leadership stand out. Finally, this study contributes to a career path mapping in Software Testing, based on insights into the actual professional qualifications required for a professional in this field.
This paper proposes an approach to support the early validation of non-functional communication requirements in the development of automotive embedded software, particularly for Advanced Driver Assistance Systems (ADAS) that incorporate V2X (Vehicle-to-Everything) communication as a complementary input. The approach integrates the ns-3 network simulator as a systematic tool during the initial stages of the V-model, focusing on the data link layer, specifically the MAC and PHY sublayers, using the IEEE 802.11p protocol and its evolution, IEEE 802.11bd. Simulations are conducted in parallel with requirements elicitation, enabling the assessment and refinement of key parameters such as latency, reliability, packet loss, and communication range. By anticipating the analysis of these properties through simulation, the approach aims to reduce rework in later verification phases and improve the definition of system architecture. The ns-3 environment offers a reproducible, scalable, and low-cost solution for evaluating communication behavior under adverse conditions, contributing to more informed decisions early in the automotive software lifecycle.
Context. Manual unit test creation is a cognitively intensive and time-consuming activity, prompting researchers and practitioners to increasingly adopt automated testing tools. Recent advancements in language models have expanded automation possibilities, including unit test generation, yet these models raise substantial sustainability concerns due to their energy consumption compared to conventional, specialized tools. Goal. Our research investigates whether the energy overhead associated with employing a small language model (SLM) for unit test generation is justified compared to a conventional, lightweight testing tool. We compare and analyze the energy consumption incurred during test suite generation, as well as the fault-finding effectiveness of the resulting test suites, for an SLM (Phi-3.1 Mini 128k) and Pynguin, a purpose-built tool for unit test generation. Method.We posed two research questions: (i) What is the difference in energy usage between Phi and Pynguin during the generation of unit test suites for Python programs?; and (ii) To what extent do unit test suites generated by Phi and Pynguin differ in their fault-finding effectiveness? To rigorously address the first research question, we employed Bayesian Data Analysis (BDA). For the second research question, we conducted a complementary empirical analysis using descriptive statistics. Results. Our Bayesian analysis provides robust evidence indicating that Phi consistently consumes significantly more energy than Pynguin during test suite generation. Conclusions. These findings underscore significant sustainability concerns associated with employing even SLMs for routine Software Engineering tasks such as unit test generation. The results challenge the assumption of universal energy efficiency benefits from smaller-scale models and emphasize the necessity for careful energy consumption evaluations in the adoption of automated software testing approaches.
