Machine learning (ML)-based Artificial Intelligence (AI) systems rely on training data to function, but the internal workings of how ML models learn from this data are often black-box. Influence analysis provides valuable insights into the model's behavior by evaluating the effect of individual training instances on the model's predictions. However, calculating the influence of each training data can be computationally expensive. In this paper, we propose a proxy model-based approach to influence analysis called Proxima. The main idea of our approach is to use a subset of training instances to create a proxy model that is simpler than the original model, and then use the proxy model and the subset of training instances to perform influence analysis. We evaluate Proxima on ML models trained using seven real-world datasets. We compare Proxima to two state-of-the-art influence analysis tools, i.e., FastIF and Scaling-Up. Our experimental results suggest that the proposed approach can successfully perform and speed up the influence analysis process, and, in most cases, perform better than FastIF and Scaling-Up.
Finding critical scenarios is essential in testing autonomous driving and automated driving functions. Such scenarios describe a sequence of interactions between the autonomous vehicle or the vehicle equipped with automated driving functions and the environment, i.e., other cars, pedestrians, and the current road conditions, which challenge the system we want to test. In this paper, we present a search-based testing solution utilizing genetic algorithms for test generation coupled with a traffic simulator. As a fitness function, we rely on the amount of emergency braking required to prevent crashes. In addition, we compare two types of hyperparameter tuning. One type uses combinations of hyperparameters obtained from previous papers. The other is based on a design of experiment method. We show that the genetic algorithm using the design of experiments method for hyperparameter tuning outperforms the other implementation in terms of criticality (i.e., the time of emergency braking) and diversity. Furthermore, we show that both genetic algorithm implementations are superior to pure random testing in the application context of autonomous and automated driving.
AI-based systems possess distinctive characteristics and introduce challenges in quality evaluation at the same time. Consequently, ensuring and validating AI software quality is of critical importance. In this paper, we present an effective AI software functional testing model to address this challenge. Specifically, we first present a comprehensive literature review of previous work, covering key facets of AI software testing processes. We then introduce a 3D classification model to systematically evaluate the image-based text extraction AI function, as well as test coverage criteria and complexity. To evaluate the performance of our proposed AI software quality test, we propose four evaluation metrics to cover different aspects. Finally, based on the proposed framework and defined metrics, a mobile Optical Character Recognition (OCR) case study is presented to demonstrate the framework's effectiveness and capability in assessing AI function quality.
Artificial Intelligence (AI) has rapidly evolved, offering transformative potential across various sectors, including healthcare, finance, education, and transportation. However, its deployment raises significant ethical, societal, and policy concerns that have necessitated careful consideration and action since its inception. This panel will discuss the multifaceted implications of AI, focusing on ethical dilemmas such as privacy, bias, transparency, and accountability, as well as the broader societal impacts on employment, social equity, and human interaction. Additionally, it examines the current state of AI policy and governance, highlighting the need for comprehensive frameworks that balance innovation with responsible AI or Machine Learning (ML) use. This panel will discuss these critical issues and aims to contribute to the current and future development of AI systems that are not only technologically advanced but also ethically sound and socially beneficial.
Real-time monitoring plays a vital role in software testing by facilitating the identification and diagnosis of issues during canary deployments or limited-scale testing phases. This approach serves as a method for testing and issue identification in the deployment phase. Furthermore, efficiently detecting anomalies in software instrumentation data is a crucial aspect of real-time monitoring. However, due to the volatility of software telemetry data, achieving accurate anomaly detection is quite challenging. In this paper, we present an efficient transformer-based framework designed for real-time anomaly detection. This approach called TEAD deeply explores the temporal correlations in time series data to uncover anomalous patterns, ensuring stability in predictions amidst fluctuating data. Empirical validation conducted at a prominent Internet company highlights the effectiveness of TEAD, emphasizing its ability to improve both efficiency and accuracy in real-time monitoring. Despite its significance, the method is constrained by its business-centric nature, warranting further cross-industry dissemination research. Overall, this study provides a novel solution for the real-time monitoring phase of software testing, validating the applicability and effectiveness of transformer-like models in this field.
