Evaluating the performance of Automatic Speech Recognition (ASR) systems is very relevant for selecting good techniques and understanding their advantages and limitations. ASR systems are usually evaluated by adopting test sets of audio speeches, ideally with different difficulty levels. In this sense, it is important to analyse whether a system under test correctly transcribes easy test speeches, while being robust to the most difficult ones. In this paper, a novel framework is proposed for evaluating ASR systems, which covers two complementary issues: (1) to measure the difficulty of each test speech; and (2) to analyse each ASR system’s performance against the difficulty level. Regarding the first issue, the framework measures speech difficulty by adopting Item Response Theory (IRT). Regarding the second issue, the Recognizer Characteristic Curve (RCC) is proposed, which is a plot of the ASR system’s performance versus speech difficulty. ASR performance is further analysed by a two-dimensional plot, in which speech difficulty is decomposed by IRT into sentence difficulty and speaker quality. In the experiments, the proposed framework was applied in a test set produced by adopting text-to-speech tools, with diverse speakers and sentences. Additionally, noise injection was applied to produce test items with even higher difficulty levels. In the experiments, noise injection actually increases difficulty and generates a wide variety of speeches to assess ASR performance. However, it is essential to pay attention that high noise levels can lead to an unreliable evaluation. The proposed plots were helpful for both identifying robust ASR systems as well as for choosing the noise level that results in both diversity and reliability.

Benchmarking is a fundamental practice in machine learning (ML) for comparing the performance of classification algorithms. However, traditional evaluation methods often overlook a critical aspect: the joint consideration of dataset complexity and an algorithm's ability to generalize. Without this dual perspective, assessments may favor models that perform well on easy instances while failing to capture their true robustness. To address this limitation, this study introduces a novel evaluation methodology that combines Item Response Theory (IRT) with the Glicko-2 rating system, originally developed to measure player strength in competitive games. IRT assesses classifier ability based on performance over difficult instances, while Glicko-2 updates performance metrics-such as rating, deviation, and volatility-via simulated tournaments between classifiers. This combined approach provides a more robust and difficulty-aware measure of algorithm capability. A case study using the OpenML-CC18 benchmark showed that only 16.66% of the datasets are truly challenging and that a reduced subset with 50% of the original datasets offers comparable evaluation power. Among the algorithms tested, Random Forest achieved the highest ability score. The results highlight the importance of improving benchmark design by focusing on dataset quality and adopting evaluation strategies that reflect both difficulty and classifier proficiency.

Clustering evaluation is a complex task. External measures, such as the Rand index, are often used for benchmarking, but they are not applicable in real, unsupervised scenarios due to the lack of ground truth. Thus, we often turn to internal measures for model evaluation, e.g. silhouette, Dunn, and Davies-Bouldin. These indexes have the advantage of evaluating models based on the clustered data points themselves, however, they rely on a chosen distance, so they are only meaningful for models that use the same distance. Additionally, they fail if all instances are assigned to a single cluster and they aim to evaluate separation and cohesion instead of quantifying a model’s ability to recover any underlying classes. Thus, internal measures are not suited to compare models that are estimated differently. In this paper, we propose CLAIRE (CLuster Agreement-based Item REsponses), a method for global evaluation of clustering models, by assuming that good models agree on whether pairs of instances should be clustered together or not. We leverage Item Response Theory to estimate model ability and instance difficulty, using response matrices obtained by measuring the agreement between models. Experiments were carried out using diverse sets of clustering methods and datasets with different numbers of clusters and varying shapes and levels of overlapping and noise. Results show that CLAIRE is robust to the presence of random partitions in the pool of models and correctly ranks models across the many tested scenarios with a surprisingly high correlation with external measures of clustering quality, meaning it also indirectly evaluates the recovery of underlying classes.

Item response theory (IRT) was developed in psychometrics to measure the latent skills of human respondents based on their observed responses to items with different difficulty levels. Human ability is high in IRT when one correctly responds to difficult items despite random mistakes in easy items. IRT has been recently framed as a powerful tool to characterise instance hardness in classification problems by measuring difficulty and discrimination levels of instances in a data set based on the correctness of a set of classifiers. Here, we generalise such a concept to the data set level by taking a pool of 509 classification data sets and assessing their difficulties and discriminations based on the performance achieved by 95 classifiers when solving these problems. The ability is estimated such that high abilities are assigned to classifiers with better behaviour in hard data sets. We further evaluated IRT in two distinct applications. First, we build a regression meta-model where complexity measures are used to predict the IRT parameters of new data sets without the need to retrain the IRT model. Second, we propose two IRT-based benchmarks with 30 data sets each to test classifiers, one selected for diversity and another selected for greater difficulty. Both benchmarks may be used to evaluate new methods more broadly, instead of the common practice of gathering random data sets from public repositories.

Understanding the difficulty of individual instances in a classification problem is important to define the limits of learning performance in the problem. Previous works are devoted to measuring Instance Hardness (IH), while solutions for explaining IH are still not deeply investigated. In this paper, we rely on using assessor models and eXplanaible AI (XAI) techniques to predict and explain IH. Many XAI techniques have been developed in the literature to explain the predictions of Machine Learning (ML) models. In our work, we are focused on explaining the difficulty of instances. Given a classification dataset, we trained and evaluated a pool of diverse ML models to measure the IH of each instance. Then, we trained an assessor model to predict the IH based on the instances’ features. Once the assessor is built, its predictions (i.e., the expected IH) can be explained using XAI techniques. In our experiments, we produced Partial Dependence Plots (PDP) to inspect the marginal effect of specific features on the IH predicted by the assessor. From the PDPs, we could check how IH is distributed along the instances’ features in a problem, and more specifically, we could visualize areas of high expected predictive difficulty.

Anomaly detection (AD) techniques are adopted to identify instances with patterns that significantly differ from the general behavior of a dataset. The development of new techniques, such as those based on deep learning, and the higher data availability have increased the use of AD techniques in challenging tasks such as in the detection of failures in industrial equipment's. Generally, an AD technique generates an anomaly score for each instance, later used to classify it as anomalous or normal, based on a threshold above which the instance is considered anomalous. A problem that is commonly observed in practice is the presence of spurious peaks in the anomaly score signal and other irregularities that may cause, for example, a high number of false positives in AD. In this paper, we investigated the use of low-pass filters in order to smooth the anomaly scores derived by a Sparse Autoencoder (SAE) model adopted for AD. In our experiments, we investigated the usefulness of the low-pass filters considering two different approaches: (1) directly applied on the anomaly scores; and (2) applied on the classification signal returned by the AD model. The experiments were performed on a case study of AD in a metro's air production unit. Generally, the filter applied directly on the anomaly score maximized true positives. In turn, the filter applied after classification minimized false positives. It was observed that in general the use of LPF was essential to detect sequences of anomalies. Thus, how to apply low-pass filters in AD must be defined according to specific application goals.

Identifying which instances in a learning problem are difficult to be predicted by a model is important to avoid critical errors at deployment time as well as to plan how to learn an improved model (e.g., by training data cleaning or augmentation). Previous works have been mainly devoted on measuring instance hardness or developing meta-learners (e.g., assessors) to predict a base model performance based on the instances’ features while neglecting interpretability. In this paper, we propose a method to explain the performance of learned models in a problem based on the induction of meta-rules. Each meta-rule identifies a local region of instances, called Local Performance Region (LPR), where the base model has a predictable performance. The meta-rules are induced using a reduced number of attributes, in such a way that each LPR can be more easily inspected (e.g., by an attribute plot). The proposed method combines assessors, data augmentation and rule induction procedures. Initially, given a dataset of interest and a base model, we build an assessor model, which will be able to predict the base model’s performance for new instances. The assessor is trained based on the test results obtained when the base model is evaluated, thus generalizing the observed errors across instances in the dataset. In our work, we built an assessor in a case study to predict the probability of incorrect classifications of a Random Forest (RF) base model, achieving a mean absolute error of 0.05 in a hold out experiment. Once learned, the assessor is used to predict the model’s errors for new instances in an augmented dataset, covering a variety of features. Finally, meta-rules are learned to approximate the assessor’s predictions in local regions of instances. Experiments show the usefulness of the proposal by finding 18 local regions of bad RF performance, demonstrating a special case of LPRs, called Local Hard Regions (LHRs). By explaining the (in)correctness of model predictions, LPRs constitute a novel application in explainable AI, but focusing on explaining model performance, which can be adopted to different ML contexts.

Machine Learning solutions have spread along many domains, including critical applications. The development of such models usually relies on a dataset containing labeled data. This dataset is then split into training and test sets and the accuracy of the models in replicating the test labels is assessed. This process is often iterated in a cross-validation procedure for obtaining average performance estimates. But is the average of the predictive performance on test sets enough for assessing the trustfulness of a Machine Learning model? This paper discusses the importance of knowing which individual observations of a dataset are more challenging than others and how this characteristic can be measured and used in order to improve classification performance and trustfulness. A set of strategies for measuring the hardness level of the instances of a dataset is surveyed and a Python package containing their implementation is provided.

Traditional Machine Learning (ML) approaches often emphasize evaluating models using global metrics over a dataset, frequently overlooking the nuances of learning data. Analyzing how hard it is to classify each instance, also known as instance hardness, furnishes such information, offering insights into reasons behind particular misclassifications. This paper introduces an unsupervised Deep Boltzmann Machine model integrated with an interpretability module that provides various latent traits related to instance hardness and classifier predictive performance. Such knowledge can facilitate in-depth analyses of the learning dataset's instances and the predictive power or ability of ML algorithms. Herein, we illustrate our approach by assessing five datasets with over 230 learning algorithms.

Preventing a Machine Learning (ML) predictor from making an unreliable prediction is extremely important in sensitive application domains, such as health contexts. In this sense, strategies based on the reject option have been increasingly explored. However, few studies explore the ability of meta-learning to inspect the errors of a base predictor under analysis, in such a way to generalize when the predictor is confident or not. Therefore, the current paper proposes a novel solution for ML with reject option based on the combination of meta-learning and novelty detection. The proposal addresses two distinct situations where a prediction should be rejected. First, novel detection is adopted to identify out-of-distribution instances, i.e., instances that significantly differ from those ones adopted to train the base predictor. Second, meta-learning is adopted to detect instances in regions of data where the base model has shown poor predictive performance during its evaluation. Such instances mainly lie in areas of class overlap or noisy regions in the training data. The results in experiments on synthetic and real data showed the superiority of the solution compared to those based only on meta-learning (aka without novelty detection) and those based on classifier confidence.

Software testing is crucial to ensure the quality of a software under development. Once a potential bug is identified, a Bug Report (BR) is opened with information to describe and reproduce the found issue. Usually in big companies, hundreds of BRs are opened weekly by different testing teams, which have to be inspected and fixed adequately. This paper is focused on the use of Machine Learning (ML) techniques to automate the Escaped Defect Analysis (EDA), which is an important (but expensive) task to improve the effectiveness of the testing teams. In our work, Escaped Defects (EDs) are bugs or issues that should have been opened by a specific team, but which was accidentally found by another team. The occurrence of EDs is risky, as it is usually related to failures in the testing activities. EDA is usually performed manually by software engineers, who read each BR’s textual content to judge whether it is an ED or not. This is challenging and time-consuming. In our solution, the BR’s content is preprocessed by textual operations and then a feature representation is adopted by a ML classifier to return the probability of EDA labels. Experiments were performed in a dataset of 3767 BRs provided by the Motorola Mobility Comércio de Produtos Eletrônicos Ltda. Different ML algorithms were adopted to build classifiers, obtaining high AUC values (usually higher than 0.8), in a cross-validation experiment. This result indicates a good trade-off between the number of EDs correctly identified and the number of BRs that have to be actually inspected in the EDA process. This paper presents a ML based approach to classify escaped defects described in bug reports. EDs are bugs missed by the QA team in charge and happened to be uncovered by a different team. To automate the identification of EDs (a costly and error-prone task), a dataset of a partner company is leveraged, text processing operators are adopted for feature engineering and 6 classical ML algorithms are applied. The results show satisfactory accuracy and AUC and the experiments indicate a good trade-off between the number of EDs correctly identified and the number of BRs that have to be inspected in the EDA.

The present article seeks to present a viable methodology for the recognition of static and dynamic expressions of the Brazilian sign language, LIBRAS, in environments whose background is complex. For this, the Keras library, which is part of TensorFlow, was used as a training tool and the OpenCV library, together with MediaPipe for the recognition of incoming frames. Concluding, therefore, with the use of perceptron neural networks for machine learning.

Item response theory aims to estimate respondent's latent skills from their responses in tests composed of items with different levels of difficulty. Several models of item response theory have been proposed for different types of tasks, such as binary or probabilistic responses, response time, multiple responses, among others. In this paper, we propose a new version of β^3-IRT, called β^4-IRT, which uses the gradient descent method to estimate the model parameters. In β^3-IRT, abilities and difficulties are bounded, thus we employ link functions in order to turn β^4-IRT into an unconstrained gradient descent process. The original β^3-IRT had a symmetry problem, meaning that, if an item was initialised with a discrimination value with the wrong sign, e.g. negative when the actual discrimination should be positive, the fitting process could be unable to recover the correct discrimination and difficulty values for the item. In order to tackle this limitation, we modelled the discrimination parameter as the product of two new parameters, one corresponding to the sign and the second associated to the magnitude. We also proposed sensible priors for all parameters. We performed experiments to compare β^4-IRT and β^3-IRT regarding parameter recovery and our new version outperformed the original β^3-IRT. Finally, we made β^4-IRT publicly available as a Python package, along with the implementation of β^3-IRT used in our experiments.

Assets deteriorate over time, as well as being covered, corroded, or becoming old in less obvious ways. Maintenance can extend the remaining useful life (RUL) of an asset system, but sooner or later it must surely be replaced. In this study, we propose a new RUL estimation methodology to assist in decision making for the maintenance and replacement of assets from prioritizing equipment in a renovation plan. Our methodology uses advanced data analysis techniques that consider multiple competing criteria with the goal of maximizing values of the asset throughout its life cycle, while considering the rules of remuneration and service quality of the current regulation, as well as the values at risk according to the decisions and actions taken. Experimental results with real datasets show the efficiency of the proposed approach. Finally, this work also presents the development of an analytical tool to optimize asset renewal decisions applying the RUL estimation methodology proposed and its application to the Brazilian electric sector.

Artificial Intelligence (AI) systems have been increasingly developed and improved. In this sense, one of the main challenges is to evaluate and compare them. However, traditional assessment methods do consider some hidden factors that may influence the quality of these systems that can be helpful in their discrimination (e.g., between poor and good techniques). Previously, we developed a work that uses Item Response Theory (IRT) to simultaneously evaluate speech synthesis and recognition. IRT is a paradigm from psychometrics to estimate the cognitive ability of human respondents based on their responses to items with different levels of difficulty. One of the measures we estimated in that previous work was the synthesized speeches’ difficulties, in turn, the factors that influence that measure were not deeply explored. So, in this paper, we navigate far on this topic and investigate what explains a synthesized speech difficulty. We found out that some of the factors that may influence are: the sentence, the locale and the service used to generate the speech. Also, we performed a preliminary study to investigate the viability of predicting the synthesized difficulty using machine learning models. So, we trained some regression models using the speech synthesis parameters as features and the difficulty as the label. The best result was achieved using a Random Forest, in which we got 0.31 as normalized R2 score.

Identifying instances in a learning task that are difficult to predict is important to avoid critical errors at deployment time. Additionally, providing explanations for good or bad predictions of a model can be useful to understand its behavior and to plan how to improve it (e.g., by data augmentation in specific areas of instances). In this paper, we propose a method to provide explanations for a model's predictive performance based on the induction of meta-rules. Each meta-rule identifies a local region in the instance space, called Local Performance Region (LPR). The meta-rules are induced using a reduced number of attributes, in such a way that each LPR can be inspected by, e.g., plotting a pairwise attribute plot. Additionally, given a group of instances to explain (or eventually an individual instance), we propose a greedy-search algorithm that finds the subset of non-redundant LPRs that maximally covers the instances. By explaining the (in)correctness of model predictions, LPRs constitute a novel use of meta-learning and a novel application in explainable AI. Experiments show the usefulness of LPRs while explaining inaccurate class predictions of Random Forest in a benchmark dataset, demonstrating a special case of LPRs, called Local Hard Regions (LHRs).

Label noise detection has been widely studied in Machine Learning because of its importance in improving training data quality. Satisfactory noise detection has been achieved by adopting ensembles of classifiers. In this approach, an instance is assigned as mislabeled if a high proportion of members in the pool misclassifies it. Previous authors have empirically evaluated this approach; nevertheless, they mostly assumed that label noise is generated completely at random in a dataset. This is a strong assumption since other types of label noise are feasible in practice and can influence noise detection results. This work investigates the performance of ensemble noise detection under two different noise models: the Noisy at Random (NAR), in which the probability of label noise depends on the instance class, in comparison to the Noisy Completely at Random model, in which the probability of label noise is entirely independent. In this setting, we investigate the effect of class distribution on noise detection performance since it changes the total noise level observed in a dataset under the NAR assumption. Further, an evaluation of the ensemble vote threshold is conducted to contrast with the most common approaches in the literature. In many performed experiments, choosing a noise generation model over another can lead to different results when considering aspects such as class imbalance and noise level ratio among different classes.

Com o início da vacinação contra a COVID-19, esta se tornou um assunto bastante debatido nas redes sociais. Porém, a vacinação infantil só veio a ter início cerca de um ano após a vacinação de adultos, o que acabou adiando esse debate específico para o fim de 2021. Este trabalho, portanto, visa analisar o posicionamento, se favorável ou contrário, dos usuários do Twitter no Brasil acerca da aplicação das vacinas nas crianças de 5 a 11 anos. Utilizando técnicas de Análise de Sentimentos, pôde-se perceber que a maior parte dos usuários se mostrou favorável ao início da vacinação no país. Também foram levantados pontos que levaram os usuários a se posicionar daquela forma.