Assessor Models for Explaining Instance Hardness in Classification Problems

Understanding the difficulty of individual instances in a classification problem is important to define the limits of learning performance in the problem. Previous works are devoted to measuring Instance Hardness (IH), while solutions for explaining IH are still not deeply investigated. In this paper, we rely on using assessor models and eXplanaible AI (XAI) techniques to predict and explain IH. Many XAI techniques have been developed in the literature to explain the predictions of Machine Learning (ML) models. In our work, we are focused on explaining the difficulty of instances. Given a classification dataset, we trained and evaluated a pool of diverse ML models to measure the IH of each instance. Then, we trained an assessor model to predict the IH based on the instances’ features. Once the assessor is built, its predictions (i.e., the expected IH) can be explained using XAI techniques. In our experiments, we produced Partial Dependence Plots (PDP) to inspect the marginal effect of specific features on the IH predicted by the assessor. From the PDPs, we could check how IH is distributed along the instances’ features in a problem, and more specifically, we could visualize areas of high expected predictive difficulty.