A Validation Study Of A Visual Analytics Tool With End Users

In this paper we describe an user evaluation that aimed to understand how a group of endusers interpret a visual analytics tool in the context of service delivery. It is common for service factories to have an organization devoted to handle incidents. Many incident management systems have strict controls on how fast incidents should be handled, often subjected to penalties when targets are not met. We call Time-Bounded Incident Management (TBIM) those systems, which require clearly defined incident resolution times. In our project, research scientists proposed a method and a visual representation named Workload Profile Chart (WPC) that had as primary goal to understand the area of incident management in a service delivery department. The objective of this visual representation is to help characterizing the performance of TBIM systems and diagnosing major issues such as resource and skill allocation problems, abnormal behavior, and incident characteristics. Researchers wanted to understand if end-users, the quality analysts (QAs), would comprehend the charts and would be able to use them to identify problems and propose effective improvement actions related to TBIM activities. The study was conducted with ten QAs of a service delivery department of a IT company based in Brazil. The data was analyzed using descriptive statistical and qualitative methods. As a result, participants were mainly guided by the axes titles and chart legends to interpret the visualizations, and not always understood what kind of data the chart was displaying. Those results served as insights of how QAs think when analyzing TBIM information in a service delivery department and what improvements in the visual representation tool may be proposed to facilitate their activity. At last we identified evidences of how to design better visual analytics tools based on participant's perceptions and interpretations of color differences and verbal information in chart labels and legend.