Evaluating Visual Analytics for Relevant Information Retrieval in Document Collections

Retrieving information from document collections is necessary in many contexts, e.g. researchers search for papers on a topic, physicians search for records of patients with a certain condition and police investigators seek relationships between different criminal reports. Finding relevant textual content in a corpus can be challenging in scenarios where the users expect a retrieval process with high recall. Visual Analytics (VA) systems that integrate interactive visualizations and machine learning algorithms are often advocated to support retrieval tasks in such complex scenarios. However, few studies report an end-user perspective on the utility of such systems. We present results from observational studies on VA-supported information retrieval conducted with graduate students and researchers using a system to explore collections of scientific papers. While users have, in general, positive views of the system's potential to facilitate their retrieval tasks, some faced practical difficulties in using it effectively, and we found considerable variation in their assessment of specific functionalities. Our findings reinforce the potential of VA systems and also the importance of carefully informing users of the underlying conceptual models in such systems and their limitations.