User Interactivity in Very Large Scale Data Mining

Knowledge discovery is widely considered to be an interactive and iterative process. The data mining phase of KDD is, on the other hand, often assumed to be an indivisible step. We argue that user interaction during discovery runs is a central element in very large scale data mining. We describe a system architecture centered around a common, persistently stored search space. This gives good performance while simultaneously allowing for high user interactivity. Furthermore, the architecture supports an extremely robust and modular implementation of the data mining process. As a concrete instantiation, we discuss the use of this architecture in the ESPRIT KESO project (Knowledge Extraction for Statistical Offices).