Modeling search and session effectiveness

Many information needs cannot be resolved with a single query, and instead lead naturally to a sequence of queries, issued as a search session. In a session test collection, each topic has an associated query sequence, with users assumed to follow that sequence when reformulating their queries. Here we propose a session-based offline evaluation framework as an extension to the existing query-based C/W/L framework, and use that framework to devise an adaptive session-based effectiveness metric, as a way of measuring the overall usefulness of a search session. To realize that goal, data from two commercial search engines is employed to model the two required behaviors: the user conditional continuation probability, and the user conditional reformulation probability. We show that the session-extended C/W/L framework allows the development of new metrics with associated user models that give rise to greater correlation with observed user behavior during search sessions than do previous session metrics, and hence provide a richer context in which to compare retrieval systems at a session level.