Once again, the author goes back in time, to a number of different fields of human activity which required systematic organisation of information: astronomy, calculating the date of Easter, tax collecting, census taking, history itself, libraries again. He then goes on to discuss the filling in of forms, as a way to introduce the idea of a database.

The idea of data processing machines arose quite independently of ideas about calculation, in the analysis and tabulation of the US census data. An entire industry developed from this in the first half of the twentieth century, which then morphed into the computer industry. This leads to a further discussion on the notion of data.

The theory and practice of search results ranking, as currently offered by most web search engines, is older than one might think. The first proposal for a system of ranking was in a JACM paper in 1960. Through the remainder of the twentieth century, extensive research work was done on ranking systems-on devising methods of ranking, on the use of learning methods in association with ranking, and on the comparative evaluation of alternative methods. The web search engines of the tail end of the century made much use of this paper.

To me, an awareness of history is a fundamental requirement for progress; and I believe that we in the field of information retrieval are currently ill-served in this domain, or at least not as aware as we should be. While it is true that a researcher in IR is expected to acquire some knowledge of what has gone before, this knowledge is typically fairly narrow in scope. there are some IR researchers who regard IR as a branch of computer science, for example, despite the fact that the field has a long and venerable history entirely outside the domain of computers, as well as a considerable current presence in academic departments well away from CSDs. Outside of our immediate community, there is a widespread belief that web search engines arose out of nothing, a totally new invention for the totally new world of the web. This is unfortunate. It is true that there were huge changes in the information retrieval world in the second half of the twentiethcentury. The ideas, methods and systems that were around in 2000 (particularly the early web search engines, up to and including Google) seem at first glance to be completely different from the ideas, methods and systems that were around in 1950. But these developments involved in large measure the usual process of evolution rather than revolution, and the usual mix of steps large or small, forwards or backwards or sideways. Some of these steps were taken in the world of IR research, and some in the domain of practical commercial systems.

A solid research path towards new information retrieval models is to further develop the theory behind existing models. A profound understanding of these models is therefore essential. In this paper, we revisit probability ranking principle (PRP)-based models, probability of relevance (PR) models, and language models, finding conceptual differences in their definition and interrelationships. The probabilistic model of the PRP has not been explicitly defined previously, but doing so leads to the formulation of two actual principles with different objectives. First, the belief probability ranking principle (BPRP), which considers uncertain relevance between known documents and the current query, and second, the popularity probability ranking principle (PPRP), which considers the probability of relevance of documents among multiple queries with the same features. Our analysis shows how some of the discussed PR models implement the BPRP or the PPRP while others do not. However, for some models the parameter estimation is challenging. Finally, language models are often presented as related to PR models. However, we find that language models differ from PR models in every aspect of a probabilistic model and the effectiveness of language models cannot be explained by the PRP.

The possibility of using fewer topics in TREC, and in TREC-like initiatives, has been studied recently, with encouraging results: even when decreasing consistently the number of topics (for example, using a topic subset of cardinality only 10, in place of the usual 50) it is possible, at least potentially, to obtain similar results when evaluating system effectiveness. However, the generality of this approach has been questioned, since the topic subset selected on one system population does not seem adequate to evaluate other systems. In this paper we reconsider that generality issue: we emphasize some limitations in the previous work and we show some experimental results that are instead more positive. The obtained results support the hypothesis that, by taking special care, the few topics selected on the basis of a given system population are also adequate to evaluate a different system population as well.

Stemming is a widely used technique in information retrieval systems to address the vocabulary mismatch problem arising out of morphological phenomena. The major shortcoming of the commonly used stemmers is that they accept the morphological variants of the query words without considering their thematic coherence with the given query, which leads to poor performance. Moreover, for many queries, such approaches also produce retrieval performance that is poorer than no stemming, thereby degrading the robustness. The main goal of this article is to present corpus-based fully automatic stemming algorithms which address these issues. A set of experiments on six TREC collections and three other non-English collections containing news and web documents shows that the proposed query-based stemming algorithms consistently and significantly outperform four state of the art strong stemmers of completely varying principles. Our experiments also confirm that the robustness of the proposed query-based stemming algorithms are remarkably better than the existing strong baselines.

Score-distribution models are used for various practical purposes in search, for example for results merging and threshold setting. In this paper, the basic ideas of the score-distributional approach to viewing and analysing the effectiveness of search systems are re-examined. All recent score-distribution modelling work depends on the availability of actual scores generated by systems, and makes assumptions about these scores. Such work is therefore not applicable to systems which do not generate or reveal such scores, or whose scoring/ranking approach violates the assumptions. We demonstrate that it is possible to apply at least some score-distributional ideas without access to real scores, knowing only the rankings produced (together with a single effectiveness metric based on relevance judgements). This new basic insight is illustrated by means of simulation experiments, on a range of TREC runs, some of whose reported scores are clearly unsuitable for existing methods.

Increasingly, web recommender systems face scenarios where they need to serve suggestions to groups of users; for example, when families share e-commerce or movie rental web accounts. Research to date in this domain has proposed two approaches: computing recommendations for the group by merging any members' ratings into a single profile, or computing ranked recommendations for each individual that are then merged via a range of heuristics. In doing so, none of the past approaches reason on the preferences that arise in individuals when they are members of a group. In this work, we present a probabilistic framework, based on the notion of information matching, for group recommendation. This model defines group relevance as a combination of the item's relevance to each user as an individual and as a member of the group; it can then seamlessly incorporate any group recommendation strategy in order to rank items for a set of individuals. We evaluate the model's efficacy at generating recommendations for both single individuals and groups using the MovieLens and MoviePilot data sets. In both cases, we compare our results with baselines and state-of-the-art collaborative filtering algorithms, and show that the model outperforms all others over a variety of ranking metrics.

We explore the notion, put forward by Cormack & Lynam and Robertson, that we should consider a document collection used for Cranfield-style experiments as a sample from some larger population of documents. In this view, any per-topic metric (such as average precision) should be regarded as an estimate of that metric's true value for that topic in the full population, and therefore as carrying its own per-topic variance or estimate precision or noise. As in the two mentioned papers, we explore this notion by simulating other samples from the same large population. We investigate different ways of performing this simulation. One use of this analysis is to refine the notion of statistical significance of a difference between two systems (in most such analyses, each per-topic measurement is treated as equally precise). We propose a mixed-effects model method to measure significance, and compare it experimentally with the traditional t-test.

Lab-based evaluations typically assess the quality of a retrieval system with respect to its ability to retrieve documents that are relevant to the information need of an end user. In a real-time search task however users not only wish to retrieve the most relevant items but the most recent as well. The current evaluation framework is not adequate to assess the ability of a system to retrieve both recent and relevant items, and the one proposed in the recent TREC Microblog Track has certain flaws that quickly became apparent to the organizers. In this poster, we redefine the experiment for a real-time ad-hoc search task, by setting new submission requirements for the submitted systems/runs, proposing metrics to be used in evaluating the submissions, and suggesting a pooling strategy to be used to gather relevance judgments towards the computation of the described metrics. The proposed task can indeed assess the quality of a retrieval system with regard to retrieving both relevant and timely information.

In this work, we propose a theory for information matching. It is motivated by the observation that retrieval is about the relevance matching between two sets of properties (features), namely, the information need representation and information item representation. However, many probabilistic retrieval models rely on fixing one representation and optimizing the other (e.g. fixing the single information need and tuning the document) but not both. Therefore, it is difficult to use the available related information on both the document and the query at the same time in calculating the probability of relevance. In this paper, we address the problem by hypothesizing the relevance as a logical relationship between the two sets of properties; the relationship is defined on two separate mappings between these properties. By using the hypothesis we develop a unified probabilistic relevance model which is capable of using all the available information. We validate the proposed theory by formulating and developing probabilistic relevance ranking functions for both ad-hoc text retrieval and collaborative filtering. Our derivation in text retrieval illustrates the use of the theory in the situation where no relevance information is available. In collaborative filtering, we show that the resulting recommender model unifies the user and item information into a relevance ranking function without applying any dimensionality reduction techniques or computing explicit similarity between two different users (or items), in contrast to the state-of-the-art recommender models.

On the basis of a theoretical analysis of issues around populations and sampling, for both topics and documents, and parameters with which we hope to characterise the effectiveness of different systems, we propose a modification to the traditional average precision metric. This modification involves both transformation and (in the estimation of the parameter) smoothing. The modified version is shown to have certain distributional advantages, on a substantial dataset. In particular, the distribution of values of the modified metric, over topics for a given system/run, is approximately normal.

We propose an approach to the retrieval of entities that have a specific relationship with the entity given in a query. Our research goal is to investigate whether related entity finding problem can be addressed by combining a measure of relatedness of candidate answer entities to the query, and likelihood that the candidate answer entity belongs to the target entity category specified in the query. An initial list of candidate entities, extracted from top ranked documents retrieved for the query, is refined using a number of statistical and linguistic methods. The proposed method extracts the category of the target entity from the query, identifies instances of this category as seed entities, and computes similarity between candidate and seed entities. The evaluation was conducted on the Related Entity Finding task of the Entity Track of TREC 2010, as well as the QA list questions from TREC 2005 and 2006. Evaluation results demonstrate that the proposed methods are effective in finding related entities.

This paper re-reviews Vickery’s book On retrieval system theory, first published 50 years ago, and discusses the changing nature of theoretical work on information retrieval and the possibility of developing a general theory of IR.