For each of the evaluation queries, we executed a query for each of the documents that required reranking. For each query-document pairing, we collected the BM25 similarity score for the “paperAbstract” and “title” fields. Each of the documents are ranked for each field based on the similarity score with the query text, with ties sharing their combined rank. This resulted in two ranked lists for each query.

We propose and evaluate a data mining system that uses a set of document features describing each document in the context of partially evaluated ranked results. We find our system to be competitive with existing metasearch ranking strategies for prioritizing the review of evidence for legal relevance.Nous proposons et évaluons un système de fouille de données basé sur une série de descripteurs de documents décrivant chaque document dans un contexte d’évaluation partielle des résultats classés. Nous concluons que notre système est concurrentiel par rapport aux stratégies existantes de classement des métarecherches pour la priorisation de l’examen des preuves en matière de pertinence juridique.

: This is the second year that our research group has participated in the TREC Legal Track. Our ad hoc retrieval system has been modified to extract the additional Boolean query fields added to the 2008 topics, and to privilege documents found by the Boolean reference run when conducting our queries. We have also submitted runs that fuse the results from existing runs. For the relevance feedback task, our system uses ranking information of relevant and non-relevant documents from previously submitted runs to the TREC Legal Track to train a classifier. The classifier is applied to the remaining unjudged documents to create a new ranked list. This approach is applied to sets of input runs, including a hybrid run where a classifier trained on one set of runs is applied to the unjudged documents from another set of runs. Our index remains unchanged from our submission to TREC Legal in 2007, and new changes to our retrieval system are accomplished through the selection of topic fields for input and post-retrieval processing.

We indexed the entire TREC collection, opting to merge the document content and the title into a single field, using the Lucene StandardAnalyzer, which strips punctuation, but recognizes and retains elements such as e-mail addresses. The StandardAnalyzer stoplist was used for indexing. For our explorations, we converted topic fields into term vectors for querying the collection. For each topic, our system returned a ranked set of results with enough documents to match in quantity either those retrieved by a reference Boolean query executed on behalf of the TREC 2006 evaluators, or enough to reach a set cap on the number of documents returned, whichever was greater.

Approved: ____________________________________ Thesis Supervisor ____________________________________ Title and Department ____________________________________ Date