Recent advances in passage retrieval have seen the introduction of pre-trained language models (PLMs) based neural rankers. While generally very effective, little attention has been paid to the robustness of these rankers. In this paper, we study the effectiveness of state-of-the-art PLM rankers in presence of typos in queries, as an indication of the rankers’ robustness. As of PLM rankers, we consider the two most promising directions explored in previous work: dense retrievers vs. sparse retrievers. We find that both types of rankers are very sensitive to queries with typos. We then apply an existing augmentation-based typos-aware training technique with the aim of creating typo-robust dense and sparse retrievers. We find that this simple technique only works for dense retrievers, while it hurts effectiveness when used on sparse retrievers.
Online social media platforms have become an important forum for public discourse, and have often been implicated in exacerbating polarisation in public sphere. Yet the precise mechanisms by which polarisation is driven are not fully understood. The study of linguistic style and features has been shown to be useful in exploring various aspects of online group discussions and, in turn, the processes which could contribute to polarisation. We present a case study around the hashtag #ArsonEmergency, collected from Australian Twittersphere during the unprecedented bushfires of 2019/2020. The dataset consists of two polarised groups and one unaffiliated group. We examine the linguistic style, moral language, and happiness profiles of 1786 users active during this catastrophic event. Our results suggest that polarised groups pushed ‘affective polarisation’ on Twitter while discussing the Australian Bushfires.
Transformer-based Dense Retrievers (DRs) are attracting extensive attention because of their effectiveness paired with high efficiency. In this context, few Pseudo-Relevance Feedback (PRF) methods applied to DRs have emerged. However, the absence of a general framework for performing PRF with DRs has made the empirical evaluation, comparison and reproduction of these methods challenging and time-consuming, especially across different DR models developed by different teams of researchers. To tackle this and speed up research into PRF methods for DRs, we showcase a new PRF framework that we implemented as a feature in Pyserini – an easy-to-use Python Information Retrieval toolkit. In particular, we leverage Pyserini’s DR framework and expand it with a PRF framework that abstracts the PRF process away from the specific DR model used. This new functionality in Pyserini allows to easily experiment with PRF methods across different DR models and datasets. Our framework comes with a number of recently proposed PRF methods built into it. Experiments within our framework show that this new PRF feature improves the effectiveness of the DR models currently available in Pyserini.
The array is a fundamental data object in most programs. Its key functionality – storage of and access to a set of same-type elements in O(1) time per operation – is also widely employed in other more sophisticated data structures. In an extensible array the number of elements in the set is unknown at the time the program is initiated, and the array might continue to grow right through the program’s execution. In this paper we explore the use of extensible arrays in connection with the task of inverted index construction. We develop and test a space-efficient extensible array arrangement that has been previously described but not to our knowledge employed in practice, and show that it adds considerable flexibility to the index construction process while incurring only modest run-time overheads as a result of access indirections.
Legal information retrieval (IR) is a form of professional search often associated with high recall. Information seeking in this context can consist of a single query with no clicks (known as updating behaviour), a literature review where a complex boolean query crafted over several iterations is performed and all documents returned are inspected, or a seeking task spanning days or weeks, consisting of multiple queries interleaved with other tasks. Analysis of query logs is paramount to the improvement of current legal IR systems, and in particular of the system we are associated with, the Dutch Legal Intelligence IR system. This analysis however requires the ability to automatically identify which queries of a user are related to the same search goal — or in other words, related to the same search task. The current practice of defining sessions — a set of user interactions with the IR system with no more than 30 minutes between user actions — and equating a session to representing a search task, might prove ineffective given the characteristics of this user group. In this paper we provide an initial analysis of a sub-set of the query log from the Dutch Legal Intelligence IR system, comprising of 970 queries issued by 10 users within the space of 1 year. From this query log, we used the 30-minutes heuristic to define sessions, and extract 126 sessions, ranging from 1 to 71 sessions per user. We then independently annotate the query log to manually identify search tasks: this activity leads to the identification of 55 tasks, ranging from 1 to 21 tasks per user. In doing this, we highlight how the currently employed heuristic is not adequate to extract search queries from a user that are related to the same search task. We also show why tasks are more informative than sessions with regards to legal information retrieval. We further describe the potential of using characteristics such as Levenshtein distance, common words and string matching for automated task classification.
Medical systematic reviews typically require assessing all the documents retrieved by a search. The reason is two-fold: the task aims for “total recall”; and documents retrieved using Boolean search are an unordered set, and thus it is unclear how an assessor could examine only a subset. Screening prioritisation is the process of ranking the (unordered) set of retrieved documents, allowing assessors to begin the downstream processes of the systematic review creation earlier, leading to earlier completion of the review, or even avoiding screening documents ranked least relevant. Screening prioritisation requires highly effective ranking methods. Pre-trained language models are state-of-the-art on many IR tasks but have yet to be applied to systematic review screening prioritisation. In this paper, we apply several pre-trained language models to the systematic review document ranking task, both directly and fine-tuned. An empirical analysis compares how effective neural methods compare to traditional methods for this task. We also investigate different types of document representations for neural methods and their impact on ranking performance. Our results show that BERT-based rankers outperform the current state-of-the-art screening prioritisation methods. However, BERT rankers and existing methods can actually be complementary, and thus, further improvements may be achieved if used in conjunction.
Approaches involving the use of post-retrieval information for a given query have been adopted in a variety of ways in the past for query performance prediction (QPP) tasks. Researchers have utilized information via document retrieval as well as passage retrieval approaches for QPP. We present a novel approach of representing the top returned passages (answer-set) as a graph where each node represents a passage and an edge weight indicates the similarity score between these passages. By examining the answer-set graph we developed new predictors that utilizes graph features such as cohesion and minimum spanning tree. Based on the empirical evaluation, we show that our answer-set graph predictors are very effective and perform even better (for Cranfield and Ohsumed Collection) than the current state-of-the-art QPP approaches.
Disinformation is deliberately designed to spread false information over internet. Recent concerns about the use of disinformation to manipulate political voting campaigns have attracted researchers’ attention. In this paper, we conduct our first study towards the understanding of how sponsored social media content is used in Australian voting campaigns. To this end, we collect the ad posts sponsored by Australian organizations on Facebook from 1 Feb, 2020 to 17 May, 2021. We also retain the screenshot of each collected ad that originally appeared on Facebook and download the images and videos that were presented in these ad posts. To obtain annotations of these ads, we generate labels that describe general objects, locations, activities presented in the images or videos by algorithms, as well as human created annotations over crowdsourcing platforms. Based on the collected human annotations, we then design a second-round crowdsourcing task to ask workers to provide more detailed annotations for the collected ads, ranging from truthfulness evaluation of the content to various political aspects (e.g., topics and sentiment). The multi-modal dataset created in our work enables future research, for example, to train supervised learning algorithms for further analysis on the use of disinformation on social media that may affect political campaigns.
Several approaches have been taken by different scientists over the years to create taxonomy of coral species by looking at their morphology. On molecular examination, the taxonomies created have revealed to have incorrect classifications. In this project we aim to find a relationship between different types of corals and classify them by using image classification and clustering techniques on a coral dataset provided by Queensland Museum (QM), Australia. We use the VGG16 [9], InceptionV3 [10] and DenseNet [5] models which are pretrained on the ImageNet dataset, to train and extract feature embeddings from the coral images in the QM dataset. These embeddings are then clustered using the Agglomerative Hierarchical Clustering to obtain a general hierarchy of corals. We show that DenseNet performs the best among the three models on the image classification task and can be used to extract the feature embeddings. Using Agglomerative Hierarchical Clustering with average link criterion on these embeddings, we can generate a general hierarchy of corals.
High-quality medical systematic reviews require comprehensive literature searches to ensure the recommendations and outcomes are sufficiently reliable. Indeed, searching for relevant medical literature is a key phase in constructing systematic reviews and often involves domain (medical researchers) and search (information specialists) experts in developing the search queries. Queries in this context are highly complex, based on Boolean logic, include free-text terms and index terms from standardised terminologies (e.g., MeSH), and are difficult and time-consuming to build. The use of MeSH terms, in particular, has been shown to improve the quality of the search results. However, identifying the correct MeSH terms to include in a query is difficult: information experts are often unfamiliar with the MeSH database and unsure about the appropriateness of MeSH terms for a query. Naturally, the full value of the MeSH terminology is often not fully exploited. This paper investigates methods to suggest MeSH terms based on an initial Boolean query that includes only free-text terms. These methods promise to automatically identify highly effective MeSH terms for inclusion in a systematic review query. Our study contributes an empirical evaluation of several MeSH term suggestion methods. We perform an extensive analysis of the retrieval, ranking, and refinement of MeSH term suggestions for each method and how these suggestions impact the effectiveness of Boolean queries.
Document collections can be characterised in a variety of ways. Two key approaches are clustering, which partitions collections into subcollections with the expectation that the contents will be thematically linked, and topic models, which describe the contents in terms of weighted lists of words that are expected to represent different themes. In this paper, we report experiments on the observed relationship between clusters and topic models in a preliminary study of a large text collection. Both produce results that appear cohesive in their own right, but surprisingly – given the very different ways in which they are formed – the descriptions of the collections that they generate are strongly similar. This unexpected mutual reinforcement creates confidence in both approaches as tools for annotating and describing the contents of document collections.
Personalized recommender systems have been one of the major ways of information acquisition. Understanding the user's intent and behavior plays an important role and has been a trending topic in research and applications. In this talk, I will briefly introduce some observations and findings on our user understanding research in recent years, which are surprising or different from what people thought before. The topic in the talk involves user behavior analyses, dynamic intent modeling, and evaluation in multiple scenarios such as news streaming, e-commerce, job hunting, and music recommendation. Related researches have been published on ACM TOIS, WWW, SIGIR, WSDM, etc.
SEEK is a leading employment marketplace with a global presence, connecting hundreds of millions of job seekers with job advertisements posted by millions of hirers across APAC and the Americas. Document processing is fundamental to all areas of our business, and multiple teams at SEEK face daily challenges about how to maximise the potential of our multinational, multilingual, multifaceted corpus of job ads, candidate resumes, and other employment documents. Although some of these challenges are familiar technical problems like accuracy, performance, and reliability of services, often the most difficult challenges we face are non-technical. How should large teams organise into autonomous units while avoiding silos of data and expertise? How can we use data and build AI services in an ethical way that is true to our purpose of helping SEEK's users and protecting their personal data? In this talk I will discuss several of these challenges that we currently face at SEEK, and the solutions we have found to deal with them.
Clinical Trials are a critical step for medical advancement; key to success is recruiting eligible patients to a trial. Retrieval methods are used to identify relevant trials given a single patient/query. After careful consideration of the clinical setting, this paper takes a different approach: cohort-based trial retrieval. We consider ranking trials that maximise recruitment opportunities across the whole patient cohort, instead of a single patient. This resolves into optimising a ranking for the whole query set formed by the patient cohort, rather than treating each query independently — and thus considering an evaluation measure based on cohort coverage. We study the adaptation of rank fusion methods and diversity reranking to this problem. Empirically, we show the surprising impact of initial ranking effectiveness (underlying initial retrieval) on cohort coverage when adapting rank fusion methods. We further highlight that devising cohort-aware methods would have a far greater impact on patient recruitment.
Index reordering techniques allow document collections to be renumbered, with the goal of developing a permutation of the initial document ordinal identifiers that places documents that are (somehow) like each other into positions near each other in the permuted ordering. The clustering that results allows inverted index size to be reduced, since each term’s posting list is more likely to contain a non-uniform set of inter-document integer gaps. Reordering is normally performed once, at the time the index is created. Here we consider the role of index reordering in collections that grow over time, noting that simply appending new documents to the collection may erode the effectiveness of an earlier reordering. In particular, we discuss methods for maintaining and reinstating reorderings as document collections grow, and measure the effectiveness of those techniques on a large corpus of English news articles. We also provide experimental results that illustrate the benefits of reordering in terms of query execution time.
Backstories provide vital contextual information for information retrieval evaluation. They are useful as textual representations of information needs, for example to aid in relevance judgements as part of test collections for performance evaluation, for studying longer search queries, and for interactive retrieval. While backstories exist for some popular search tasks and domains thanks to evaluation campaigns such as TREC, NTCIR, and CLEF, they are not available for a large range of other tasks and domains. In this paper, we explore crowdsourcing as an approach for obtaining high-quality backstories, with the aim of supporting the development of backstories as key resources for new domains and search tasks. Compared to typical crowdsourcing tasks in the IR domain, such as gathering relevance judgements or short textual search queries, obtaining backstories is more complex. Workers are required to think of information need scenarios and put these thoughts into comparatively lengthy text fragments. This possibly entails a higher cognitive load and longer working time. We describe a crowdsourcing methodology to maximise the usefulness of results, using the creation of backstories for the job search domain as an example. We also present and release a collection of 756 job search backstories, which was obtained via the proposed methodology.
Annotating handwritten documents for training deep learning models is a major issue in handwritten text recognition. It requires manual effort to annotate each word in a document to specify the ground truth. Often documents contain struck-out text which needs to be ignored by the recognition process. In preparing training data, struck-out text needs to be represented in a way that can help deep learning models to learn to deal appropriately with the strike-outs. The question is how to do this. In this paper, we have investigated two approaches for struck-out text annotation: (1) provide no annotation, thus reducing the annotation burden, and (2) mark the struck-out text with a special symbol, we have used the symbol #. We have trained two models on a synthetically generated dataset using a convolutional neural network and LSTM. We obtained 8.8% and 9.0% character error rates for models one and two respectively. There was no statistically significant difference in the performance of the two models. This indicates that a model trained with minimal annotations can perform as well as a model trained with extra annotations for struck-out text.
Computing similarity between a query and a document is fundamental in any information retrieval system. In search engines, computing query-document similarity is an essential step in both retrieval and ranking stages. In eBay search, document is an item and the query-item similarity can be computed by comparing different facets of the query-item pair. Query text can be compared with the text of the item title. Likewise, a category constraint applied on the query can be compared with the listing category of the item. However, images are one signal that are usually present in the items but are not present in the query. Images are one of the most intuitive signals used by users to determine the relevance of the item given a query. Including this signal in estimating similarity between the query-item pair is likely to improve the relevance of the search engine. We propose a novel way of deriving image information for queries. We attempt to learn image information for queries from item images instead of generating explicit image features or an image for queries. We use canonical correlation analysis (CCA) to learn a new subspace where projecting the original data will give us a new query and item representation. We hypothesize that this new query representation will also have image information about the query. We estimate the query-item similarity using a vector space model and report the performance of the proposed method on eBay's search data. We show 11.89% relevance improvement over the baseline using Area Under the Receiver Operating Characteristic curve (AUROC) as the evaluation metric. We also show 3.1% relevance improvement over the baseline with Area Under the Precision Recall Curve (AUPRC).
In common law legal systems, judges decide issues between parties (legal decision or case law) by reference to previous decisions that consider similar factual situations. Accordingly, these decisions typically feature rich citation networks, i.e., a new decision frequently cites previous relevant decisions (citation). These citations may, in varying degrees, express that a cited decision is applicable, not-applicable, or no longer current law. Such treatment label is important to a lawyer's process of determining whether a case is proper law. These labels serve as a matter of convenience in citation indices enabling lawyers to prioritise decisions to examine to understand the current state of the law. They also prove useful in other areas such as prioritisation for manual summarisation of cases, where not all cases can be summarised, and automatic summarisation, or, potentially, as a ranking feature in case law retrieval. While a lawyer can determine the treatment of a cited case by reading a decision, this is time consuming and can increase legal costs. Currently, not all newly decided cases feature these treatment labels. Further, older cases typically do not. Given the large amount of new legal decisions decided each year, manual annotation of such treatment is not feasible. In this paper, we explore the effectiveness of neural network architectures for identifying case law citation treatment and importance (whether a case is important to a lawyer's reasoning process). We find that these tasks are very difficult and various methods for text classification perform poorly. We address more comprehensively the task of citation importance for this reason while limiting our examination of the task of citation treatment to the modelling of the problem and the highlight of the intrinsic difficulty of the task. We make a test dataset available at github.com/ielab/caselaw-citations to stimulate further research that tackles this challenging problem. We also contribute a range of word embeddings learned over a large amount of processed case law text.
