Effective information retrieval (IR) depends on accurate relevance classification. But when the criteria are subjective or underspecified, small variations in classification can cause consequential shifts in retrieval results. The potential for such variability becomes critical for institutions when they use IR for research assessment. Retrieval instability can lead to relevant literature being overlooked, hindering a comprehensive understanding of the research landscape, and potentially undermining the validity of subsequent analyses and decisions. We investigate this problem within the context of the United Nations Sustainable Development Goals (SDGs), a global framework for addressing environmental, social, and economic challenges. Scholarly research is vital for understanding, implementing, and monitoring SDG progress. Universities report SDG-related research to demonstrate impact, and international rankings incorporate SDG alignment into evaluations, influencing funding, policy, and institutional strategy. However, the nuanced nature of the SDGs makes it difficult to define what constitutes an SDG contribution [1]. Commonly used Boolean queries and controlled vocabularies for SDG retrieval cannot reliably differentiate substantive contributions (based on semantic relevance) from mere term occurrences. In prior work, Large Language Models (LLMs) have been used to filter Boolean search results in systematic reviews by scoring documents for relevance to a specific information need [2]. Other studies demonstrate that LLMs can generate high-quality relevance labels for IR evaluation [4]. This prompted an investigation into using LLMs to judge SDG contribution through relevance filtering, which revealed variability in the judgments made by different LLMs on the same set of documents [3]. This observation suggests that the classification behavior of LLMs are sensitive to the specific parameters inherent to each model. In this study, we prompt multiple LLMs to judge the SDG relevance of abstracts retrieved using Boolean queries. Abstracts judged relevant are used as positive training examples for fine-tuning multilabel SDG classifiers. We use these classifiers to simulate retrieval, applying fixed scoring functions to isolate fluctuations in ranking stability attributable to the different LLM relevance judgments. Our goal is to analyze how the structured signal of upstream inconsistencies in LLM-derived relevance judgments manifests as variations in retrieval outcomes, providing a novel lens for investigating ranking stability under classification uncertainty. This research centers on three key questions: center dot RQ1: How do different LLMs diverge in their filtering decisions, and what effect does this have on ranking stability in retrieval systems trained on filtered data? center dot RQ2: Can divergence in labeling decisions be systematically explained or predicted from document content? center dot RQ3: What distinguishes documents where LLMs disagree on relevance, and can these differences be predicted from lexical or surface-level features? Using SDG classification as a case study of subjective relevance, we evaluate retrieval stability under classification uncertainty and address broader concerns regarding the reproducibility of LLMbased classification pipelines and their downstream effects.
Traditionally, the users of an information retrieval (IR) system have been human users. We present a new perspective on IR research in which the users of an IR system are intelligent agents instead of human users. Extending the current work on retrieval-augmented generation (RAG), we identify five novel IR tasks that an intelligent agent must be able to perform in order to achieve Human-Level Artificial Intelligence, or Artificial General Intelligence (AGI), including 1) External Information Retrieval (EIR) to access new information unseen by the agent, 2) Provenance Information Retrieval (PIR) to trace the provenance of information, 3) Curriculum Information Retrieval (CIR) to actively acquire the most useful new data and information for lifelong learning, 4) Rule Information Retrieval (RIR) to perform reasoning and problem solving, and 5) Scenario Information Retrieval (SIR) to leverage past scenarios for problem solving and decision making. We compare these new IR tasks with the traditional IR tasks performed by an IR system that serves human users and systematically examine the challenges involved in the five new IR tasks, providing a roadmap for new IR research within the broader context of AGI development.
In theWeb ecosystem, document authors often strategically modify content to maximize visibility in search rankings. These rankingincentivized modifications pose challenges to retrieval effectiveness, content diversity, and long-term stability. At the same time, the rise of large language models (LLMs) is transforming IR: queries and documents that were once assumed to be generated exclusively by humans can now also be created by automated agents [5]. These agents can formulate queries, generate documents, and perform ranking. This introduces a variety of new challenges and opportunities in competitive search environments, where interactions between strategic agents can substantially influence outcomes. This research pursues two primary goals. First, it seeks to understand strategic interactions, particularly how different configurations of users (query), authors (document) and systems (ranker) influence competition outcomes. Second, it explores methods for designing systems that foster beneficial outcomes, such as stability and fairness. To this end, this research employs a combination of game-theoretic analysis, empirical ranking competitions between humans, and simulation-based experiments to investigate competitive dynamics in information retrieval systems [1-4]. Our theoretical analysis begins by modeling settings where authors compete to promote their documents across multiple queries as a continuous game in which each author allocates emphasis across different queries. We fully characterize when pure Nash equilibria exist and show that, unlike single-query settings, a stable state almost never exists [4]. To address this instability, we propose a novel method of strategic corpus enrichment. Specifically, we show using theoretical analysis how the mediator, by strategically planting documents in the corpus, can achieve stability in the system [2]. We conduct empirical experiments involving ranking competitions in which authors iteratively modify their documents to promote them across multiple related queries. These experiments also examine how the use of LLMs impacts human authors' behavior and strategies [4], and include direct competition between human and LLM generated documents. We propose a multi-agent framework in which query formulation, document generation, and ranking are each modeled as interacting agents (lexical, semantic, or LLM-based). Our empirical findings reveal that alignment or misalignment of agent strategies significantly impacts retrieval effectiveness and long-term dynamics [3]. This underscores the broader implications of integrating automated agents into IR systems. There are multiple promising directions for future research. First, we aim to extend our theoretical model by relaxing several assumptions and incorporating more sophisticated ranking functions. A more realistic model will allow us to design empirical experiments that validate our theoretical findings and bridge the gap between theory and practice. Additionally, we propose exploring different strategic agents in dynamic multi-agent retrieval ecosystems. We aim to develop learning-based query agents that adapt their formulations to maximize retrieval effectiveness and robust ranker agents that are capable of mitigating adversarial document modifications. Finally, we will explore techniques for enhancing ranking effectiveness in heterogeneous environments that include both human and AI-generated content.
Impact-ordered indexes, and thus Score-at-a-Time (SaaT) retrieval, is seeing a resurgence due to the emergence of learned sparse representations [4] (LSR) which employ the use of neural networks to learn the term-document weights. Within this context, SaaT has been found to be a competitive alternative to the more studied Document-at-a-Time (DaaT) approach. Even so, JASSv2 [2, 6] and IOQP [4] remain the only well-known open-source SaaT search engines. We believe that a thorough study into their operation and components may provide valuable insights. We wish to use this knowledge to build a better baseline system for SaaT retrieval. Once established, we aim to improve upon it. Specifically, we ask: (1) Can we build a better baseline for Score-at-a-Time retrieval by finding the most efficient and effective components from existing systems? (2) Can we improve upon the baseline by updating QMX to make use of more recent SIMD technologies? (3) Can we improve upon the baseline with a new accumulator management strategy? (4) Can we improve upon the baseline with a new approach to early termination/pruning? It is not obvious what the current baseline for SaaT should be, or what components are the most efficient. Through investigating various aspects of the search engines - specifically integer compression, accumulator management, and early termination (p) we will build a more indicative baseline system. We highlight these components in particular as we have reason to believe they will be influential. Architectures have changed considerably over the last ten years. We believe it would be worthwhile to update QMX [3, 5] (an integer compression codec) to work with these new technologies - and on both ARM and Intel CPUs. Similarly, we aim to produce a more efficient accumulator management strategy. Although Jia et al. [1] highlight its importance, there have been few developments regarding this aspect. Anecdotally, prior approaches with pruning and SaaT retrieval did not work well. However, new insights from LSR retrieval inspire a new approach in which we embed embeddings into an impactordered index. To date, we have reproduced previous experiments by Mackenzie et al. [4] comparing the performance of JASSv2 and IOQP, finding that it is not clear-cut as to which is the most efficient. We also replicated prior work by Trotman [5] and Trotman & Lilly [7] on index compression by exploring the use of a common codec across both systems. We found that SIMD BP-128 and QMX outperformed Elias Gamma SIMD VB in situ. Finally, alongside these investigations, we provide a means of quantizing the ciff indexes used by both search engines in our ciffTools 1 - which eliminates any discrepancies caused by different ranking functions and quantization methods. Next, we plan to approach accumulator management strategies and early termination (p) as we did compression - by swapping out components and observing the impact on performance. The results of this, along with our previous findings, will help inform our baseline for SaaT retrieval.
The increasing use of retrieval-augmented applications, where large language models (LLMs) are instructed to generate queries, assess relevance, and synthesise responses, has introduced new challenges in Information Retrieval (IR). The lack of transparency in LLMs means that even subtle variations in instructions can significantly impact the quality, consistency, and reliability of their responses. To address this issue, we propose Instruction-Response Study, an experimental framework for systematically analysing how task instructions influence LLM-generated responses in IR tasks. This paper presents the core components of the framework and demonstrates its utility through four case studies, examining 1) the effect of IR tasks on query formulation, 2) the impact of topic information size on retrieval effectiveness, 3) the reproducibility of LLM-generated queries, and 4) the role of meta-instructions in diversifying instruction design. The findings highlight how the proposed framework enables controlled experimentation on instruction design and its effects, offering a foundation for optimising prompt engineering and enhancing retrieval-augmented applications.
When a paired or two-sample t-test shows that a difference is not statistically significant (i.e., the null hypothesis.. is "accepted"), all we obtain as a "conclusion" is: "we cannot conclude from the data whether the difference is real or not." In general, statistical significance tests with a significance level of alpha give us a proper conclusion with 100( 1 - alpha)% confidence only if the.. is rejected. Hence, for example, in a reproducibility study in IR, traditional significance tests like the t-test are not adequate if one wishes to claim that the reproduced run is as good as the target run. In this paper, we show that the equivalence test, which has been gaining popularity in the medical domain for comparing new drugs with standard drugs, is applicable to IR research in situations such as above. We also discuss the less ambitious noninferiority test, which can be used, for example, when an IR researcher wants to claim that their system performs no worse than a state-of-the-art system that is computationally much more expensive and therefore earth-unfriendly. The equivalence and noninferiority tests require researchers to pre-define an indifference zone, which defines how much difference is "practically negligible"; hence we also discuss one possible approach to establishing an indifference zone for a given IR effectiveness measure. In addition, we provide recommendations for IR researchers who wish to conduct equivalence/noninferiority tests involving more than two systems. It is hoped that equivalence and noninferiority tests will become standard practices in IR, so that we can avoid making claims based on lack of statistical significance.
The integration of artificial intelligence into healthcare holds immense potential to transform medical services through personalized care, early diagnosis, and more informed decision-making. However, despite significant advances in digital health technologies, the adoption of AI-driven systems remains limited, particularly in sensitive domains such as mental health, due to concerns around transparency, safety, and user trust. These concerns are compounded by the complexity of clinical language, the high stakes of decisionmaking, and the need for accountability in automated systems. This research addresses these challenges by investigating how Large Language Models (LLMs) and related techniques can be adapted to create explainable, interpretable, and safe health information systems. Positioned at the intersection of health informatics, Natural Language Processing (NLP), and Information Retrieval (IR), this work explores both the potential and limitations of LLMs in clinical contexts. While LLMs are effective at processing large textual datasets and producing fluent responses, their black-box nature, tendency to hallucinate, and associated privacy risks make them unsuitable for direct application in healthcare without significant adaptation. This thesis proposes methodologies and systems focused on improving explanation generation, clinical relevance, and the safety of model outputs in health-related queries. The research is guided by eight specific goals: (1) Characterizing health-related language: Modeling the vocabulary used by individuals with specific health conditions using relevance-based statistical language models; (2) Generating natural language explanations: Designing interpretable models that explain classification outputs in ways meaningful to clinicians; (3) Adapting LLMs to clinical reasoning: Fine-tuning LLMs to reflect clinical workflows, particularly for retrieving symptom evidence and supporting diagnostic decisions; (4) Improving answer retrieval: Developing search systems for health queries with better result quality and contextual relevance; (5) Building health recommendation systems: Creating personalized tools for discovering reliable and relevant health information; (6) Ensuring safety and reliability: Creating "guardrails" to reduce hallucinations, biases, and misinformation in model outputs; and (7) Supporting multilingual and under-resourced settings: Extending solutions to diverse linguistic and cultural environments. Substantial progress has already been made. Goals G2 and G3 led to a Q1 journal publication on explainable depression detection using social media data [1], where we proposed and evaluated two architectures: a dual-model approach separating classification and explanation, and a unified LLM-based model that performs both tasks. For G4, we developed MindWell, a conversational agent for depression screening [2]. Based on transformer models and grounded in the Beck Depression Inventory-II (BDI-II), MindWell contextualizes user-generated content from social media to assist clinicians in identifying emotional patterns and symptoms. Ongoing work on G1 involves using statistical language models to identify lexical patterns linked to depressive behaviors online. For G7, we have trained an LLM adapted to Galician, an underrepresented co-official language in Spain, and are currently evaluating its performance in localized health information contexts [3].
Millions of consumers search for products online each day, aiming to find items that meet their needs at an acceptable price. While price and quality are major factors in purchasing decisions, ethical considerations increasingly influence consumer behavior-giving rise to the socially responsible consumer. Insights from a recent survey of over 600 consumers reveal that many barriers to ethical shopping stem from information-seeking challenges, often leading to decisions made under uncertainty. These challenges contribute to the intention-behaviour gap, where consumers' desire to make ethical choices is undermined by limited or inaccessible information and inefficacy of search systems in supporting responsible decision-making. In this perspectives paper, we argue that the field of Information Retrieval (IR) has a critical role to play by empowering consumers to make more informed and more responsible choices. We present three interrelated perspectives: (1) reframing responsible consumption as an information extraction problem aimed at reducing information asymmetries; (2) redefining product search as a complex task requiring interfaces that lower the cost and burden of responsible search; and (3) reimagining search as a process of knowledge calibration that helps consumers bridge gaps in awareness when making purchasing decisions. Taken together, these perspectives outline a path from query to conscience - one where IR systems help transform everyday product searches into opportunities for more ethical and informed choices. We advocate for the development of new and novel IR systems and interfaces that address the intricacies of socially responsible consumerism, and call on the IR community to build technologies that make ethical decisions more informed, convenient, and aligned with economic realities.
For decades, scholars have emphasized that tasks should be the central focus in Information Retrieval (IR). This point of view holds even more significance with the advent of Generative Artificial Intelligence (GenAI) models, which can, among other capabilities, understand natural language, engage in dialog with users, generate bespoke user interfaces, and power agents to help complete tasks. GenAI presents an unprecedented opportunity to finally realize the potential of tasks in IR, enhance task-focused retrieval and interaction, and create ''magical'' task completion moments for users. In this paper, we explore the rationale and methodology behind this argument. Traditional IR systems support mostly simple tasks. The emergence of GenAI creates an opportunity for IR systems to help users achieve complex tasks and for the IR community to rekindle its interest and demonstrate leadership in this sizable and significant problem space. We underscore the pivotal role of tasks in IR and introduce new evidence supporting the notion that task-centric approaches, abstracted from specific modalities, represent the future of IR. Building on this foundation, we envision the development, utilization, and evaluation of next-generation IR systems. We propose a promising future where IR agents prioritize users, their tasks, and their situations. However, despite their potential to address task-focused and modality-independent IR, agents alone are insufficient. We propose a robust ecosystem around these agents that transcends traditional queries, questions, prompts, and modalities to address users' fundamental needs, tasks, and goals.
Large Language Models (LLMs) have shown remarkable proficiency in Natural Language Generation (NLG) across various tasks. However, they often require additional resources beyond their internal knowledge to respond reliably to user queries. Determining the optimal methods, content, and timing for introducing new knowledge remains a critical challenge without clear solutions [2]. Our main objective is to enhance knowledge injection in LLMs to generate trustworthy responses. Therefore, this research is centered around two main research questions. (RQ1) What are the most effective and efficient choices of knowledge injection for questionanswering over less popular knowledge? A key challenge in knowledge injection is determining how to introduce new knowledge into an LLM while balancing both effectiveness and efficiency [4]. RAG and FT with synthetic data have emerged as two distinct paradigms, yet there has been no comprehensive comparison that highlights both their strengths and limitations [3]. To address this, we conduct an extensive evaluation of RAG and FT for handling less popular factual knowledge, assuming limited textual descriptions are available for a given domain and application. Through this analysis, we find that RAG substantially outperforms FT in this setup. Our second research question is: (RQ2) How can we quantify the uncertainty of LLMs during response generation and leverage it to improve the reliability of their outputs? By knowing LLMs uncertainty, one can determine when to leverage RAG and when to rely solely on LLMs' internal knowledge. However, existing Uncertainty Estimation (UE) techniques primarily focus on scenarios where the input consists only of a user query, overlooking the complexities introduced by retrieved knowledge. We investigate UE in the context of RAG and find that the performance of current UE methods is inconsistent, often degrading when non-parametric knowledge is incorporated into the input prompt. We propose an axiomatic framework to formalize optimal behavior of UE methods. Recent advancements in active RAG aim to enhance the dynamic interaction between retrievers and generators [1]. In this context, UE is primarily employed to determine when retrieval is necessary. Typically, uncertainty is measured based on next-token probabilities, and the interpretation of an uncertainty value is based on comparing it with accuracy. However, we raise three key questions: (RQ2.1) What is the optimal method for measuring uncertainty? We argue that relying solely on probability-based methods may not be the most effective approach. Furthermore, in conversational systems, previous turns or sessions do not directly influence nexttoken probabilities, highlighting the need for a new UE method. (RQ2.2) How should uncertainty values be interpreted? Current approaches, which define uncertainty through relative comparisons with other values, lack precision. Additionally, some applications correlate uncertainty with correctness, but how should uncertainty be interpreted in cases where correctness labels are unavailable? Moreover, we aim to enhance conversational interactions by leveraging an appropriate UE method. Specifically, we will investigate the research question: (RQ2.3) Can an LLM anticipate its next action, identify its information needs, and then generate an appropriate response? To explore this, we will utilize UE to help the LLM determine its next step, whether to generate a response, retrieve relevant information, ask a clarification question, or take an alternative action.
Conversational Information Seeking (CIS) systems aim to understand a user's evolving information needs in the context of a conversation and respond effectively [8]. These systems are especially valuable for users who may struggle with traditional interfaces, offering a more natural and accessible mode of interaction. To ensure the groundedness and accuracy of responses, existing methods break the task into several subtasks, namely, dialogue context modeling, retrieval, and answer generation. The user's information need is often represented by either a single rewritten query or a single representation in the query embedding space. This leads to several limitations, especially in cases where the information need cannot be answered using a single passage and requires complex reasoning over multiple facts from different sources. We address this limitation by proposing the MQ4CS model [1], which breaks the information need of the user into multiple queries with different aspects. In MQ4CS, the retrieval is done for each query and rank list fusion is done over the list of documents retrieved for each query Evaluation of responses generated by Retrieval Augmented Generation (RAG) systems remains an open problem. The quality of the response is mainly assessed using surface-based QA metrics, human evaluation, or instructing the Large Language Models (LLMs). Since RAG-generated responses integrate both retrieved documents and the LLM's internal knowledge, traditional surface-based QA metrics are not effective for assessment. Furthermore, existing RAG benchmarks address more complex information needs compared to ad-hoc retrieval and QA, highlighting the necessity of measuring the completeness of generated responses. To better evaluate completeness and correctness, we propose a nugget-based evaluation pipeline called CONE-RAG [2] that measures precision and recall of key information nuggets in generated answers. Personalization is an emerging key challenge in Conversational Search (CS). A personalized CS system must adapt its response based on the user's personal information and search history. Given the same user question, the response of a personalized CS system must be different for different users. To facilitate the research on development and evaluation of personalized CS systems, I have coorganized the Interactive Knowledge Assistance Track (iKAT) track at TREC [5], where we released the iKAT 2023 and 2024 datasets. Additionally, creating cost-effective, reusable test collections to evaluate both current and future search systems has been a persistent challenge in Information Retrieval (IR). The standard approach for building these collections is pooling. While pooling is a fairly robust method, it introduces evaluation biases. Systems that do not contribute to the pool can be unfairly penalized [7], as unjudged documents are assumed irrelevant. We address this challenge by measuring the reusability of the existing CS test collections and proposing an LLM-based relevance assessor model to enhance it [3]. Furthermore, user simulation is an important emerging research frontier for CS development and evaluation [6], where the focus mainly is on simulating the user behavior under a certain condition, such as answering clarifying questions or giving feedback on system's answer. The main drawback of existing research on user simulation is its reactive nature, where the simulated user just passively responds to the system's utterance. In real-world scenarios, however, users' actions are a mix of proactive and reactive actions. We address this challenge by proposing a proactive simulator which simulates both sides of the conversation. We use the proposed simulator to collect the SimQuAC dataset [4]. For the future, in summary, I plan to continue my research in two directions, including the development and evaluation of the personalized CS systems and improving the RAG systems based on the proposed nugget-based metrics.
Evaluating Information Retrieval (IR) systems is essential yet challenging. The dynamic nature of information and relevance further complicates IR. For example, Adar et al. observed that as early as 2009, websites frequently changed multiple times per hour [1]. In response, recent IR systems have become more personalized, semantic, and context-aware [3]. They evolved from lexical ranking functions to complex neural models embedded in feature-rich systems. While this often improves retrieval quality, it also challenges their reliability and robustness. The proposed research is motivated by the overarching goal of maintaining the trustworthiness of IR systems. To do so a rigorous evaluation is needed. Only by assessing the quality of a system can it be maintained and improved. Such an evaluation can not take place in isolation but must consider the dynamics of the search setting at all stages of a system-from development to maintenance and improvement. Based on the CRISP-DM methodology [2], these stages are sketched out as an "IR Life Cycle" in Figure 1, which we will further define and oppose with a continuous evaluation framework in future work. In this regard, the changing search setting and the alignment between evaluation methods are key problems to be investigated. The dynamic nature of the search setting makes it difficult to compare evaluation outcomes from different points in time. As systems and testbeds evolve, a common ground for comparison becomes elusive. Furthermore, when evaluations are conducted under dynamic conditions, their temporal validity becomes limited once the testbeds no longer match the current search setting. This calls into question how temporally robust evaluations are. Therefore, we aim to systematically reintroduce changes into evaluations and to identify synergies among evaluation methods. This will be addressed along the following research areas: Classifying changes: Initially, a formal definition of the search setting and its dynamics is created [4]. Accordingly, we will examine how search settings and systems evolve. To track the effectiveness over time, especially the relation between observable queries and latent information needs, need to be investigated. Comparing results: The changing setting makes a direct comparison of retrieval results difficult. To retain expressible and interpretable results and attribute the measured effects, we explored comparison strategies as a reproducibility problem [5]. Repeating evaluations: When the search setting deviates from the testbed, its validity becomes uncertain, and systems need to be re-evaluated. By estimating this uncertainty, it should be assessed if it is worth to update the testbed and re-evaluate the system. Aligning evaluation methods: To improve the feasibility of repeated evaluations, we aim to align evaluation methods to identify alternatives for demanding components and to uncover synergies between complementary components. In summary, it is intended to scrutinize the abstractions made by conventional IR evaluations and assess systems from multiple perspectives to gain a more holistic understanding of IR systems.
Continual knowledge graph embedding (CKGE) has gained wide attention for managing dynamic knowledge graphs (KGs), which are continuously updated with new facts. Unlike traditional methods designed for static KGs, CKGE enables incremental updates to KG embeddings to accommodate new facts while retaining previously learned knowledge. Despite these advancements, current CKGE studies and benchmarks primarily focus on handling the increasing scale of data while overlooking changes in graph patterns. These changes, altering the graph structure of KGs, are referred to as pattern shifts in this paper. Pattern shifts frequently arise as new facts are added, introducing significant challenges to the stability and adaptability of CKGE methods. To address this gap, we introduce a suite of novel and challenging benchmarks, called PS-CKGE, specifically designed to evaluate CKGE methods under pattern shifts, where logic rules are utilized to capture and manage structural changes in dynamic KGs. Through these benchmarks, we comprehensively evaluate current CKGE methods in terms of their overall performance, resistance to catastrophic forgetting, and adaptability to new knowledge. The results show that pattern shifts not only exacerbate their risk of catastrophic forgetting but also impair their adaptability, usually with greater performance degradation over triples associated with more significant changes.
Deep learning models usually require large labeled datasets to generalize well, but this is computationally and financially costly. Cold-start few-shot data selection enables fast model generalization by selecting a few diverse, representative samples from an unlabeled data pool. To achieve this goal, previous work usually divides the training data into several clusters and performs sampling from these clusters. Yet, such a way tends to have two issues. First, imbalanced data distribution in the training data pool still exists in the selected subset, causing models' performance biases and suboptimal generalization ability. Second, these methods improve sample diversity in each cluster by considering either the feature dissimilarity among instances, or model uncertainty for individual instance. They ignore the entire representativeness of samples within a cluster. To tackle these challenges, we propose a novel framework HCDS : Hierarchical Clustering for Cold-Start Few-Shot Data Selection. Specifically, we first perform class-level clustering, using pseudo-labels for class supervision and applying contrastive clustering to derive class-rich features. We then refine these features within the class-level clusters into semantically meaningful features and perform representation-level clustering. Finally, we sample data from the representation-level clusters based on global similarity to ensure representativeness. Experimental results on six public datasets, including both balanced and imbalanced ones, show that HCDS achieves state-of-the-art performance, particularly with limited and imbalanced data.
With the continuous growth of open data on the Web, dataset search has become a prominent specialized retrieval problem to find datasets relevant to a query. Recent solutions rank datasets based on not only their metadata, but also data snippets extracted from their actual data. While the goodness of a data snippet has been studied from various aspects, in this paper we propose to, for the first time, jointly optimize compactness, relevance, representativeness, and cohesiveness in snippet extraction. To extract such multi-objective data snippets, we formulate a new combinatorial optimization problem and design an efficient algorithm with a proved worst-case approximation ratio. We evaluate the data snippets extracted by our algorithm intrinsically through a set of quality metrics and extrinsically by applying them to dataset search.
As Named Entity Recognition (NER) tasks have evolved, artificial intelligence has been widely applied in this field. However, most benchmarks are limited to English, making it challenging to replicate successful experiences in other languages. To expand NER to informal and diverse Chinese text scenarios, we have proposed a new large-scale Chinese NER dataset, OmniNER2025. This dataset, obtained from user posts on a popular Chinese social media platform Xiaohongshu, contains 195,568 samples and 89 categories, all manually annotated. To our knowledge, it is currently the largest Chinese open-source NER dataset in terms of sample size, category diversity, and domain coverage. This dataset is more challenging than existing Chinese NER datasets and better reflects real-world applications. The large sample size and diverse entity types provide valuable research resources. Additionally, we introduced the ERRTA tool for error analysis and teacher model guidance, significantly reducing model errors and improving performance. In the future, we will refine the ERRTA framework and explore optimization strategies to enhance the practical value of NER models. By releasing the OmniNER2025 dataset and introducing the ERRTA tool, we have advanced fine-grained NER research and improved model performance, promoting its application and development in real-world scenarios.
Interactions between entities are often time-dependent in real-world systems such as e-commerce, social networks, streaming platforms, finance, and healthcare, and are best modeled as temporal interaction graphs. The temporal dimension plays a crucial role in modern recommendation systems, which rely on future link predictions. Temporal Graph Neural Networks (TGNN) have demonstrated state-of-the-art performance in future link prediction tasks for temporal interaction graphs. However, these models often require substantial training data unavailable in real-world settings. A potential solution to data scarcity is model pre-training on semantically related datasets. Unfortunately, transferring the TGNN model from one dataset to another is not trivial, as it contains node-specific memory modules vital for performance, resulting in them being inherently non-transferable. To overcome this limitation, we propose a novel transfer method that effectively utilizes common attributes between source and target datasets by decoupling graph nodes and corresponding attributes via bipartite encoding. This decoupling facilitates the transfer of memories and other inductive biases from source datasets to a target dataset. We evaluate the proposed transfer technique on real-world datasets and establish that it improves the performance of TGNN on the target dataset by 56% compared to the no-transfer methods and 36% over the state-of-the-art baselines in data-scarce settings.
Ship Re-Identification (ReID) aims to accurately identify ships with the same identity across different times and camera views, playing a crucial role in intelligent waterway transportation. However, compared to the widely researched pedestrian and vehicle ReID, Ship ReID has received much less attention, primarily due to the scarcity of large-scale and high-quality ship ReID datasets available for public access. Moreover, several unique challenges make ship ReID particularly difficult: ships are large objects that are hard to capture fully, and the visible area of ships vary significantly due to changes in cargo loading or water surface conditions. These challenges make it difficult to achieve ideal results by directly applying existing ReID methods. To address these challenges, in this paper, we introduce ShipReID-2400, a dataset for ship ReID compiled from a real-world intelligent waterway traffic monitoring system. It comprises 17,241 images of 2,400 distinct ship identities collected over 53 months, ensuring diversity and representativeness. Furthermore, we propose the Disentangle-to-Interact Network ( D2InterNet ), a simple but strong baseline for ship ReID designed to extract discriminative local features despite significant scale variations. Extensive experimental results show that D2InterNet achieves state-of-the-art performance on both the ShipReID-2400 and VesselReID datasets. In addition, despite being designed for ship ReID, D2InterNet also achieves competitive results on the MSMT17 pedestrian ReID dataset, showcasing its good generalization capability. Our dataset and code are publicly available at https://github.com/HuiGuanLab/ShipReID-2400.
Existing multimedia recommender systems have made the best efforts to predict user preferences for items by utilizing behavioral similarities between users and the modality features of items a user has interacted with. However, we identify two key limitations in existing methods regarding preferences for modality features: (L1) although preferences for modality features is an important aspect of users' preferences, existing methods only leverage neighbors with similar interactions and do not consider the neighbors who may have similar preferences for modality features while having different interactions; (L2) although modality features of a user and an item may have a complex geometric relationship in the latent space, existing methods overlook and face challenges in precisely capturing this relationship. To address these two limitations, we propose a novel multimedia recommendation framework, named MELON, which is based on two core ideas: (Idea 1) Modality-cEntered embedding extraction; (Idea 2) reLatiOnship-ceNtered embedding extraction. We validate the effectiveness and validity of MELON through extensive experiments with four real-world datasets, showing 10.51% higher accuracy compared to the best competitor in terms of recall@10. The code and dataset of MELON is available at https://github.com/Bigdasgit/MELON.
Low-resource complex named entity recognition focuses on identifying complex entities such as creative work, product name and so on, in scenarios where annotated training data is limited. Recent advanced works deal with this task through data augmentation and make substantial progress. However, existing methods ignore the influence of different types or levels of augmented data on model optimization in different learning stages. To address it, we propose a dual-perspective data augmentation and curriculum learning framework. Specifically, we first employ the large language model (LLM) to construct two kinds of augmented datasets from context-perspective and entity-perspective, respectively. Then, we present a multi-stage curriculum learning strategy including a novel adaptive curriculum arrangement algorithm to automatically select the most suitable kind of augmented set to optimize the target model at each training epoch, thus using the augmented data more effectively and controllably. Experimental results on the public benchmark across various low-resource settings show that our framework outperforms previous works.
