Content platforms typically engage with their users through small recommendation sets of items drawn from an extensive catalog. These sets are curated using machine-learned models optimized to present choices most likely to align with user preferences. We present surprising findings about such platforms. Even with complete information on user preferences within sets of up to k items, these models can only predict preferences within a "quadratic horizon" of k 2 items and might fail to identify the best items in larger sets. To illustrate, we present striking examples where a platform interacting with users through small item sets, despite knowing that one item is favored by millions of users, cannot identify this item with better than random chance. Through both theoretical analysis and studies across various datasets, we demonstrate that "hidden gems," items preferred by many users but invisible to platforms, exist in real-world datasets of moderate size, highlighting a significant gap in current recommendation platforms.

Traffic congestion remains a persistent challenge for urban mobility, increasing travel delays and elevating CO2 emissions. The widespread use of smartphones and GPS navigation creates new opportunities to mitigate congestion through routing optimizations in apps, yet real-world evidence for the effectiveness of such interventions is limited. Here we report large-scale empirical experiments evaluating routing-based traffic interventions on similar to 100 highly congested road segments across 10 major US cities. By rerouting a small share of Google Maps trips from targeted congested highway and arterial segments to less congested alternatives of equivalent road-classes with comparable travel times, we observe a city-average 2% increase in vehicle speeds on the intervened segments, along with improved travel times 0.7% and potential annual reductions exceeding 1,000 tons of CO2-equivalent emissions per city in the majority of studied locations. These findings provide evidence that marginal routing interventions involving a small proportion of vehicles can measurably enhance overall road network efficiency, offering a practical pathway to easing congestion and advancing urban sustainability.

A Multinomial Logit (MNL) model is composed of a finite universe of items [n]={1,..., n}, each assigned a positive weight. A query specifies an admissible subset – called a slate – and the model chooses one item from that slate with probability proportional to its weight. This query model is also known as the Plackett-Luce model or conditional sampling oracle in the literature. Although MNLs have been studied extensively, a basic computational question remains open: given query access to slates, how efficiently can we learn weights so that, for every slate, the induced choice distribution is within total variation distance ε of the ground truth? This question is central to MNL learning and has direct implications for modern recommender system interfaces. We provide two algorithms for this task, one with adaptive queries and one with non-adaptive queries. Each algorithm outputs an MNL M' that induces, for each slate S, a distribution M'_S on S that is within ε total variation distance of the true distribution. Our adaptive algorithm makes O(n/ε^3log n) queries, while our non-adaptive algorithm makes O(n^2/ε^3log n logn/ε) queries. Both algorithms query only slates of size two and run in time proportional to their query complexity. We complement these upper bounds with lower bounds of Ω(n/ε^2log n) for adaptive queries and Ω(n^2/ε^2log n) for non-adaptive queries, thus proving that our adaptive algorithm is optimal in its dependence on the support size n, while the non-adaptive one is tight within a log n factor.

Agent skills package specialized knowledge and workflows into reusable resources that extend AI agent capabilities. Recent work automatically discovers such skills from agent experience, which enables agents to progressively adapt through interaction. However, the insights that guide skill development typically remain scattered across optimization histories, limiting their systematic reuse across iterations. We introduce WikiSkill, a framework that co-evolves agent skills with a persistent knowledge base (wiki). At a high level, WikiSkill separates raw execution experience, accumulated knowledge, and executable skills, while continuously consolidating experience into the wiki, which subsequent skill updates can build on. Across diverse benchmarks and models, WikiSkill consistently outperforms state-of-the-art skill-evolution methods and improves over no-skill baselines in most model-benchmark settings. We find that skill evolution complements model scaling: larger models generally benefit more from evolved skills, while smaller models with skills can outperform substantially larger models without them. We also find that evolved skills transfer effectively across models and model families, and skills evolved by other models can outperform self-evolved skills. Finally, our ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution. These results demonstrate the benefits of systematically accumulating and refining agent experience for developing reusable and transferable skills.

In this work we settle the complexity of three sketching problems. (i) We show that sketching vertex neighborhood sizes in graphs requires Ω(n^2) bits, standing in sharp contrast to the Õ(n) complexity of sketching edge cuts. (ii) We obtain tight lower and upper bounds of (n^2) for sketching coverage functions with additive and multiplicative errors. (iii) We prove an Ω(n^2) lower bound for sketching Random Utility Models under the ℓ_∞-norm, improving upon the previous Ω(n log n) bound and matching a known upper bound to within logarithmic factors. These bounds are obtained through a connection with the problem of sketching the intersection profile of a distribution D on 2^[n]. Specifically, we seek a succinct data structure that, for any query set S ⊆ [n], approximates the quantity _T ∼ D[T ∩ S ≠∅] to within a small constant additive error. One can obtain lower bounds for this latter problem directly from known results about the itemset frequency estimation problem in databases for which tight bounds are known. As an additional contribution, we also provide an alternative proof for the intersection profile sketching lower bound, in the setting in which the accuracy parameter is constant. This proof relies solely on elementary probability avoiding the heavier machinery used in previous proofs.

Recent work suggests that LLMs can improve their abilities through \textit{self-evolution}, using only internally generated supervision. A central open question, however, is not whether self-evolution can help, but: \textit{how far is it from oracle-supervised training under minimal assumptions?} To address this question, we present a controlled empirical analysis of LLM self-evolution under a strict formulation: self-evolution is allowed access only to (i) an unlabeled prompt set and (ii) a base language model, with all supervision signals generated from this model. Under this formulation, we can evaluate many self-evolution approaches in a unified preference optimization framework. Specifically, we analyze four representative self-evolution methods, ranging from single-round verification to multi-turn feedback, iterative training, and curriculum learning. For our primary analysis, we use a clean setting based on the Knights and Knaves logical reasoning dataset, which provides deterministic solutions, systematic verification, and a hierarchy of difficulty levels that enables an evaluation of easy-to-hard generalization. Across this controlled setting, we find that increasingly complex self-evolution strategies yield consistent but limited gains. In general, a substantial performance gap persists relative to oracle supervision. One strategy stands out as effective: we nearly match the oracle performance by using a larger model (Gemma 12B) with iterative revision based on natural language feedback. We also study self-evolution on the OpenThoughts reasoning corpus and evaluate on standard problem-solving benchmarks. In this regime, self-evolution only leads to modest improvements, including when using more resource-intensive strategies or online RL. Overall, our results shed new light on the empirical limits of various types of self-evolution.

Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the self-evolution algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework: single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that self-evolution consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non-trivial gap to oracle supervision. We find that multi-turn critic-revision with large models can reach strong self-evolution performance, with Gemma 12B nearly matching oracle-supervised training. Beyond Knights and Knaves, we also evaluate self-evolution on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop self-evolution can help and show how internally generated supervision remains insufficient under this minimal formulation.

Discrete choice models are used to describe, explain, and predict choices made by people among a finite set of alternatives. However, standard discrete choice models come with an unrealistic assumption: that users are able to provide an unequivocal clear winner from any slate of alternatives. Often the user knows the winner but cannot report it, as when a UI does not allow a user to specify which of two movies they rated five stars is better. And often, among the myriad options available, the user is able to identify some good candidates, but finds it difficult to distinguish between the top contenders. In this paper, we study the problem of interacting with user choice data in which, sometimes, the user is unable to settle on a single compelling winner. To address this issue, we introduce an extension to the well-known random utility models (RUMs), which we call RUMs-with-Ties, where comparisons can result in a tie. We begin with an axiomatic formulation of Luce dating to the 1950s, and provide algorithms and matching lower bounds for operating on data with ties. We also provide a comprehensive comparison of RUMs versus RUMs-with-Ties from different angles. We present theoretical results indicating that simple ways of incorporating ties into existing approaches are unlikely to perform well. We also prove in our setting that the presence of additional items, even if lower in quality, allows an algorithm to learn the highest ranked element with far fewer trials. Finally, we provide experimental evaluations of different approaches to handling indistinguishable items in choice settings and demonstrate the advantages of direct modeling of ties via our approach.

Estimating Origin-Destination (OD) travel demand is vital for effective urban planning and traffic management. Developing universally applicable OD estimation methodologies is significantly challenged by the pervasive scarcity of high-fidelity traffic data and the difficulty in obtaining city-specific prior OD estimates (or seed ODs), which are often prerequisite for traditional approaches. Our proposed method directly estimates OD travel demand by systematically leveraging aggregated, anonymized statistics from Google Maps Traffic Trends, obviating the need for conventional census or city-provided OD data. The OD demand is estimated by formulating a single-level, one-dimensional, continuous nonlinear optimization problem with nonlinear equality and bound constraints to replicate highway path travel times. The method achieves efficiency and scalability by employing a differentiable analytical macroscopic network model. This model by design is computationally lightweight, distinguished by its parsimonious parameterization that requires minimal calibration effort and its capacity for instantaneous evaluation. These attributes ensure the method's broad applicability and practical utility across diverse cities globally. Using segment sensor counts from Los Angeles and San Diego highway networks, we validate our proposed approach, demonstrating a two-thirds to three-quarters improvement in the fit to segment count data over a baseline. Beyond validation, we establish the method's scalability and robust performance in replicating path travel times across diverse highway networks, including Seattle, Orlando, Denver, Philadelphia, and Boston. In these expanded evaluations, our method not only aligns with simulation-based benchmarks but also achieves an average 13

Modern machine learning and AI have revolutionized the generation of ranking and recommendations across many domains, taking data-driven approaches to inferring the candidate items a user is most likely to select. The theory of discrete choice provides the theoretical underpinnings for study of these problems. This theory is central to economics and behavioral sciences, and was recognized with the 2000 Nobel Prize in economics awarded to Daniel McFadden for his work on the analysis of discrete choice. Classical work in this area, and a wide range of recent advances, have much to offer in thinking about how to support users in making choices. However, many of the central tools of discrete choice are not broadly known to the researchers in our area. In this proposed tutorial, we will cover the foundations of the field, and provide connections to common tools of machine learning such as logistic regression, multinomial regression, and softmax. We will cover a number of results about the representational power and complexity of learning various models of choice. And finally, we will suggest both open problems in the field, and areas where discrete choice tools are relevant to key problems in the Web Conference community.

A Random Utility Model (RUM) is a classical model of user behavior defined by a distribution over $\mathbb{R}^n$. A user, presented with a subset of $\\{1,\ldots,n\\}$, will select the item of the subset with the highest utility, according to a utility vector drawn from the specified distribution. In practical settings, the subset is often of small size, as in the ``ten blue links'' of web search. In this paper, we consider a learning setting with complete information on user choices from subsets of size at most $k$. We show that $k=\Theta(\sqrt{n})$ is both necessary and sufficient to predict the distribution of all user choices with an arbitrarily small, constant error. Based on the upper bound, we obtain new algorithms for approximate RUM learning and variations thereof. Furthermore, we employ our lower bound for approximate RUM learning to derive lower bounds to fractional extensions of the well-studied $k$-deck and trace reconstruction problems.

Probes are small networks that predict properties of underlying data from embeddings, and they provide a targeted, effective way to illuminate the information contained in embeddings. While analysis through the use of probes has become standard in NLP, there has been much less exploration in vision. Image foundation models have primarily been evaluated for semantic content. Better understanding the non-semantic information in popular embeddings (e.g., MAE, SimCLR, or CLIP) will shed new light both on the training algorithms and on the uses for these foundation models. We design a systematic transformation prediction task and measure the visual content of embeddings along many axes, including image style, quality, and a range of natural and artificial transformations. Surprisingly, six embeddings (including SimCLR) encode enough non-semantic information to identify dozens of transformations. We also consider a generalization task, where we group similar transformations and hold out several for testing. We find that image-text models (CLIP and ALIGN) are better at recognizing new examples of style transfer than masking-based models (CAN and MAE). Overall, our results suggest that the choice of pre-training algorithm impacts the types of information in the embedding, and certain models are better than others for non-semantic downstream tasks.

The 16th ACM International Conference on Web Search and Data Mining (WSDM 2023) was held in Singapore. It was held as in-person conference that also featured rich virtual elements. This brief report provides an overview of WSDM 2023 with organization and program details and statistics from Conference Chairs and Program Committee Chairs, as well as a message from the WSDM Steering Committee Chair. Date: 27 February-3 March 2023. Website: https://www.wsdm-conference.org/2023.

The 16th ACM International Conference on Web Search and Data Mining (WSDM 2023) was successfully held on 2/27 - 3/3/2023 at Carlton Hotel, Singapore. It was held as in-person conference that also featured rich virtual elements. The five-day event began with tutorials and Industry Day. The middle three days were dedicated to keynote speeches, single-session presentations, poster reception, and demo sessions. The final day was event-filled with workshops, WSDM Cup Day, Doctoral Consortium, and Smart City Day. This brief report provides an overview of WSDM 2023 with organization and program details and statistics from Conference Chairs and Program Committee Chairs, as well as a message from the WSDM Steering Committee Chair. Further details can be found at https://www.wsdm-conference.org/2023/. At the conclusion of this report, we express our deep gratitude to authors, volunteers, special interest groups (SIGs), steering committee, sponsors and supporters.

Work in information retrieval has largely been centered around ranking and relevance: given a query, return some number of results ordered by relevance to the user. The problem of result list truncation, or where to truncate the ranked list of results, however, has received less attention despite being crucial in a variety of applications. Such truncation is a balancing act between the overall relevance, or usefulness of the results, with the user cost of processing more results. Result list truncation can be challenging because relevance scores are often not well-calibrated. This is particularly true in large-scale IR systems where documents and queries are embedded in the same metric space and a query's nearest document neighbors are returned during inference. Here, relevance is inversely proportional to the distance between the query and candidate document, but what distance constitutes relevance varies from query to query and changes dynamically as more documents are added to the index. In this work, we propose Surprise scoring, a statistical method that leverages the Generalized Pareto Distribution that arises in extreme value theory to produce interpretable and calibrated relevance scores at query time using nothing more than the ranked scores. We demonstrate its effectiveness on the result list truncation task across image, text, and IR datasets and compare it to both classical and recent baselines. We draw connections to hypothesis testing and p-values.

In this work we consider the problem of fitting Random Utility Models (RUMs) to user choices. Given the winner distributions of the subsets of size k of a universe, we obtain a polynomial-time algorithm that finds the RUM that best approximates the given distribution on average. Our algorithm is based on a linear program that we solve using the ellipsoid method. Given that its corresponding separation oracle problem is NP-hard, we devise an approximate separation oracle that can be viewed as a generalization of the weighted feedback arc set problem to hypergraphs. Our theoretical result can also be made practical: we obtain a heuristic that is effective and scales to real-world datasets.

Random utility models (RUMs) encode the likelihood that a particular item will be selected from a slate of competing items. RUMs are well-studied objects in both discrete choice theory and, more recently, in the machine learning community, as they encode a fairly broad notion of rational user behavior. In this paper, we focus on slates of size two representing head-to-head contests. Given a tournament matrix M such that M-i,M-j is the probability that item j will be selected from {i , j}, we consider the problem of finding the RUM that most closely reproduces M. For this problem we obtain a polynomial-time algorithm returning a RUM that approximately minimizes the average error over the pairs. Our experiments show that RUMs can perfectly represent many of the tournament matrices that have been considered in the literature; in fact, the maximum average error induced by RUMs on the matrices we considered is negligible (approximate to 0.001). We also show that RUMs are competitive, on prediction tasks, with previous approaches.

This paper considers the calibration of travel demand inputs, defined as a set of origin-destination matrices (ODs) , for stochastic microscopic urban traffic simulators. The goal of calibration is to find a (set of) travel demand input(s) that replicate sparse field count data statistics. While traditional approaches use only first-order moment information from the field data, it is well known that the OD calibration problem is underdetermined in realistic networks. We study the value of using higher-order statistics from spatially sparse field data to mitigate underdetermination, proposing a variational inference technique that identifies an OD distribution. We apply our approach to a high-dimensional setting in Salt Lake City, Utah. Our approach is flexible---it can be readily extended to account for arbitrary types of field data (e.g., road, path or trip data).

A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem.