Cloud computing customers often submit repeating jobs and computation pipelines on approximately regular schedules, with arrival and running times that exhibit variance. This pattern, typical of training tasks in machine learning, allows customers to partially predict future job requirements. We develop a model of cloud computing platforms that receive statements of work (SoWs) in an online fashion. The SoWs describe future jobs whose arrival times and durations are probabilistic, and whose utility to the submitting agents declines with completion time. The arrival and duration distributions, as well as the utility functions, are considered private customer information and are reported by strategic agents to a scheduler that is optimizing for social welfare. We design pricing, scheduling, and eviction mechanisms that incentivize truthful reporting of SoWs. An important challenge is maintaining incentives despite the possibility of the platform becoming saturated. We introduce a framework to reduce scheduling under uncertainty to a relaxed scheduling problem without uncertainty. Using this framework, we tackle both adversarial and stochastic submissions of statements of work, and obtain logarithmic and constant competitive mechanisms, respectively.

Over the past 10 years, many recommendation techniques have been based on embedding users and items in latent vector spaces, where the inner product of a (user,item) pair of vectors represents the predicted affinity of the user to the item. A wealth of literature has focused on the various modeling approaches that result in embeddings, and has compared their quality metrics, learning complexity, etc. However, much less attention has been devoted to the issues surrounding productization of an embeddings-based high throughput, low latency recommender system. In particular, how the system might keep up with the changing embeddings as new models are learnt. This paper describes a reference architecture of a high-throughput, large scale recommendation service which leverages a search engine as its runtime core. We describe how the search index and the query builder adapt to changes in the embeddings, which often happen at a different cadence than index builds. We provide solutions for both id-based and feature-based embeddings, as well as for batch indexing and incremental indexing setups. The described system is at the core of a Web content discovery service that serves tens of billions recommendations per day in response to billions of user requests.

Contextual Multi-Armed Bandits is a well-known and accepted online optimization algorithm, that is used in many Web experiences to tailor content or presentation to users' traffic. Much has been published on theoretical guarantees (e.g. regret bounds) of proposed algorithmic variants, but relatively little attention has been devoted to the challenges encountered while productizing contextual bandits schemes in large scale settings. This work enumerates several productization challenges we encountered while leveraging contextual bandits for two concrete use cases at scale. We discuss how to (1) determine the context (engineer the features) that model the bandit arms; (2) sanity check the health of the optimization process; (3) evaluate the process in an offline manner; (4) add potential actions (arms) on the fly to a running process; (5) subject the decision process to constraints; and (6) iteratively improve the online learning algorithm. For each such challenge, we explain the issue, provide our approach, and relate to prior art where applicable.

Recommender systems are first and foremost about matching users with items the systems believe will delight them. The "main street" of personalization is thus about modeling users and items, and matching per user the items predicted to best satisfy the user. This holds for both collaborative filtering and content-based methods. In content discovery engines, difficulties arise from the fact that the content users natively consume on publisher sites does not necessarily match the sponsored content that drives the monetization and sustains those engines. The first part of this talk addresses this gap by sharing lessons learned and by discussing how the gap may be bridged at scale with proper techniques. The second part of the talk focuses on personalization of audiences on behalf of content marketing campaigns. From the marketers' side, optimizing audiences was traditionally done by refining targeting criteria, basically limiting the set of users to be exposed to their campaigns. Marketers then began sharing conversion data with systems, and the systems began optimizing campaign conversions by serving the campaign to users likely to transact with the marketer. Today, a hybrid approach of lookalike modeling combines marketers' targeting criteria with recommendation systems' models to personalize audiences for campaigns, with marketer ROI as the target.

Outbrain is the Web's leading content discovery service, recommending billions of stories daily to a global audience across many of the world's most prestigious and respected publishers. Outbrain's recommendation technology com- bines contextual cues with personalization, where the per- sonalization aspects are a combination of content-based and collaborative filtering techniques. This paper, and the accompanying demo, offer a behind- the-scenes view of the content-based aspects of Outbrain's personalization technology. We detail the types of features we extract from content, as well as the attributes we keep in each user's content-affinity profile. We then describe and demonstrate how we update each user's profile, in real time, as the user consumes content while browsing the Web.

It is well known that collaborative filtering (CF) based recommender systems provide better modeling of users and items associated with considerable rating history. The lack of historical ratings results in the user and the item cold-start problems. The latter is the main focus of this work. Most of the current literature addresses this problem by integrating content-based recommendation techniques to model the new item. However, in many cases such content is not available, and the question arises is whether this problem can be mitigated using CF techniques only. We formalize this problem as an optimization problem: given a new item, a pool of available users, and a budget constraint, select which users to assign with the task of rating the new item in order to minimize the prediction error of our model. We show that the objective function is monotone-supermodular, and propose efficient optimal design based algorithms that attain an approximation to its optimum. Our findings are verified by an empirical study using the Netflix dataset, where the proposed algorithms outperform several baselines for the problem at hand.

As consumers of television are presented with a plethora of available programming, improving recommender systems in this domain is becoming increasingly important. Television sets, though, are often shared by multiple users whose tastes may greatly vary. Recommendation systems are challenged by this setting, since viewing data is typically collected and modeled per device, aggregating over its users and obscuring their individual tastes. This paper tackles the challenge of TV recommendation, specifically aiming to provide recommendations for the next program to watch following the currently watched program the device. We present an empirical evaluation of several recommendation methods over large-scale, real-life TV viewership data. Our extentions of common state-of-the-art recommendation methods, exploiting the current watching context, demonstrate a significant improvement in recommendation quality.

Multifaceted search is a popular interaction paradigm that allows users to analyze and navigate through multidimensional data. A crucial aspect of faceted search applications is selecting the list of facets to display to the user following each query. We call this the facet selection problem. When refining a query by drilling down into a facet, documents that are associated with that facet are promoted in the rankings, as better-ranking documents not associated with the facet are filtered out. We formulate facet selection as an optimization problem aiming to maximize the rank promotion of certain documents. As the optimization problem is NP-Hard, we propose an approximation algorithm for selecting an approximately optimal set of facets per query. We conducted experiments over hundreds of queries and search results of a large commercial search engine, comparing two flavors of our algorithm to facet selection algorithms appearing in the literature. The results show that our algorithm significantly outperforms those baseline schemes.

The main monetization vehicle of many Web media sites are display ads located on article pages. Those ads are typically displayed either as banners on top of the page, or on the page's side bar. Advertiser ROI depends on the quality of ad targeting, as well as on how noticeable those ads are to users reading the article. Focusing on the latter issue, previous work has studied which ad positions are, on aggregate, more noticed by users. This work takes the first step toward the personalized positioning of ads on article pages. We demonstrate a correlation between the level of attention that users devote to a story, and the position of the most noticeable graphic element on the side bar. In particular, we find that the graphic element most noticed by a user is roughly to the side of the point in the article where the user's attention waned. We argue that this finding lays the foundation for increasing display advertising effectiveness by tailoring ad positions on each article page impression to the user viewing it.