Optimized Cost per Mille in Feeds Advertising

Advertising has become a dominant source of revenue generation on the Internet. Billions of advertisement slots are sold via auctions. And there are many pricing methods ,e.g., CPM (cost-per-mille), CPC (cost-per-click), CPA (cost-per-action), OCPM (optimized cost-per-mille) and so on. In this paper, we study the OCPM method (i.e., advertisers bid for conversions while pay per mille) under VCG auction. However, automatically bid in each view to maximize advertisers' conversions while still meet their target cost-per-conversion in feeds is difficult. To deal with these difficulties, we propose a reinforcement learning framework, i.e., RSDRL (ROI-sensitive distributional reinforcement learning). By making full use of the characteristics of auction rules which are missed by other methods, we design a reward function to surrogate conversion events and a bid generation method based on theoretical results. We also provide some theoretical results to guide hyperparameter tuning. Last, we validate RSDRL on a large industrial dataset with millions of auctions. Plenty of experiments (both online and offline) are used to evaluate the performance of our framework and RSDRL yields substantially better results than compared algorithms.