Pre-training text representations has recently been shown to significantly improve the state-of-the-art in many natural language processing tasks. The central goal of pre-training is to learn text representations that are useful for subsequent tasks. However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling. In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps. The standard multi-task learning objective adopted in BERT is a special case of our learning algorithm where the depth of meta-train is zero. We study the problem in two settings: unsupervised pre-training and supervised pre-training with different pre-training objects to verify the generality of our approach.Experimental results show that our algorithm brings improvements and learns better initializations for a variety of downstream tasks.

Result ranking in commercial web search engines is based on a wide array of signals, from keywords appearing on web pages to behavioral (clickthrough) data aggregated across many users or from the current user only. The recent emergence of wearable devices has enabled the collection of physiological data such as heart rate, skin temperature, and galvanic skin response at a population scale. These data are useful for many public health tasks, but they may also provide novel clues about people's interests and intentions as they engage in online activities. In this paper, we focus on heart rate and show that there are strong relationships between heart rate and various measures of user interest in a search result. We integrate features of heart rate, including heart rate dynamics, as additional attributes in a competitive machine-learned web search ranking algorithm. We show that we can obtain significant relevance improvements from this physiological sensing that vary depending on the search topic.