Strategies for Pre-training Graph Neural Networks

ICLR, 2020.

被引用54|引用|浏览768|来源
EI
关键词
Pre-training Transfer learning Graph Neural Networks
微博一下
Many applications of machine learning require a model to make accurate predictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training

摘要

Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it o...更多
简介
  • The authors' strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in.
  • Syntax Error: XObject 'Im2' is unknown
  • Syntax Error: Unknown font tag 'F30'
重点内容
  • In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in
  • Many applications of machine learning require a model to make accurate predictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training
  • An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and fine-tune it on a downstream task of interest
  • We develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs)
  • We find that naïve strategies, which pre-train Graph Neural Networks at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks
结果
  • Syntax Error (1312317): No font in show/space
  • Many applications of machine learning require a model to make accurate predictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training.
  • An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and fine-tune it on a downstream task of interest.
  • The authors develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs).
  • The key to the success of the strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously.
  • The authors systematically study pre-training on multiple graph classification datasets.
  • The authors find that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks.
  • Transfer learning refers to the setting where a model, initially trained on some tasks, is re-purposed on different but related tasks.
结论
  • Despite being an effective approach to transfer learning, few studies have generalized pre-training to graph data.
  • Pre-training has the potential to provide an attractive solution to the following two fundamental challenges with learning on graph datasets (Pan & Yang, 2009; Hendrycks et al, 2019): First, task-specific labeled data can be extremely scarce.
  • Out-of-distribution prediction is common in real-world graph datasets, for example, when one wants to predict chemical properties of a brand-new, just synthesized molecule, which is different from all molecules synthesized so far, and thereby different from all molecules in the training set.
总结
  • The authors' strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in.
  • Syntax Error: XObject 'Im2' is unknown
  • Syntax Error: Unknown font tag 'F30'
  • Syntax Error (1312317): No font in show/space
  • Many applications of machine learning require a model to make accurate predictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training.
  • An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and fine-tune it on a downstream task of interest.
  • The authors develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs).
  • The key to the success of the strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously.
  • The authors systematically study pre-training on multiple graph classification datasets.
  • The authors find that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks.
  • Transfer learning refers to the setting where a model, initially trained on some tasks, is re-purposed on different but related tasks.
  • Despite being an effective approach to transfer learning, few studies have generalized pre-training to graph data.
  • Pre-training has the potential to provide an attractive solution to the following two fundamental challenges with learning on graph datasets (Pan & Yang, 2009; Hendrycks et al, 2019): First, task-specific labeled data can be extremely scarce.
  • Out-of-distribution prediction is common in real-world graph datasets, for example, when one wants to predict chemical properties of a brand-new, just synthesized molecule, which is different from all molecules synthesized so far, and thereby different from all molecules in the training set.
基金
  • Develops a new strategy and self-supervised methods for pre-training Graph Neural Networks
  • Finds that naïve strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks
  • Our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in
您的评分 :
0

 

标签
评论