AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
Our work suggests a transition from studying individual Graph Neural Networks designs and tasks to systematically studying a GNN design space and a GNN task space

Design Space for Graph Neural Networks

NIPS 2020, (2021)

Cited by: 5|Views448
EI
Full Text
Bibtex
Weibo

Abstract

The rapid evolution of Graph Neural Networks (GNNs) has led to a growing number of new architectures as well as novel applications. However, current research focuses on proposing and evaluating specific architectural designs of GNNs, as opposed to studying the more general design space of GNNs that consists of a Cartesian product of dif...More

Code:

Data:

0
Introduction
  • The field of Graph Neural Network (GNN) research has made substantial progress in recent years.
  • In current GNN literature, GNN models are defined and evaluated as specific architectural designs
  • Architectures, such as GCN, GraphSAGE, GIN and GAT, are widely adopted in existing works [3, 4, 28, 41, 48, 37].
Highlights
  • The field of Graph Neural Network (GNN) research has made substantial progress in recent years
  • Based on the guidelines we discovered in Section 7.3, we fix several design dimensions to condense the GNN design space
  • Regarding (1), in Figure 5(e) last column, we show that the best GNN discovered in our condensed design space significantly outperforms existing SOTA (ROC AUC 0.792 versus 0.771)
  • Our extensive experimental results showed that coherently studying both spaces via tractable design space evaluation techniques can lead to exciting new understandings of GNN models and tasks, saving algorithm development costs as well as empirical performance gains
  • Our work suggests a transition from studying individual GNN designs and tasks to systematically studying a GNN design space and a GNN task space
  • Our approach serves as a tool to demonstrate the innovation of a novel GNN model, or a novel GNN task
Methods
  • Design space evaluation

    The authors' goal is to gain insights from the defined GNN design space, such as “Is batch normalization generally useful for GNNs?” the defined design and task space lead to over 10M possible combinations, prohibiting a full grid search.
  • The authors develop a controlled random search evaluation procedure to efficiently understand the trade-offs of each design dimension
  • Based on these innovations, the work provides the following key results: (1) A comprehensive set of guidelines for designing well-performing GNNs (Sec. 7.3).
  • (2) While best GNN designs for different tasks/datasets vary significantly, the defined GNN task space allows for transferring the best designs across tasks (Sec. 7.4)
  • This saves redundant algorithm development efforts for highly similar GNN tasks, while being able to identify novel GNN tasks which can inspire new GNN designs.
  • Using the proposed GraphGym platform reproducing experiments and fairly comparing models requires minimal effort
Results
  • The authors rank the design choices of BN ∈ [TRUE, FALSE] within each of the 96 setups by their performance (Figure 2(b)).
  • BN = TRUE has an average rank of 1.15, while the average rank of BN = FALSE is 1.44, indicating that adding BatchNorm to GNNs is generally helpful
  • This controlled random search technique can be generalized to design dimensions with multiple design choices.
  • Since task A has a high measured similarity, directly using the best design in task A already significantly outperforms SOTA on ogbg-molhiv (ROC AUC 0.785 versus 0.771)
Conclusion
  • In this paper the authors offered a principled approach to building a general GNN design space and a GNN task space with quantitative similarity metric.
  • The authors' extensive experimental results showed that coherently studying both spaces via tractable design space evaluation techniques can lead to exciting new understandings of GNN models and tasks, saving algorithm development costs as well as empirical performance gains.
  • Rather than criticizing the weakness of existing GNN architectures, the goal is to build a framework that can help researchers understand GNN design choices when developing new models suitable for different applications.
  • The authors' approach serves as a tool to demonstrate the innovation of a novel GNN model, or a novel GNN task
Summary
  • Introduction:

    The field of Graph Neural Network (GNN) research has made substantial progress in recent years.
  • In current GNN literature, GNN models are defined and evaluated as specific architectural designs
  • Architectures, such as GCN, GraphSAGE, GIN and GAT, are widely adopted in existing works [3, 4, 28, 41, 48, 37].
  • Objectives:

    The authors' goal is to gain insights from the defined GNN design space, such as “Is batch normalization generally useful for GNNs?” the defined design and task space lead to over 10M possible combinations, prohibiting a full grid search.
  • The authors' goal is to find the most diverse set of GNN designs that can reveal different aspects of a given GNN task.
  • Rather than criticizing the weakness of existing GNN architectures, the goal is to build a framework that can help researchers understand GNN design choices when developing new models suitable for different applications.
  • The authors' aim is not to cover all the design and evaluation aspects; in contrast, the authors wish to present a systematic framework which can inspire researchers to propose and understand new design dimensions and new tasks
  • Methods:

    Design space evaluation

    The authors' goal is to gain insights from the defined GNN design space, such as “Is batch normalization generally useful for GNNs?” the defined design and task space lead to over 10M possible combinations, prohibiting a full grid search.
  • The authors develop a controlled random search evaluation procedure to efficiently understand the trade-offs of each design dimension
  • Based on these innovations, the work provides the following key results: (1) A comprehensive set of guidelines for designing well-performing GNNs (Sec. 7.3).
  • (2) While best GNN designs for different tasks/datasets vary significantly, the defined GNN task space allows for transferring the best designs across tasks (Sec. 7.4)
  • This saves redundant algorithm development efforts for highly similar GNN tasks, while being able to identify novel GNN tasks which can inspire new GNN designs.
  • Using the proposed GraphGym platform reproducing experiments and fairly comparing models requires minimal effort
  • Results:

    The authors rank the design choices of BN ∈ [TRUE, FALSE] within each of the 96 setups by their performance (Figure 2(b)).
  • BN = TRUE has an average rank of 1.15, while the average rank of BN = FALSE is 1.44, indicating that adding BatchNorm to GNNs is generally helpful
  • This controlled random search technique can be generalized to design dimensions with multiple design choices.
  • Since task A has a high measured similarity, directly using the best design in task A already significantly outperforms SOTA on ogbg-molhiv (ROC AUC 0.785 versus 0.771)
  • Conclusion:

    In this paper the authors offered a principled approach to building a general GNN design space and a GNN task space with quantitative similarity metric.
  • The authors' extensive experimental results showed that coherently studying both spaces via tractable design space evaluation techniques can lead to exciting new understandings of GNN models and tasks, saving algorithm development costs as well as empirical performance gains.
  • Rather than criticizing the weakness of existing GNN architectures, the goal is to build a framework that can help researchers understand GNN design choices when developing new models suitable for different applications.
  • The authors' approach serves as a tool to demonstrate the innovation of a novel GNN model, or a novel GNN task
Tables
  • Table1: Condensed GNN design space based on the analysis in Section 7.3
Download tables as Excel
Related work
  • Graph Architecture Search. Architecture search techniques have been applied to GNNs [5, 51]. However, these works only focus on the design within each GNN layer instead of a general GNN design space, and only evaluate the designs on a small number of node classification tasks.
Funding
  • We also gratefully acknowledge the support of DARPA under Nos
Reference
  • H. Abdi. The kendall rank correlation coefficient. Encyclopedia of Measurement and Statistics. Sage, Thousand Oaks, CA, pages 508–510, 2007.
    Google ScholarFindings
  • J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
    Google ScholarLocate open access versionFindings
  • V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.
    Findings
  • F. Errica, M. Podda, D. Bacciu, and A. Micheli. A fair comparison of graph neural networks for graph classification. International Conference on Learning Representations (ICLR), 2020.
    Google ScholarFindings
  • Y. Gao, H. Yang, P. Zhang, C. Zhou, and Y. Hu. Graphnas: Graph neural architecture search with reinforcement learning. arXiv preprint arXiv:1904.09981, 2019.
    Findings
  • W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
    Google ScholarLocate open access versionFindings
  • K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision (ICCV), 2015.
    Google ScholarLocate open access versionFindings
  • K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
    Google ScholarLocate open access versionFindings
  • P. Holme and B. J. Kim. Growing scale-free networks with tunable clustering. Physical review E, 65(2):026107, 2002.
    Google ScholarLocate open access versionFindings
  • W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems (NeurIPS), 2020.
    Google ScholarLocate open access versionFindings
  • G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
    Google ScholarLocate open access versionFindings
  • S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
    Findings
  • W. Jin, R. Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. International Conference on Machine Learning (ICML), 2018.
    Google ScholarLocate open access versionFindings
  • K. Kersting, N. M. Kriege, C. Morris, P. Mutzel, and M. Neumann. Benchmark data sets for graph kernels, 2016.
    Google ScholarFindings
  • T. N. Kipf and M. Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.
    Findings
  • T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR), 2017.
    Google ScholarLocate open access versionFindings
  • G. Li, M. Muller, A. Thabet, and B. Ghanem. Deepgcns: Can gcns go as deep as cnns? In International Conference on Computer Vision (ICCV), 2019.
    Google ScholarLocate open access versionFindings
  • H. Maron, H. Ben-Hamu, H. Serviansky, and Y. Lipman. Provably powerful graph networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
    Google ScholarLocate open access versionFindings
  • C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
    Google ScholarLocate open access versionFindings
  • R. L. Murphy, B. Srinivasan, V. Rao, and B. Ribeiro. Relational pooling for graph representations. International Conference on Machine Learning (ICML), 2019.
    Google ScholarLocate open access versionFindings
  • V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML), 2010.
    Google ScholarLocate open access versionFindings
  • L. Page, S. Brin, R. Motwani, and T. Winograd. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab, 1999.
    Google ScholarFindings
  • I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár. Designing network design spaces. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
    Google ScholarLocate open access versionFindings
  • P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
    Findings
  • S. Santurkar, D. Tsipras, A. Ilyas, and A. Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems (NeurIPS), 2018.
    Google ScholarLocate open access versionFindings
  • P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
    Google ScholarLocate open access versionFindings
  • A. Shaw, W. Wei, W. Liu, L. Song, and B. Dai. Meta architecture search. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
    Google ScholarLocate open access versionFindings
  • O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann. Pitfalls of graph neural network evaluation. NeurIPS R2L Workshop, 2018.
    Google ScholarFindings
  • N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014.
    Google ScholarLocate open access versionFindings
  • M. R. Stoline. The status of multiple comparisons: simultaneous estimation of all pairwise comparisons in one-way anova designs. The American Statistician, 35(3):134–141, 1981.
    Google ScholarLocate open access versionFindings
  • P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. International Conference on Learning Representations (ICLR), 2018.
    Google ScholarLocate open access versionFindings
  • P. Velikovi, R. Ying, M. Padovano, R. Hadsell, and C. Blundell. Neural execution of graph algorithms. In International Conference on Learning Representations (ICLR), 2020.
    Google ScholarLocate open access versionFindings
  • D. J. Watts and S. H. Strogatz. Collective dynamics of ‘small-world’networks. Nature, 393(6684):440, 1998.
    Google ScholarLocate open access versionFindings
  • E. W. Weisstein. Bonferroni correction. https://mathworld.wolfram.com/, 2004.
    Findings
  • A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
    Google ScholarLocate open access versionFindings
  • C. Wong, N. Houlsby, Y. Lu, and A. Gesmundo. Transfer learning with neural automl. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
    Google ScholarLocate open access versionFindings
  • K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? International Conference on Learning Representations (ICLR), 2019.
    Google ScholarLocate open access versionFindings
  • K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka. Representation learning on graphs with jumping knowledge networks. International Conference on Machine Learning (ICML), 2018.
    Google ScholarLocate open access versionFindings
  • R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec. Graph convolutional neural networks for web-scale recommender systems. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2018.
    Google ScholarLocate open access versionFindings
  • R. Ying, Z. Lou, J. You, C. Wen, A. Canedo, and J. Leskovec. Neural subgraph matching. arXiv preprint, 2020.
    Google ScholarFindings
  • Z. Ying, D. Bourgeois, J. You, M. Zitnik, and J. Leskovec. GNNExplainer: Generating explanations for graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
    Google ScholarLocate open access versionFindings
  • Z. Ying, J. You, C. Morris, X. Ren, W. Hamilton, and J. Leskovec. Hierarchical graph representation learning with differentiable pooling. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
    Google ScholarLocate open access versionFindings
  • J. You, J. Leskovec, K. He, and S. Xie. Graph structure of neural networks. International Conference on Machine Learning (ICML), 2020.
    Google ScholarLocate open access versionFindings
  • J. You, B. Liu, R. Ying, V. Pande, and J. Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. Advances in Neural Information Processing Systems (NeurIPS), 2018.
    Google ScholarLocate open access versionFindings
  • J. You, X. Ma, D. Ding, M. Kochenderfer, and J. Leskovec. Handling missing data with graph representation learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
    Google ScholarLocate open access versionFindings
  • J. You, Y. Wang, A. Pal, P. Eksombatchai, C. Rosenburg, and J. Leskovec. Hierarchical temporal convolutional networks for dynamic recommender systems. In The Web Conference (WWW), 2019.
    Google ScholarLocate open access versionFindings
  • J. You, H. Wu, C. Barrett, R. Ramanujan, and J. Leskovec. G2SAT: Learning to generate sat formulas. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
    Google ScholarLocate open access versionFindings
  • J. You, R. Ying, and J. Leskovec. Position-aware graph neural networks. International Conference on Machine Learning (ICML), 2019.
    Google ScholarLocate open access versionFindings
  • A. R. Zamir, A. Sax, W. Shen, L. J. Guibas, J. Malik, and S. Savarese. Taskonomy: Disentangling task transfer learning. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
    Google ScholarLocate open access versionFindings
  • G. Zhang, H. He, and D. Katabi. Circuit-gnn: Graph neural networks for distributed circuit design. In International Conference on Machine Learning (ICML), 2019.
    Google ScholarLocate open access versionFindings
  • K. Zhou, Q. Song, X. Huang, and X. Hu. Auto-gnn: Neural architecture search of graph neural networks. arXiv preprint arXiv:1909.03184, 2019.
    Findings
  • M. Zitnik and J. Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):i190–i198, 2017.
    Google ScholarLocate open access versionFindings
  • B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科