AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
Beyond the single machine scenario, we demonstrate the scalability of PPRGo in a distributed setting and show that it is more efficient compared to multi-hop models

Scaling Graph Neural Networks with Approximate PageRank

KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining Virtual Event..., pp.2464-2473, (2020)

Cited by: 45|Views163
EI

Abstract

Graph neural networks (GNNs) have emerged as a powerful approach for solving many network mining tasks. However, learning on large graphs remains a challenge -- many recently proposed scalable GNN approaches rely on an expensive message-passing procedure to propagate information through the graph. We present the PPRGo model which utilizes...More

Code:

Data:

0
Introduction
  • Graph Neural Networks (GNNs) excel on a wide variety of network mining tasks from semi-supervised node classification and link prediction [25, 32, 44, 55] to community detection and graph classification [3, 15, 22, 37].
  • Increasing the number of layers is desirable since: (i) it allows the model to incorporate information from more distant neighbors; and it enables hierarchical feature extraction and the learning of richer node representations.
  • The recursive neighborhood expansion at each layer implies an exponential increase in the overall number of nodes the authors need to aggregate to produce the output at the final layer which is computationally prohibitive for large graphs.2
  • It has been shown [34, 52] that naively stacking multiple layers may suffer from over-smoothing that can reduce predictive performance
  • The recursive neighborhood expansion at each layer implies an exponential increase in the overall number of nodes the authors need to aggregate to produce the output at the final layer which is computationally prohibitive for large graphs.2 Second, it has been shown [34, 52] that naively stacking multiple layers may suffer from over-smoothing that can reduce predictive performance
Highlights
  • Graph Neural Networks (GNNs) excel on a wide variety of network mining tasks from semi-supervised node classification and link prediction [25, 32, 44, 55] to community detection and graph classification [3, 15, 22, 37]
  • There are few large graph baseline datasets available; apart from a handful of exceptions [16, 54], the scalability of most Graph Neural Networks methods has been demonstrated on graphs with fewer than 250K nodes
  • The majority of existing work focuses on improving scalability on a single machine
  • We present PPRGo, a Graph Neural Networks model that scales to large graphs in both single and multi-machine environments by using an adapted propagation scheme based on approximate personalized PageRank
  • We propose a Graph Neural Networks for semi-supervised node classification that scales to graphs with millions of nodes
  • Beyond the single machine scenario, we demonstrate the scalability of PPRGo in a distributed setting and show that it is more efficient compared to multi-hop models
Methods
  • Cluster-GCN SGC PPRGo (1 PI step) PPRGo (2 PI steps) Preprocessing.
  • 1175(25) 313(9) 2.26(4) 2.22(12) Runtime (s) Training Inference.
  • Per Epoch Overall Forward Propagation
Results
  • The authors show a significantly reduced inference time and propose sparse inference to achieve an additional 2x speed-up.
  • In contrast to most previously proposed methods [25, 49, 54] the authors utilize distributed computing techniques which significantly reduce the overall runtime of the method.
  • The authors can trade in a small amount of accuracy to significantly reduce inference time, in this case by 50 %
Conclusion
  • The authors propose a GNN for semi-supervised node classification that scales to graphs with millions of nodes.
  • Present MAG-Scholar – a new large-scale graph (12.4M nodes, 173M edges, and 2.8M node features) with coarse/fine-grained "groundtruth" node labels.
  • On this web-scale dataset PPRGo achieves high performance in under 2 minutes on a single machine.
  • Beyond the single machine scenario, the authors demonstrate the scalability of PPRGo in a distributed setting and show that it is more efficient compared to multi-hop models
Tables
  • Table1: Breakdown of the runtime, memory, and predictive performance on a single machine for different models on the Reddit dataset. We use 820 (20 · #classes) nodes for training. We see that PPRGo has a total runtime of less than 20 s and is two orders of magnitude faster than SGC and Cluster-GCN. PPRGo also requires less memory overall
  • Table2: Single machine runtime (s), memory (GB), and accuracy (%) for different models and datasets using 20 · #classes training nodes. PPRGo shows comparable accuracy and scales much better to large datasets than its competitors
Download tables as Excel
Related work
  • Scalability. GNNs were first proposed in Gori et al [24] and in Scarselli et al [42] and have since emerged as a powerful approach for solving many network mining tasks [1, 2, 9, 17, 22, 32, 42, 44]. Most GNNs do not scale to large graphs since they typically need to perform a recursive neighborhood expansion to compute the hidden representations of a given node. While several approaches have been proposed to improve the efficiency of graph neural networks [13, 14, 16, 21, 25, 27, 41, 49, 54], the scalability of GNNs to massive (web-scale) graphs is still under-studied. As we discussed in § 1 the most prevalent approach to scalability is to sample a subset of the graph, e.g. based on different importance scores for the nodes [16, 21, 25, 41, 54].3 Beyond sampling, Gao et al [21] collect the representations from a node’s neighborhood into a matrix, sort independently along each column/feature, and use the k largest entries as input to a 1-dimensional CNN. These techniques all focus on single-machine environments with limited (GPU) memory.
Funding
  • This research was supported by the Deutsche Forschungsgemeinschaft (DFG) through the Emmy Noether grant GU 1409/2-1 and the TUM International Graduate School of Science and Engineering (IGSSE), GSC 81
Reference
  • Sami Abu-El-Haija, Amol Kapoor, Bryan Perozzi, and Joonseok Lee. 2018. N-gcn: Multi-scale graph convolution for semi-supervised node classification. arXiv preprint arXiv:1802.08888 (2018).
    Findings
  • Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing. In ICML. 21–29.
    Google ScholarLocate open access versionFindings
  • Rami Al-Rfou, Dustin Zelle, and Bryan Perozzi. 2019. Ddgk: Learning graph representations for deep divergence graph kernels. In WWW. 37–48.
    Google ScholarLocate open access versionFindings
  • Reid Andersen, Christian Borgs, Jennifer T. Chayes, John E. Hopcroft, Vahab S. Mirrokni, and Shang-Hua Teng. 2008. Local Computation of PageRank Contributions. Internet Mathematics 5 (2008), 23–45.
    Google ScholarLocate open access versionFindings
  • Reid Andersen, Fan Chung, and Kevin Lang. 2006. Local graph partitioning using pagerank vectors. In FOCS. 475–486.
    Google ScholarLocate open access versionFindings
  • Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018).
    Findings
  • Aleksandar Bojchevski and Stephan Günnemann. 2018. Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking. (2018).
    Google ScholarFindings
  • Paolo Boldi, Violetta Lonati, Massimo Santini, and Sebastiano Vigna. 2006. Graph fibrations, graph isomorphism, and PageRank. RAIRO Theor. Informatics Appl. 40, 2 (2006), 227–253. https://doi.org/10.1051/ita:2006004
    Locate open access versionFindings
  • Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 (2013).
    Findings
  • Eliav Buchnik and Edith Cohen. 2018. Bootstrapped graph diffusions: Exposing the power of nonlinearity. In SIGMETRICS. 8–10.
    Google ScholarLocate open access versionFindings
  • Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R Henry, Robert Bradshaw, and Nathan Weizenbaum. 2010. FlumeJava: easy, efficient data-parallel pipelines. ACM Sigplan Notices 45, 6 (2010), 363–375.
    Google ScholarLocate open access versionFindings
  • Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, and Kevin Murphy. 2020. Machine Learning on Graphs: A Model and Comprehensive Taxonomy. arXiv preprint arXiv:2005.03675 (2020).
    Findings
  • Jie Chen, Tengfei Ma, and Cao Xiao. 2018. FastGCN: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:1801.10247 (2018).
    Findings
  • Jianfei Chen, Jun Zhu, and Le Song. 2018. Stochastic Training of Graph Convolutional Networks with Variance Reduction.. In ICML. 941–949.
    Google ScholarLocate open access versionFindings
  • Zhengdao Chen, Lisha Li, and Joan Bruna. 2018. Supervised Community Detection with Line Graph Neural Networks. (2018).
    Google ScholarFindings
  • Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. 2019. Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks. In KDD. ACM, 257–266.
    Google ScholarLocate open access versionFindings
  • Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems. 3844–3852.
    Google ScholarLocate open access versionFindings
  • Matthias Fey and Jan Eric Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. CoRR abs/1903.02428 (2019). arXiv:1903.02428 http://arxiv.org/abs/1903.02428
    Findings
  • Dániel Fogaras and Balázs Rácz. 2004. Towards Scaling Fully Personalized PageRank. In WAW.
    Google ScholarLocate open access versionFindings
  • Yasuhiro Fujiwara, Makoto Nakatsuji, Hiroaki Shiokawa, Takeshi Mishima, and Makoto Onizuka. 2013. Fast and Exact Top-k Algorithm for PageRank. In AAAI.
    Google ScholarFindings
  • Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. 2018. Large-scale learnable graph convolutional networks. In KDD. 1416–1424.
    Google ScholarFindings
  • Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212 (2017).
    Findings
  • David F Gleich, Kyle Kloster, and Huda Nassar. 2015. Localization in Seeded PageRank. arXiv preprint arXiv:1509.00016 (2015).
    Findings
  • Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In IJCNN. 729–734.
    Google ScholarLocate open access versionFindings
  • Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems. 1024–1034.
    Google ScholarLocate open access versionFindings
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR. 770–778.
    Google ScholarLocate open access versionFindings
  • Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. 2018. Adaptive Sampling Towards Fast Graph Representation Learning. In Advances in Neural Information Processing Systems. 4563–4572.
    Google ScholarLocate open access versionFindings
  • Glen Jeh and Jennifer Widom. 2003. Scaling personalized web search. In WWW.
    Google ScholarFindings
  • Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, Laszlo Lukacs, Marina Ganea, Peter Young, et al. 2016. Smart reply: Automated response suggestion for email. In KDD. 955–964.
    Google ScholarFindings
  • George Karypis and Vipin Kumar. 1998. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM Journal on scientific Computing 20, 1 (1998).
    Google ScholarLocate open access versionFindings
  • Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
    Findings
  • Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
    Findings
  • Johannes Klicpera, Aleksandar Bojchevski, and Stephan GÃijnnemann. 2019. Predict then Propagate: Graph Neural Networks meet Personalized PageRank. In ICLR. https://openreview.net/forum?id=H1gL-2A9Ym
    Locate open access versionFindings
  • Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. arXiv preprint arXiv:1801.07606 (2018).
    Findings
  • Peter Lofgren, Siddhartha Banerjee, and Ashish Goel. 2016. Personalized PageRank Estimation and Search: A Bidirectional Approach. In WSDM.
    Google ScholarFindings
  • Huda Nassar, Kyle Kloster, and David F Gleich. 2015. Strong localization in personalized PageRank vectors. In International Workshop on Algorithms and Models for the Web-Graph. Springer, 190–202.
    Google ScholarLocate open access versionFindings
  • Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. 2016. Learning convolutional neural networks for graphs. In ICML. 2014–2023.
    Google ScholarFindings
  • Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1998. The PageRank Citation Ranking: Bringing Order to the Web.
    Google ScholarFindings
  • Bryan Perozzi, Michael Schueppert, Jack Saalweachter, and Mayur Thakur. 2016. When Recommendation Goes Wrong: Anomalous Link Discovery in Recommendation Networks. In KDD. 569–578.
    Google ScholarLocate open access versionFindings
  • Sujith Ravi. 2016. Graph-powered Machine Learning at Google. https://ai.googleblog.com/2016/10/graph-powered-machine-learning-at-google.html.
    Locate open access versionFindings
  • Ryoma Sato, Makoto Yamada, and Hisashi Kashima. 2019. Constant Time Graph Neural Networks. arXiv preprint arXiv:1901.07868 (2019).
    Findings
  • Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2009. The graph neural network model. IEEE Transactions on Neural Networks 20, 1 (2009).
    Google ScholarLocate open access versionFindings
  • Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015. An Overview of Microsoft Academic Service (MAS) and Applications. In WWW 2015.
    Google ScholarLocate open access versionFindings
  • Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 1, 2 (2017).
    Findings
  • Sibo Wang, Youze Tang, Xiaokui Xiao, Yin Yang, and Zengxiang Li. 2016. HubPPR: Effective Indexing for Approximate Personalized PageRank. PVLDB 10 (2016), 205–216.
    Google ScholarLocate open access versionFindings
  • Sibo Wang, Renchi Yang, Xiaokui Xiao, Zhewei Wei, and Yin Yang. 2017. FORA: Simple and Effective Approximate Single-Source Personalized PageRank. In KDD.
    Google ScholarFindings
  • Xuanhui Wang, Azadeh Shakery, and Tao Tao. 2005. Dirichlet PageRank. In SIGIR.
    Google ScholarLocate open access versionFindings
  • Zhewei Wei, Xiaodong He, Xiaokui Xiao, Sibo Wang, Shuo Shang, and Ji-Rong Wen. 2018. TopPPR: Top-k Personalized PageRank Queries with Precision Guarantees on Large Graphs. In SIGMOD.
    Google ScholarFindings
  • Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. 2019. Simplifying Graph Convolutional Networks. arXiv preprint arXiv:1902.07153 (2019).
    Findings
  • Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. 2019. A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596 (2019).
    Findings
  • Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How Powerful are Graph Neural Networks? arXiv preprint arXiv:1810.00826 (2018).
    Findings
  • Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation Learning on Graphs with Jumping Knowledge Networks. arXiv preprint arXiv:1806.03536 (2018).
    Findings
  • Jaewon Yang and Jure Leskovec. 2015. Defining and evaluating network communities based on ground-truth. Knowledge and Information Systems 42, 1 (2015), 181–213.
    Google ScholarLocate open access versionFindings
  • Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018. Graph Convolutional Neural Networks for Web-Scale Recommender Systems. arXiv preprint arXiv:1806.01973 (2018).
    Findings
  • Muhan Zhang and Yixin Chen. 2018. Link Prediction Based on Graph Neural Networks. arXiv preprint arXiv:1802.09691 (2018).
    Findings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科