## AI帮你理解科学

## AI 精读

AI抽取本论文的概要总结

微博一下：

# Scalable Graph Neural Networks via Bidirectional Propagation

NIPS 2020, (2020)

EI

关键词

摘要

Graph Neural Networks (GNN) is an emerging field for learning on non-Euclidean data. Recently, there has been increased interest in designing GNN that scales to large graphs. Most existing methods use "graph sampling" or "layer-wise sampling" techniques to reduce training time. However, these methods still suffer from degrading performa...更多

代码：

数据：

简介

- The field of Graph Neural Networks (GNNs) has drawn increasing attention due to its wide range of applications such as social analysis [23, 20, 28], biology [10, 26], recommendation systems [36], and computer vision [39, 7, 13].
- On the other hand, training GCN with mini-batches is difficult, as the neighborhood size could grow exponentially with the number of layers.
- These techniques can be broadly divided into three categories: 1) Layer-wise sampling methods: GraphSAGE [11] proposes a neighbor-sampling method to sample a fixed number of neighbors for each node.
- FastGCN [5] samples nodes of each layer independently

重点内容

- The field of Graph Neural Networks (GNNs) has drawn increasing attention due to its wide range of applications such as social analysis [23, 20, 28], biology [10, 26], recommendation systems [36], and computer vision [39, 7, 13]
- The vanilla Graph Convolutional Network (GCN) uses a full-batch training process and stores each node’s representation in the GPU memory, which leads to limited scalability
- VRGCN [6] leverages historical activations to restrict the number of sampled nodes and reduce the variance of sampling
- To reduce the time complexity, we propose approximating the Generalized PageRank matrix P with a localized bidirectional propagation algorithm from both the training/testing nodes and the feature vectors
- This paper presents GBP, a scalable GNN based on localized bidirectional propagation
- GBP is the first method that can scale to billion-edge networks on a single machine

方法

- GCN GAT APPNP GDC SGC LADIES PPRGo GraphSAINT

URL https://github.com/rusty1s/pytorch_geometric https://github.com/rusty1s/pytorch_geometric https://github.com/rusty1s/pytorch_geometric https://github.com/klicperajo/gdc https://github.com/Tiiiger/SGC https://github.com/acbull/LADIES https://github.com/TUM-DAML/pprgo_pytorch https://github.com/GraphSAINT/GraphSAINT Commit

5692a8 5692a8 5692a8 14333f 6c450f c7f987 d9f991 cd31c3.

结果

- The authors observe that GBP can achieve a significantly higher F1-score with less running time.

结论

- This paper presents GBP, a scalable GNN based on localized bidirectional propagation.
- The bidirectional propagation process computes a Generalized PageRank matrix that can express various existing graph convolutions.
- Extensive experiments on real-world graphs show that GBP obtains significant improvement over the state-of-the-art methods in terms of efficiency and performance.
- GBP is the first method that can scale to billion-edge networks on a single machine.
- An interesting direction is to extend GBP to heterogeneous networks

总结

## Introduction:

The field of Graph Neural Networks (GNNs) has drawn increasing attention due to its wide range of applications such as social analysis [23, 20, 28], biology [10, 26], recommendation systems [36], and computer vision [39, 7, 13].- On the other hand, training GCN with mini-batches is difficult, as the neighborhood size could grow exponentially with the number of layers.
- These techniques can be broadly divided into three categories: 1) Layer-wise sampling methods: GraphSAGE [11] proposes a neighbor-sampling method to sample a fixed number of neighbors for each node.
- FastGCN [5] samples nodes of each layer independently
## Methods:

GCN GAT APPNP GDC SGC LADIES PPRGo GraphSAINT

URL https://github.com/rusty1s/pytorch_geometric https://github.com/rusty1s/pytorch_geometric https://github.com/rusty1s/pytorch_geometric https://github.com/klicperajo/gdc https://github.com/Tiiiger/SGC https://github.com/acbull/LADIES https://github.com/TUM-DAML/pprgo_pytorch https://github.com/GraphSAINT/GraphSAINT Commit

5692a8 5692a8 5692a8 14333f 6c450f c7f987 d9f991 cd31c3.## Results:

The authors observe that GBP can achieve a significantly higher F1-score with less running time.## Conclusion:

This paper presents GBP, a scalable GNN based on localized bidirectional propagation.- The bidirectional propagation process computes a Generalized PageRank matrix that can express various existing graph convolutions.
- Extensive experiments on real-world graphs show that GBP obtains significant improvement over the state-of-the-art methods in terms of efficiency and performance.
- GBP is the first method that can scale to billion-edge networks on a single machine.
- An interesting direction is to extend GBP to heterogeneous networks

- Table1: Summary of time complexity for GNN training and inference
- Table2: Dataset statistics
- Table3: Hyper-parameters of GBP. rmax is the Reverse Push Threshold, w is the number of random walks from the training nodes, w is the weight sequence, r is the Laplacian parameter in the convolutional matrix Dr−1AD−r
- Table4: Results on Cora, Citeseer and Pubmed
- Table5: Results of inductive learning with scalable GNNs
- Table6: Results for semi-supervised learning on Friendster
- Table7: URLs of baseline codes

基金

- Acknowledgments and Disclosure of Funding Ji-Rong Wen was supported by National Natural Science Foundation of China (NSFC) No.61832017, and by Beijing Outstanding Young Scientist Program NO
- Zhewei Wei was supported by NSFC No 61972401 and No 61932001, by the Fundamental Research Funds for the Central Universities and the Research Funds of Renmin University of China under Grant 18XNLG21, and by Alibaba Group through Alibaba Innovative Research Program
- Ye Yuan was supported by NSFC No 61932004 and No 61622202, and by FRFCU No N181605012
- Xiaoyong Du was supported by NSFC No U1711261

研究对象与分析

open graph datasets with different size: 7

Datasets. We use seven open graph datasets with different size: three citation networks Cora, Citeser and Pubmed [25], a Protein-Protein interaction network PPI [11], a customer interaction. Cora Citeseer Pubmed PPI Yelp Amazon Friendster

across all datasets: 4

On the Friendster dataset, where the features are random noises, we use both Personalized PageRank and transition probability (wL = 1, w0 =, . . . , = wL−1 = 0) for GBP. We set L = 4 across all datasets. Table 3 summaries other hyper-parameters of GBP on different datasets

large datasets: 3

Inductive learning on medium to large graphs. Table 5 reports the F1-score and running time (precomputation + training) of each method with various depths on three large datasets PPI, Yelp, and Amazon. For each dataset, we set the hidden dimension to be the same across all methods: 2048(PPI), 2048(Yelp), and 1024(Amazon)

引用论文

- ABU-EL-HAIJA, S., PEROZZI, B., KAPOOR, A., ALIPOURFARD, N., LERMAN, K., HARUTYUNYAN, H., STEEG, G. V., AND GALSTYAN, A. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML (2019).
- ANDERSEN, R., CHUNG, F. R. K., AND LANG, K. J. Local graph partitioning using pagerank vectors. In FOCS (2006), IEEE Computer Society, pp. 475–486.
- BANERJEE, S., AND LOFGREN, P. Fast bidirectional probability estimation in markov models. In NIPS (2015), pp. 1423–1431.
- BOJCHEVSKI, A., KLICPERA, J., PEROZZI, B., KAPOOR, A., BLAIS, M., RÓZEMBERCZKI, B., LUKASIK, M., AND GÜNNEMANN, S. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY, USA, 2020), ACM.
- CHEN, J., MA, T., AND XIAO, C. Fastgcn: Fast learning with graph convolutional networks via importance sampling. In ICLR (2018).
- CHEN, J., ZHU, J., AND SONG, L. Stochastic training of graph convolutional networks with variance reduction. In ICML (2018).
- CHEN, Z., WEI, X., WANG, P., AND GUO, Y. Multi-label image recognition with graph convolutional networks. In CVPR (2019).
- CHIANG, W., LIU, X., SI, S., LI, Y., BENGIO, S., AND HSIEH, C. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In KDD (2019), ACM, pp. 257–266.
- CHUNG, F. R. K., AND LU, L. Survey: Concentration inequalities and martingale inequalities: A survey. Internet Math. 3, 1 (2006), 79–127.
- FOUT, A., BYRD, J., SHARIAT, B., AND BEN-HUR, A. Protein interface prediction using graph convolutional networks. In NeurIPS (2017), pp. 6530–6539.
- HAMILTON, W. L., YING, R., AND LESKOVEC, J. Inductive representation learning on large graphs. In NeurIPS (2017).
- HE, K., ZHANG, X., REN, S., AND SUN, J. Deep residual learning for image recognition. In CVPR (2016), pp. 770–778.
- HOSSEINI, B., MONTAGNE, R., AND HAMMER, B. Deep-aligned convolutional neural network for skeleton-based action recognition and segmentation. In Data Science and Engineering (2020).
- KARYPIS, G., AND KUMAR, V. A fast and high quality multilevel scheme for partitioning irregular graphs. SIAM J. Scientific Computing 20, 1 (1998), 359–392.
- KIPF, T. N., AND WELLING, M. Semi-supervised classification with graph convolutional networks. In ICLR (2017).
- KLICPERA, J., BOJCHEVSKI, A., AND GÜNNEMANN, S. Predict then propagate: Graph neural networks meet personalized pagerank. In ICLR (2019).
- KLICPERA, J., WEISSENBERGER, S., AND GÜNNEMANN, S. Diffusion improves graph learning. In NeurIPS (2019), pp. 13333–13345.
- KLOSTER, K., AND GLEICH, D. F. Heat kernel based community detection. In KDD (2014), ACM, pp. 1386–1395.
- LESKOVEC, J., LANG, K. J., AND MAHONEY, M. W. Empirical comparison of algorithms for network community detection. In WWW (2010), ACM, pp. 631–640.
- LI, C., AND GOLDWASSER, D. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In ACL (2019).
- LI, P., CHIEN, I. E., AND MILENKOVIC, O. Optimizing generalized pagerank methods for seed-expansion community detection. In NeurIPS (2019), H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., pp. 11705–11716.
- LU, H., HUANG, S. H., YE, T., AND GUO, X. Graph star net for generalized multi-task learning. CoRR abs/1906.12330 (2019).
- QIU, J., TANG, J., MA, H., DONG, Y., WANG, K., AND TANG, J. Deepinf: Social influence prediction with deep learning. In KDD (2018), ACM, pp. 2110–2119.
- RONG, Y., HUANG, W., XU, T., AND HUANG, J. Dropedge: Towards deep graph convolutional networks on node classification. In ICLR (2020).
- SEN, P., NAMATA, G., BILGIC, M., GETOOR, L., GALLAGHER, B., AND ELIASSI-RAD, T. Collective classification in network data. AI Magazine 29, 3 (2008), 93–106.
- SHANG, J., XIAO, C., MA, T., LI, H., AND SUN, J. Gamenet: Graph augmented memory networks for recommending medication combination. In AAAI (2019).
- THEKUMPARAMPIL, K. K., WANG, C., OH, S., AND LI, L. Attention-based graph neural network for semi-supervised learning. CoRR abs/1803.03735 (2018).
- TONG, P., ZHANG, Q., AND YAO, J. Leveraging domain context for question answering over knowledge graph. In Data Science and Engineering (2019).
- VELIC KOVIC, P., CUCURULL, G., CASANOVA, A., ROMERO, A., LIÒ, P., AND BENGIO, Y. Graph Attention Networks. ICLR (2018).
- WU, F., SOUZA, A., ZHANG, T., FIFTY, C., YU, T., AND WEINBERGER, K. Simplifying graph convolutional networks. In ICML (2019), pp. 6861–6871.
- XU, K., HU, W., LESKOVEC, J., AND JEGELKA, S. How powerful are graph neural networks? In ICLR (2019).
- XU, K., LI, C., TIAN, Y., SONOBE, T., KAWARABAYASHI, K., AND JEGELKA, S. Representation learning on graphs with jumping knowledge networks. In ICML (2018).
- YANG, C., XIAO, Y., ZHANG, Y., SUN, Y., AND HAN, J. Heterogeneous network representation learning: Survey, benchmark, evaluation, and beyond. arXiv preprint arXiv:2004.00216 (2020).
- YANG, J., AND LESKOVEC, J. Defining and evaluating network communities based on ground-truth. In ICDM (2012), IEEE Computer Society, pp. 745–754.
- YANG, R., XIAO, X., WEI, Z., BHOWMICK, S. S., ZHAO, J., AND LI, R. Efficient estimation of heat kernel pagerank for local clustering. In SIGMOD Conference (2019), ACM, pp. 1339– 1356.
- YING, R., HE, R., CHEN, K., EKSOMBATCHAI, P., HAMILTON, W. L., AND LESKOVEC, J. Graph convolutional neural networks for web-scale recommender systems. In KDD (2018), ACM, pp. 974–983.
- ZENG, H., ZHOU, H., SRIVASTAVA, A., KANNAN, R., AND PRASANNA, V. GraphSAINT: Graph sampling based inductive learning method. In ICLR (2020).
- ZHAO, L., AND AKOGLU, L. Pairnorm: Tackling oversmoothing in gnns. In ICLR (2020).
- ZHAO, L., PENG, X., TIAN, Y., KAPADIA, M., AND METAXAS, D. N. Semantic graph convolutional networks for 3d human pose regression. In CVPR (2019), pp. 3425–3435.
- ZOU, D., HU, Z., WANG, Y., JIANG, S., SUN, Y., AND GU, Q. Layer-dependent importance sampling for training deep and large graph convolutional networks. In NeurIPS (2019), pp. 11247–11256. (2) If t =, we have R( ) is decremented by R( )(u, k) · Iuk and Q( ) is incremented by R( )(u, k) · Iuk. Consequently, we have

标签

评论