# Unsupervised Inductive Graph-Level Representation Learning via Graph-Graph Proximity

IJCAI, pp. 1988-1994, 2019.

EI

Weibo:

Abstract:

We introduce a novel approach to graph-level representation learning, which is to embed an entire graph into a vector space where the embeddings of two graphs preserve their graph-graph proximity. Our approach, UGraphEmb, is a general framework that provides a novel means to performing graph-level embedding in a completely unsupervised an...More

Code:

Data:

Introduction

- There has been a rich body of work [Belkin and Niyogi, 2003; Tang et al, 2015; Qiu et al, 2018] on node-level embeddings that turn each node in a graph into a vector preserving node-node proximity
- Most of these models are unsupervised and demonstrate superb performance in node classification and link prediction.
- GRAPH2VEC is transductive, i.e. it does not naturally generalize to unseen graphs outside the training set

Highlights

- Recent years we have witnessed the great popularity of graph representation learning with success in node-level tasks such as node classification [Kipf and Welling, 2016a] and link prediction [Zhang and Chen, 2018], and graphlevel tasks such as graph classification [Ying et al, 2018] and graph similarity/distance computation [Bai et al, 2019].

There has been a rich body of work [Belkin and Niyogi, 2003; Tang et al, 2015; Qiu et al, 2018] on node-level embeddings that turn each node in a graph into a vector preserving node-node proximity - Inspired by the recent progress on graph proximity modeling [Ktena et al, 2017; Bai et al, 2019], we propose a novel framework, UGRAPHEMB ( Unsupervised Graph-level Embbedding) that employs multi-scale aggregations of nodelevel embeddings, guided by the graph-graph proximity defined by well-accepted and domain-agnostic graph proximity metrics such as Graph Edit Distance (GED) [Bunke, 1983], Maximum Common Subgraph (MCS) [Bunke and Shearer, 1998], etc
- A simple aggregation of node embeddings without any learnable parameters limits the expressive power of existing graphlevel embedding models. To tackle both challenges in the graph embedding generation layer, we propose the following Multi-Scale Node Attention (MSNA) mechanism
- UGRAPHEMB, against a number of state-of-the-art approaches designed for unsupervised node and graph embeddings, to answer the following questions: Q1 How superb are the graph-level embeddings generated by UGRAPHEMB, when evaluated with downstream tasks including graph classification and similarity ranking?
- We present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance (GED)
- Experiments show that the produced graph-level embeddings achieve competitive performance on three downstream tasks: graph classification, similarity ranking, and graph visualization

Methods

- For (3) and (4), the authors try different averaging schemes on node embeddings to obtain the graph-level embeddings and report their best accuracy.
- With finetuning (UGRAPHEMB-F), the model can achieve the best result on 4 out of 5 datasets.
- Methods designed for graph-level embeddings (GRAPH KERNELS, GRAPH2VEC, and UGRAPHEMB) consistently outperform methods designed for node-level embeddings (NETMF and GRAPHSAGE), suggesting that good node-level embeddings do not naturally imply good graph-level representations.
- The authors split it into training, validation, and testing sets by 6:2:2, and report the averaged Mean Squared Error, Kendall’s Rank Correlation Coefficient (τ ) [Kendall, 1938], and Precision at 10 (p@10) to test the ranking performance

Conclusion

- The authors present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance (GED).
- (a) GK (b) SP (c) WL (d) Graph2Vec (e) NetMF (g) UGraphEmb (i) UGraphEmb (f) GraphSAGE (h) UGraphEmb (j) UGraphEmb (k) UGraphEmb (l) UGraphEmb level embeddings is proposed.
- Experiments show that the produced graph-level embeddings achieve competitive performance on three downstream tasks: graph classification, similarity ranking, and graph visualization

Summary

## Introduction:

There has been a rich body of work [Belkin and Niyogi, 2003; Tang et al, 2015; Qiu et al, 2018] on node-level embeddings that turn each node in a graph into a vector preserving node-node proximity- Most of these models are unsupervised and demonstrate superb performance in node classification and link prediction.
- GRAPH2VEC is transductive, i.e. it does not naturally generalize to unseen graphs outside the training set
## Objectives:

Since the goal is to embed each graph as a single point in the embedding space that preserves graph-graph proximity, the graph embedding generation model should:.## Methods:

For (3) and (4), the authors try different averaging schemes on node embeddings to obtain the graph-level embeddings and report their best accuracy.- With finetuning (UGRAPHEMB-F), the model can achieve the best result on 4 out of 5 datasets.
- Methods designed for graph-level embeddings (GRAPH KERNELS, GRAPH2VEC, and UGRAPHEMB) consistently outperform methods designed for node-level embeddings (NETMF and GRAPHSAGE), suggesting that good node-level embeddings do not naturally imply good graph-level representations.
- The authors split it into training, validation, and testing sets by 6:2:2, and report the averaged Mean Squared Error, Kendall’s Rank Correlation Coefficient (τ ) [Kendall, 1938], and Precision at 10 (p@10) to test the ranking performance
## Conclusion:

The authors present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance (GED).- (a) GK (b) SP (c) WL (d) Graph2Vec (e) NetMF (g) UGraphEmb (i) UGraphEmb (f) GraphSAGE (h) UGraphEmb (j) UGraphEmb (k) UGraphEmb (l) UGraphEmb level embeddings is proposed.
- Experiments show that the produced graph-level embeddings achieve competitive performance on three downstream tasks: graph classification, similarity ranking, and graph visualization

- Table1: Graph classification accuracy in percent. “-” indicates that the computation did not finish after 72 hours. We highlight the top 2 accuracy in bold
- Table2: Similarity ranking performance. BEAM, HUNGARIAN, and VJ are three approximate GED computation algorithms returning upper bounds of exact GEDs. We take the minimum GED computed by the three as ground-truth GEDs for training and evaluating all the methods on both Task 1 and 2. Their results are labeled with “∗”. HED is another GED solver yielding lower bounds. “-” indicates that the computation did not finish after 72 hours

Related work

- Unsupervised graph representation learning has a long history. Classic works including NETMF [Qiu et al, 2018], LINE [Tang et al, 2015], DeepWalk [Perozzi et al, 2014], etc., which typically generate an embedding for each node in one graph. Theoretical analysis shows that many of these works cannot handle embeddings for multiple graphs in the sense that the node embeddings in one graph are not comparable to those in another graph in any straightforward way [Heimann and Koutra, 2017]. A simple permutation of node indices could cause the node embedding to be very different.

More recently, some of the methods based on Graph Convolutional Networks (GCN) [Defferrard et al, 2016; Kipf and Welling, 2016a], such as VGAE [Kipf and Welling, 2016b], satisfy the desired permutation-invariance property. Categorized as “graph autoencoders” [Wu et al, 2019], they also belong to the family of graph neural network methods. Although satisfying the permutation-invariance requirement, these autoencoders are still designed to generate unsuperised node embeddings.

Funding

- This work is partially supported by NIH R01GM115833 and U01HG008488, NSF DBI-1565137, DGE-1829071, NSF III1705169, NSF CAREER Award 1741634, and Amazon Research Award

Reference

- [Bai et al., 2019] Yunsheng Bai, Hao Ding, Song Bian, Ting Chen, Yizhou Sun, and Wei Wang. Simgnn: A neural network approach to fast graph similarity computation. WSDM, 2019.
- [Belkin and Niyogi, 2003] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
- [Bunke and Shearer, 1998] Horst Bunke and Kim Shearer. A graph distance metric based on the maximal common subgraph. Pattern recognition letters, 19(3-4):255–259, 1998.
- [Bunke, 1983] H Bunke. What is the distance between graphs. Bulletin of the EATCS, 20:35–39, 1983.
- [Defferrard et al., 2016] Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS, pages 3844–3852, 2016.
- [Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- [Fankhauser et al., 2011] Stefan Fankhauser, Kaspar Riesen, and Horst Bunke. Speeding up graph edit distance computation through fast bipartite matching. In International Workshop on Graph-Based Representations in Pattern Recognition, pages 102–111.
- [Fischer et al., 2015] Andreas Fischer, Ching Y Suen, Volkmar Frinken, Kaspar Riesen, and Horst Bunke. Approximation of graph edit distance based on hausdorff matching. Pattern Recognition, 48(2):331–343, 2015.
- [Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1024–1034, 2017.
- [Heimann and Koutra, 2017] Mark Heimann and Danai Koutra. On generalizing neural node embedding methods to multi-network problems. In KDD MLG Workshop, 2017.
- [Kendall, 1938] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81–93, 1938.
- [Kipf and Welling, 2016a] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2016.
- [Kipf and Welling, 2016b] Thomas N Kipf and Max Welling. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning, 2016.
- [Ktena et al., 2017] Sofia Ira Ktena, Sarah Parisot, Enzo Ferrante, Martin Rajchl, Matthew Lee, Ben Glocker, and Daniel Rueckert. Distance metric learning using graph convolutional networks: Application to functional brain networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 469– 477.
- [Liang and Zhao, 2017] Yongjiang Liang and Peixiang Zhao. Similarity search in graph databases: A multi-layered indexing approach. In ICDE, pages 783–794. IEEE, 2017.
- [Ma et al., 2018] Tengfei Ma, Cao Xiao, Jiayu Zhou, and Fei Wang. Drug similarity integration through attentive multiview graph auto-encoders. IJCAI, 2018.
- [Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605, 2008.
- [Narayanan et al., 2017] Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. KDD MLG Workshop, 2017.
- [Neuhaus et al., 2006] Michel Neuhaus, Kaspar Riesen, and Horst Bunke. Fast suboptimal algorithms for the computation of graph edit distance. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 163–172.
- [Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In SIGKDD, pages 701–710. ACM, 2014.
- [Peters et al., 2018] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. NAACL, 2018.
- [Qiu et al., 2018] Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifyingdeepwalk, line, pte, and node2vec. WSDM, 2018.
- [Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
- [Riesen and Bunke, 2009] Kaspar Riesen and Horst Bunke. Approximate graph edit distance computation by means of bipartite graph matching. Image and Vision computing, 27(7):950–959, 2009.
- [Tang et al., 2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Largescale information network embedding. In WWW, pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015.
- [Velickovic et al., 2018] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ICLR, 2018.
- [Williams, 2001] Christopher KI Williams. On a connection between kernel pca and metric multidimensional scaling. In Advances in neural information processing systems, pages 675–681, 2001.
- [Wu et al., 2019] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596, 2019.
- [Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
- [Yan et al., 2005] Xifeng Yan, Philip S Yu, and Jiawei Han. Substructure similarity search in graph databases. In SIGMOD, pages 766–777. ACM, 2005.
- [Yanardag and Vishwanathan, 2015] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In SIGKDD, pages 1365–1374. ACM, 2015.
- [Ying et al., 2018] Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. NeurIPS, 2018.
- [Zhang and Chen, 2018] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In NeurIPS, pages 5171–5181, 2018.
- [Zhang and Chen, 2019] Xinyi Zhang and Lihui Chen. Capsule graph neural network. ICLR, 2019.

Full Text

Tags

Comments