Unsupervised Inductive Graph-Level Representation Learning via Graph-Graph Proximity

IJCAI, pp. 1988-1994, 2019.

Cited by: 6|Bibtex|Views97|DOI:https://doi.org/10.24963/ijcai.2019/275
EI
Other Links: academic.microsoft.com|dblp.uni-trier.de
Weibo:
We present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance

Abstract:

We introduce a novel approach to graph-level representation learning, which is to embed an entire graph into a vector space where the embeddings of two graphs preserve their graph-graph proximity. Our approach, UGraphEmb, is a general framework that provides a novel means to performing graph-level embedding in a completely unsupervised an...More

Code:

Data:

0
Introduction
  • There has been a rich body of work [Belkin and Niyogi, 2003; Tang et al, 2015; Qiu et al, 2018] on node-level embeddings that turn each node in a graph into a vector preserving node-node proximity
  • Most of these models are unsupervised and demonstrate superb performance in node classification and link prediction.
  • GRAPH2VEC is transductive, i.e. it does not naturally generalize to unseen graphs outside the training set
Highlights
  • Recent years we have witnessed the great popularity of graph representation learning with success in node-level tasks such as node classification [Kipf and Welling, 2016a] and link prediction [Zhang and Chen, 2018], and graphlevel tasks such as graph classification [Ying et al, 2018] and graph similarity/distance computation [Bai et al, 2019].

    There has been a rich body of work [Belkin and Niyogi, 2003; Tang et al, 2015; Qiu et al, 2018] on node-level embeddings that turn each node in a graph into a vector preserving node-node proximity
  • Inspired by the recent progress on graph proximity modeling [Ktena et al, 2017; Bai et al, 2019], we propose a novel framework, UGRAPHEMB ( Unsupervised Graph-level Embbedding) that employs multi-scale aggregations of nodelevel embeddings, guided by the graph-graph proximity defined by well-accepted and domain-agnostic graph proximity metrics such as Graph Edit Distance (GED) [Bunke, 1983], Maximum Common Subgraph (MCS) [Bunke and Shearer, 1998], etc
  • A simple aggregation of node embeddings without any learnable parameters limits the expressive power of existing graphlevel embedding models. To tackle both challenges in the graph embedding generation layer, we propose the following Multi-Scale Node Attention (MSNA) mechanism
  • UGRAPHEMB, against a number of state-of-the-art approaches designed for unsupervised node and graph embeddings, to answer the following questions: Q1 How superb are the graph-level embeddings generated by UGRAPHEMB, when evaluated with downstream tasks including graph classification and similarity ranking?
  • We present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance (GED)
  • Experiments show that the produced graph-level embeddings achieve competitive performance on three downstream tasks: graph classification, similarity ranking, and graph visualization
Methods
  • For (3) and (4), the authors try different averaging schemes on node embeddings to obtain the graph-level embeddings and report their best accuracy.
  • With finetuning (UGRAPHEMB-F), the model can achieve the best result on 4 out of 5 datasets.
  • Methods designed for graph-level embeddings (GRAPH KERNELS, GRAPH2VEC, and UGRAPHEMB) consistently outperform methods designed for node-level embeddings (NETMF and GRAPHSAGE), suggesting that good node-level embeddings do not naturally imply good graph-level representations.
  • The authors split it into training, validation, and testing sets by 6:2:2, and report the averaged Mean Squared Error, Kendall’s Rank Correlation Coefficient (τ ) [Kendall, 1938], and Precision at 10 (p@10) to test the ranking performance
Conclusion
  • The authors present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance (GED).
  • (a) GK (b) SP (c) WL (d) Graph2Vec (e) NetMF (g) UGraphEmb (i) UGraphEmb (f) GraphSAGE (h) UGraphEmb (j) UGraphEmb (k) UGraphEmb (l) UGraphEmb level embeddings is proposed.
  • Experiments show that the produced graph-level embeddings achieve competitive performance on three downstream tasks: graph classification, similarity ranking, and graph visualization
Summary
  • Introduction:

    There has been a rich body of work [Belkin and Niyogi, 2003; Tang et al, 2015; Qiu et al, 2018] on node-level embeddings that turn each node in a graph into a vector preserving node-node proximity
  • Most of these models are unsupervised and demonstrate superb performance in node classification and link prediction.
  • GRAPH2VEC is transductive, i.e. it does not naturally generalize to unseen graphs outside the training set
  • Objectives:

    Since the goal is to embed each graph as a single point in the embedding space that preserves graph-graph proximity, the graph embedding generation model should:.
  • Methods:

    For (3) and (4), the authors try different averaging schemes on node embeddings to obtain the graph-level embeddings and report their best accuracy.
  • With finetuning (UGRAPHEMB-F), the model can achieve the best result on 4 out of 5 datasets.
  • Methods designed for graph-level embeddings (GRAPH KERNELS, GRAPH2VEC, and UGRAPHEMB) consistently outperform methods designed for node-level embeddings (NETMF and GRAPHSAGE), suggesting that good node-level embeddings do not naturally imply good graph-level representations.
  • The authors split it into training, validation, and testing sets by 6:2:2, and report the averaged Mean Squared Error, Kendall’s Rank Correlation Coefficient (τ ) [Kendall, 1938], and Precision at 10 (p@10) to test the ranking performance
  • Conclusion:

    The authors present UGRAPHEMB, an end-to-end neural network based framework aiming to embed an entire graph into an embedding preserving the proximity between graphs in the dataset under a graph proximity metric, such as Graph Edit Distance (GED).
  • (a) GK (b) SP (c) WL (d) Graph2Vec (e) NetMF (g) UGraphEmb (i) UGraphEmb (f) GraphSAGE (h) UGraphEmb (j) UGraphEmb (k) UGraphEmb (l) UGraphEmb level embeddings is proposed.
  • Experiments show that the produced graph-level embeddings achieve competitive performance on three downstream tasks: graph classification, similarity ranking, and graph visualization
Tables
  • Table1: Graph classification accuracy in percent. “-” indicates that the computation did not finish after 72 hours. We highlight the top 2 accuracy in bold
  • Table2: Similarity ranking performance. BEAM, HUNGARIAN, and VJ are three approximate GED computation algorithms returning upper bounds of exact GEDs. We take the minimum GED computed by the three as ground-truth GEDs for training and evaluating all the methods on both Task 1 and 2. Their results are labeled with “∗”. HED is another GED solver yielding lower bounds. “-” indicates that the computation did not finish after 72 hours
Download tables as Excel
Related work
  • Unsupervised graph representation learning has a long history. Classic works including NETMF [Qiu et al, 2018], LINE [Tang et al, 2015], DeepWalk [Perozzi et al, 2014], etc., which typically generate an embedding for each node in one graph. Theoretical analysis shows that many of these works cannot handle embeddings for multiple graphs in the sense that the node embeddings in one graph are not comparable to those in another graph in any straightforward way [Heimann and Koutra, 2017]. A simple permutation of node indices could cause the node embedding to be very different.

    More recently, some of the methods based on Graph Convolutional Networks (GCN) [Defferrard et al, 2016; Kipf and Welling, 2016a], such as VGAE [Kipf and Welling, 2016b], satisfy the desired permutation-invariance property. Categorized as “graph autoencoders” [Wu et al, 2019], they also belong to the family of graph neural network methods. Although satisfying the permutation-invariance requirement, these autoencoders are still designed to generate unsuperised node embeddings.
Funding
  • This work is partially supported by NIH R01GM115833 and U01HG008488, NSF DBI-1565137, DGE-1829071, NSF III1705169, NSF CAREER Award 1741634, and Amazon Research Award
Reference
  • [Bai et al., 2019] Yunsheng Bai, Hao Ding, Song Bian, Ting Chen, Yizhou Sun, and Wei Wang. Simgnn: A neural network approach to fast graph similarity computation. WSDM, 2019.
    Google ScholarLocate open access versionFindings
  • [Belkin and Niyogi, 2003] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6):1373–1396, 2003.
    Google ScholarLocate open access versionFindings
  • [Bunke and Shearer, 1998] Horst Bunke and Kim Shearer. A graph distance metric based on the maximal common subgraph. Pattern recognition letters, 19(3-4):255–259, 1998.
    Google ScholarLocate open access versionFindings
  • [Bunke, 1983] H Bunke. What is the distance between graphs. Bulletin of the EATCS, 20:35–39, 1983.
    Google ScholarLocate open access versionFindings
  • [Defferrard et al., 2016] Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS, pages 3844–3852, 2016.
    Google ScholarLocate open access versionFindings
  • [Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
    Findings
  • [Fankhauser et al., 2011] Stefan Fankhauser, Kaspar Riesen, and Horst Bunke. Speeding up graph edit distance computation through fast bipartite matching. In International Workshop on Graph-Based Representations in Pattern Recognition, pages 102–111.
    Google ScholarLocate open access versionFindings
  • [Fischer et al., 2015] Andreas Fischer, Ching Y Suen, Volkmar Frinken, Kaspar Riesen, and Horst Bunke. Approximation of graph edit distance based on hausdorff matching. Pattern Recognition, 48(2):331–343, 2015.
    Google ScholarLocate open access versionFindings
  • [Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1024–1034, 2017.
    Google ScholarLocate open access versionFindings
  • [Heimann and Koutra, 2017] Mark Heimann and Danai Koutra. On generalizing neural node embedding methods to multi-network problems. In KDD MLG Workshop, 2017.
    Google ScholarLocate open access versionFindings
  • [Kendall, 1938] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81–93, 1938.
    Google ScholarLocate open access versionFindings
  • [Kipf and Welling, 2016a] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2016.
    Google ScholarLocate open access versionFindings
  • [Kipf and Welling, 2016b] Thomas N Kipf and Max Welling. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning, 2016.
    Google ScholarLocate open access versionFindings
  • [Ktena et al., 2017] Sofia Ira Ktena, Sarah Parisot, Enzo Ferrante, Martin Rajchl, Matthew Lee, Ben Glocker, and Daniel Rueckert. Distance metric learning using graph convolutional networks: Application to functional brain networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 469– 477.
    Google ScholarLocate open access versionFindings
  • [Liang and Zhao, 2017] Yongjiang Liang and Peixiang Zhao. Similarity search in graph databases: A multi-layered indexing approach. In ICDE, pages 783–794. IEEE, 2017.
    Google ScholarLocate open access versionFindings
  • [Ma et al., 2018] Tengfei Ma, Cao Xiao, Jiayu Zhou, and Fei Wang. Drug similarity integration through attentive multiview graph auto-encoders. IJCAI, 2018.
    Google ScholarLocate open access versionFindings
  • [Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605, 2008.
    Google ScholarLocate open access versionFindings
  • [Narayanan et al., 2017] Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. KDD MLG Workshop, 2017.
    Google ScholarFindings
  • [Neuhaus et al., 2006] Michel Neuhaus, Kaspar Riesen, and Horst Bunke. Fast suboptimal algorithms for the computation of graph edit distance. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition (SSPR), pages 163–172.
    Google ScholarLocate open access versionFindings
  • [Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In SIGKDD, pages 701–710. ACM, 2014.
    Google ScholarLocate open access versionFindings
  • [Peters et al., 2018] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. NAACL, 2018.
    Google ScholarLocate open access versionFindings
  • [Qiu et al., 2018] Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. Network embedding as matrix factorization: Unifyingdeepwalk, line, pte, and node2vec. WSDM, 2018.
    Google ScholarFindings
  • [Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018.
    Google ScholarFindings
  • [Riesen and Bunke, 2009] Kaspar Riesen and Horst Bunke. Approximate graph edit distance computation by means of bipartite graph matching. Image and Vision computing, 27(7):950–959, 2009.
    Google ScholarLocate open access versionFindings
  • [Tang et al., 2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Largescale information network embedding. In WWW, pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015.
    Google ScholarLocate open access versionFindings
  • [Velickovic et al., 2018] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ICLR, 2018.
    Google ScholarLocate open access versionFindings
  • [Williams, 2001] Christopher KI Williams. On a connection between kernel pca and metric multidimensional scaling. In Advances in neural information processing systems, pages 675–681, 2001.
    Google ScholarLocate open access versionFindings
  • [Wu et al., 2019] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596, 2019.
    Findings
  • [Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
    Google ScholarLocate open access versionFindings
  • [Yan et al., 2005] Xifeng Yan, Philip S Yu, and Jiawei Han. Substructure similarity search in graph databases. In SIGMOD, pages 766–777. ACM, 2005.
    Google ScholarLocate open access versionFindings
  • [Yanardag and Vishwanathan, 2015] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In SIGKDD, pages 1365–1374. ACM, 2015.
    Google ScholarLocate open access versionFindings
  • [Ying et al., 2018] Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. NeurIPS, 2018.
    Google ScholarLocate open access versionFindings
  • [Zhang and Chen, 2018] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In NeurIPS, pages 5171–5181, 2018.
    Google ScholarLocate open access versionFindings
  • [Zhang and Chen, 2019] Xinyi Zhang and Lihui Chen. Capsule graph neural network. ICLR, 2019.
    Google ScholarLocate open access versionFindings
Full Text
Your rating :
0

 

Tags
Comments