AI helps you reading Science
AI generates interpretation videos
AI extracts and analyses the key points of the paper to generate videos automatically
AI parses the academic lineage of this thesis
AI extracts a summary of this paper
We perform two sets of experiments: experiments to show that ACR-Graph neural networks can learn a very simple FOC2 node classifier that AC-Graph neural networks cannot learn, and experiments involving complex FOC2 classifiers that need more intermediate readouts to be learned
The Logical Expressiveness of Graph Neural Networks
The ability of graph neural networks (GNNs) for distinguishing nodes in graphs has been recently characterized in terms of the Weisfeiler-Lehman (WL) test for checking graph isomorphism. This characterization, however, does not settle the issue of which Boolean node classifiers (i.e., functions classifying nodes in graphs as true or fals...More
PPT (Upload PPT)
- Graph neural networks (GNNs) (Merkwirth & Lengauer, 2005; Scarselli et al, 2009) are a class of neural network architectures that has recently become popular for a wide range of applications dealing with structured data, e.g., molecule classification, knowledge graph completion, and Web page ranking (Battaglia et al, 2018; Gilmer et al, 2017; Kipf & Welling, 2017; Schlichtkrull et al, 2018).
- One such FOC2 classifier is γ(x) in Equation (4), but there are infinitely many and even simpler FOC2 formulas that cannot be captured by AC-GNNs. Intuitively, the main problem is that an ACGNN has only a fixed number L of layers and the information of local aggregations cannot travel further than at distance L of every node along edges in the graph.
- Graph neural networks (GNNs) (Merkwirth & Lengauer, 2005; Scarselli et al, 2009) are a class of neural network architectures that has recently become popular for a wide range of applications dealing with structured data, e.g., molecule classification, knowledge graph completion, and Web page ranking (Battaglia et al, 2018; Gilmer et al, 2017; Kipf & Welling, 2017; Schlichtkrull et al, 2018)
- We prove that each FOC2 formula can be captured by an ACR-Graph neural networks
- We show that on synthetic graph data conforming to FOC2 formulas, ACGNNs struggle to fit the training data while ACR-Graph neural networks can generalize even to graphs of sizes not seen during training
- We perform two sets of experiments: experiments to show that ACR-Graph neural networks can learn a very simple FOC2 node classifier that AC-Graph neural networks cannot learn, and experiments involving complex FOC2 classifiers that need more intermediate readouts to be learned
- We report results on a real benchmark (PPI) where we did not observe an improvement of ACR-Graph neural networks over AC-Graph neural networks
- Separating AC-Graph neural networks and ACR-Graph neural networks We consider a very simple FOC2 formula defined by α(x) := Red(x) ∧ ∃y Blue(y), which is satisfied by every red node in a graph provided that the graph contains at least one blue node
- To see how a readout function could help in capturing non-local properties, consider again the logical classifier γ(x) in Equation (4), that assigns true to every red node v as long as there is another node not connected with v having two blue neighbors.
- The construction is similar to that of Proposition 4.1 and uses simple, homogeneous ACR-GNNs— that is, the readout function is just the sum of all the local node feature vectors.
- The authors leave as a challenging open problem whether FOC2 classifiers are exactly the logical classifiers captured by ACR-GNNs. 5.3 COMPARING THE NUMBER OF READOUT LAYERS
- The proof of Theorem 5.1 constructs GNNs whose number of layers depends on the formula being captured—that is, readout functions are used unboundedly many times in ACR-GNNs for capturing different FOC2 classifiers.
- It is based on a refinement of the GIN architecture proposed by Xu et al (2019) to obtain as much information as possible about the local neighborhood in graphs, followed by a readout and combine functions that use this information to deal with non-local constructs in formulas.
- The authors report results on a real benchmark (PPI) where the authors did not observe an improvement of ACR-GNNs over AC-GNNs. Separating AC-GNNs and ACR-GNNs The authors consider a very simple FOC2 formula defined by α(x) := Red(x) ∧ ∃y Blue(y), which is satisfied by every red node in a graph provided that the graph contains at least one blue node.
- This combined with the fact that random graphs that are more dense make the maximum distances between nodes shorter, may explain the boost in performance for AC-GNNs. Complex FOC2 properties In the second experiment the authors consider classifiers αi(x) constructed as α0(x) := Blue(x), αi+1(x) := ∃[N,M]y αi(y) ∧ ¬E(x, y) , (6)
- The authors' work is close in spirit to that of Xu et al (2019) and Morris et al (2019) establishing the correspondence between the WL test and GNNs. In contrast to the work, they focus on graph classification and do not consider the relationship with logical classifiers.
- Table1: Results on synthetic data for nodes labeled by classifier α(x) := Red(x) ∧ ∃y Blue(y)
- Table2: Results on E-R synthetic data for nodes labeled by classifiers αi(x) in Equation (6)
- Table3: Synthetic data for the experiment with classifier α(x) := Red(x) ∧ ∃y Blue(y)
- Table4: Detailed results for Erdos-Renyi synthetic graphs with different connectivities
- Table5: Synthetic data for the experiment with classifier αi(x) in Equation (6)
- Table6: Performance of AC-GNN and ACR-GNN in the PPI benchmark
- This work was partly funded by the Millennium Institute for Foundational Research on Data2
- Franz Baader and Carsten Lutz. Description logic. In Handbook of modal logic, pp. 757–819. North-Holland, 2007.
- Franz Baader, Diego Calvanese, Deborah L. McGuinness, Daniele Nardi, and Peter F. PatelSchneider (eds.). The description logic handbook: theory, implementation, and applications. Cambridge University Press, 2003.
- Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinıcius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/1806.01261, 2018. URL http://arxiv.org/abs/1806.01261.
- Jin-Yi Cai, Martin Furer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4):389–410, 1992.
- Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? A dissection on graph classification. CoRR, abs/1905.04579, 2019. URL https://arxiv.org/abs/1905.04579.
- Maarten de Rijke. A Note on graded modal logic. Studia Logica, 64(2):271–283, 2000.
- Haowen Deng, Tolga Birdal, and Slobodan Ilic. PPFnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18–22, 2018, pp. 195–205, 2018.
- Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with PyTorch Geometric. CoRR, abs/1903.02428, 2019. URL https://arxiv.org/abs/1903.02428.
- Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August, 2017, pp. 1263–1272, 2017.
- William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, NIPS 2017, Long Beach, CA, USA, December 4–9, 2017, pp. 1024–1034, 2017.
- Lu Haonan, Seth H Huang, Tian Ye, and Guo Xiuyan. Graph star net for generalized multi-task learning. arXiv preprint arXiv:1906.12330, 2019.
- Lauri Hella, Matti Jarvisalo, Antti Kuusisto, Juhana Laurinharju, Tuomo Lempiainen, Kerkko Luosto, Jukka Suomela, and Jonni Virtema. Weak models of distributed computing, with connections to modal logic. Distributed Computing, 28(1):31–53, 2015.
- Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, 2017.
- Anurag Koul, Sam Greydanus, and Alan Fern. Learning finite state representations of recurrent policy networks. In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019, 2019.
- Carsten Lutz, Ulrike Sattler, and Frank Wolter. Modal logic and the two-variable fragment. In Proceedings of the International Workshop on Computer Science Logic, CSL 2001, Paris, France, September 10–13, 2001, pp. 247–261.
- Christian Merkwirth and Thomas Lengauer. Automatic generation of complementary descriptors with molecular graph networks. J. of Chemical Information and Modeling, 45(5):1159–1168, 2005.
- Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and Leman go neural: higher-order graph neural networks. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, Hawaii, USA, January 27 – February 1, 2019, pp. 4602–4609, 2019.
- Boris Motik, Bernardo Cuenca Grau, Ian Horrocks, Zhe Wu, Achille Fokoue, and Carsten Lutz. OWL 2 Web ontology language profiles (second edition). W3C recommendation, W3C, 2012. URL http://www.w3.org/TR/owl2-profiles/.
- Christian Oliva and Luis F. Lago-Fernandez. On the interpretation of recurrent neural networks as finite state machines. In Part I of the Proceedings of the 28th International Conference on Artificial Neural Networks, ICANN 2019, Munich, Germany, September 17–19, 2019, pp. 312– 323.
- Martin Otto. Graded modal logic and counting bisimulation. https://www2.mathematik.tu-darmstadt.de/̃otto/papers/cml19.pdf, 2019.
- Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Approximation Ratios of Graph Neural Networks for Combinatorial Problems. arXiv preprint arXiv:1905.10261, 2019.
- Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Trans. Neural Networks, 20(1):61–80, 2009.
- Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In Proceedings of The Semantic Web - 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, pp. 593–607, 2018.
- W3C OWL Working Group. OWL 2 Web ontology language document overview (second edition). W3C recommendation, W3C, 2012. URL https://www.w3.org/TR/owl2-overview/.
- Boris Yu. Weisfeiler and Andrei A. Leman. A Reduction of a graph to a canonical form and an algebra arising during this reduction. Nauchno-Technicheskaya Informatsia, 2(9):12–16, 1968. Translated from Russian.
- Gail Weiss, Yoav Goldberg, and Eran Yahav. Extracting automata from recurrent neural networks using queries and counterexamples. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10–15, 2018, pp. 5244–5253, 2018.
- Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are graph neural networks? In Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6–9, 2019, 2019.
- Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, California, USA, June 9–15, 2019, pp. 7134–7143, 2019.
- Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. CoRR, abs/1707.04638, 2017. URL http://arxiv.org/abs/1707.04638.