AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We propose a robust aggregation function, Soft Medoid, for the internal use within Graph Neural Networks

Reliable Graph Neural Networks via Robust Aggregation

NIPS 2020, (2020)

Cited by: 0|Views150
EI
Full Text
Bibtex
Weibo

Abstract

Perturbations targeting the graph structure have proven to be extremely effective in reducing the performance of Graph Neural Networks (GNNs), and traditional defenses such as adversarial training do not seem to be able to improve robustness. This work is motivated by the observation that adversarially injected edges effectively can be ...More

Code:

Data:

0
Introduction
  • Learning on graph data has gained strong attention in recent years, powered by the success of graph neural networks [29, 34].
  • While recent research suggests that effective defenses against attribute attacks can be found, e.g. robust training [59], defenses against structure attacks remain an unsolved topic [18, 54, 60]
  • Approaches such as [24, 52], solely focus on defending against specific attack characteristics.
Highlights
  • Learning on graph data has gained strong attention in recent years, powered by the success of graph neural networks [29, 34]
  • We propose a novel robust aggregation function for Graph Neural Networks (GNNs) to address this drawback
  • We report the perturbed accuracy for Dice [50], a FGSM-like [28] attack that greedily flips the element in A which contributes most to the test loss and Projected Gradient Descent (PGD) for L0 perturbations [54]
  • We propose a robust aggregation function, Soft Medoid, for the internal use within GNNs
  • We show that the Soft Medoid—a fully differentiable generalization of the Medoid—comes with the best possible breakdown point of 0.5 and an upper bound of the error/bias of the internal aggregations
  • Focusing on the negative aspects of contemporary applications, robust GNNs might cause, e.g., an increased automation bias [43], or fewer loopholes e.g. in the surveillance implemented in authoritarian systems [2]
Results
  • The authors' method improves the robustness of its base architecture w.r.t. structural perturbations by up to 550%, and outperforms previous state-of-the-art defenses.
  • The authors outperform all baseline and the other defenses [24, 52, 58] w.r.t. robustness against structural perturbations by a relative margin of up to 450% and for low-degree edges even 700%
Conclusion
  • The authors propose a robust aggregation function, Soft Medoid, for the internal use within GNNs.
  • This work is one step on the path towards the adversarial robustness of GNNs. all potential applications of GNNs could benefit.
  • Robust machine learning models certainly come with less opportunity of manipulation.
  • At some point, the discussion of risks and opportunities for AI [3, 11] and robust machine learning will converge.
  • Focusing on the negative aspects of contemporary applications, robust GNNs might cause, e.g., an increased automation bias [43], or fewer loopholes e.g. in the surveillance implemented in authoritarian systems [2]
Tables
  • Table1: Average duration (time cost in ms) of Weighted Soft Medoid tWSM(X, a) for one training epoch (over 200 epochs, preprocessthe AGGREGATION. Thus, for node v ing counts once). For the other defenses we used in layer l, X represents the stacked em- DeepRobust’s implementation. We report “-” for an beddings {hu(l−1)W(l), ∀ u ∈ N (v) ∪ OOM. We used one 2.20 GHz core and one GeForce v}, and a the weight vector consists of GTX 1080 Ti (11 Gb). For hyperparameters see § 5
  • Table2: Accumulated certifications (first to third data Graph diffusion. Node degrees in realcolumn) and average certifiable radii (fourth and fifth world graphs typically follow a powerdata column) for the different architectures (top two high- law distribution. Consequently, we lighted). In the last column we list the clean accuracy of must be able to deal with a large fraction the base classifier (binary node attributes)
  • Table3: Statistics of the largest connected component of the used datasets
  • Table4: Targeted attack in the same setup as the a surrogate GCN to perform the respective at- evasion Nettack attack [<a class="ref-link" id="c61" href="#r61">61</a>]. We report the average tack and adapt the adjacency matrix. We train margin and the failure rate of the attack (higher is the other models on the clean graph and only better)
  • Table5: Perturbed accuracy for the global attacks on Cora ML and Citeseer. Here ǫ denotes the fraction of edges perturbed (relative to the clean graph)
  • Table6: Summary of accumulated certifications and accuracy for the different architectures on Cora ML and Citeseer. We also report the accuracy of the base and smooth classifier (binary attr.)
Download tables as Excel
Related work
  • GNNs are an important class of deep neural networks, both from a scientific and application standpoint. Following the recent, trendsetting work in [29, 34], a vast number of approaches were proposed [1, 26, 35, 36, 49, 55]. A magnitude of adversarial attacks have been introduced [5, 18, 24, 42, 52, 54, 59, 61], pointing out their sensitivity regarding such attacks. Many of the proposed attacks directly propose an appropriate defense. We can classify the approaches into the categories of preprocessing [24, 52], robust training [54, 59], and modifications of the architecture [56, 58]. Perhaps the most similar approach, due to their statistical motivation, is RGCN [58].
Funding
  • Acknowledgments and Disclosure of Funding This research was supported by the German Research Foundation, Emmy Noether grant GU 1409/21, the German Federal Ministry of Education and Research (BMBF), grant no. 01IS18036B, and the Helmholtz Association under the joint research school “Munich School for Data Science - MUDS.” The authors of this work take full responsibilities for its content
Study subjects and analysis
samples: 50
Soft Medoid (T = 10) Soft Medoid (T = 50) Soft Medoid (T = 100). Proof Sketch Due to the orthogonal equivari- Figure 2: Empirical bias B(ǫ), for 50 samples ance we may choose tSM(X) = 0, without loss of generality. Let Xǫ be decomposable such from a centered (tSM(X) = 0) bivariate normal distribution. (a) shows the bias for a perturbation that Xǫ = Xǫ(clean) ∪ Xǫ(pert.)

datasets with error estimates: 3
For a complete comparison, we also report the accuracy of the base classifier. In § B.4, we report results on all three datasets with error estimates. Our Soft Medoid GDC architecture comes with a relative increase on the accumulated certifications of more than 200% w.r.t. adversarially added edges (most challenging case) for a wide range of baselines, alternative architectures, and defenses [24, 52, 58]

samples: 50
We show the output layer (l = 2) message passing step, i.e. the input of AGGREGATE(l), for adversarially added edges of an exemplary node v. The adversarial edges are obtained with a Nettack [61] evasion attack (at test time). For a two-dimensional visualization we used PCA on the weighted node embeddings Aswhw(l−1)W(l) of all edges (s, w) ∈ A, but solely plot v’s neighborhood. We show the aggregation for 17, 29, and 50 perturbations in figure (a) to (c), respectively. Empirical bias B(ǫ), for 50 samples ance we may choose tSM(X) = 0, without loss of generality. Let Xǫ be decomposable such from a centered (tSM(X) = 0) bivariate normal distribution. (a) shows the bias for a perturbation that Xǫ = Xǫ(clean) ∪ Xǫ(pert.). Clearly the worst- with norm 1000, and (b) 10. Influence of the temperature T on the accumulated certifications (solid) and accuracy of the base classifier (dashed)

Reference
  • S. Abu-El-Haija, A. Kapoor, B. Perozzi, and J. Lee. N-GCN: Multi-scale graph convolution for semi-supervised node classification. 35th Conference on Uncertainty in Artificial Intelligence, UAI 2019, 2019.
    Google ScholarLocate open access versionFindings
  • [3] B. Balaram, T. Greenham, and J. Leonard. Artificial Intelligence: Real Public Engagement. Technical report, RSA (Royal Society for the encouragement of Arts, Manufactures and Commerce), 2018.
    Google ScholarFindings
  • [4] A. Bojchevski and S. Günnemann. Deep Gaussian embedding of graphs: Unsupervised inductive learning via ranking. 6th International Conference on Learning Representations, ICLR 2018, pages 1–13, 2018.
    Google ScholarLocate open access versionFindings
  • [5] A. Bojchevski and S. Günnemann. Adversarial attacks on node embeddings via graph poisoning. 36th International Conference on Machine Learning, ICML 2019, 2019-June:1112–1123, 2019.
    Google ScholarLocate open access versionFindings
  • [6] A. Bojchevski and S. Günnemann. Certifiable Robustness to Graph Perturbations. Neural Information Processing Systems, NeurIPS, 2019.
    Google ScholarLocate open access versionFindings
  • [7] A. Bojchevski, J. Klicpera, and S. Günnemann. Efficient Robustness Certificates for Graph Neural Networks via Sparsity-Aware Randomized Smoothing (see URL for an anonymous preprint of the accepted paper). 37th International Conference on Machine Learning, ICML 2020, 2020.
    Google ScholarLocate open access versionFindings
  • [8] K. Boudt, P. J. Rousseeuw, S. Vanduffel, and T. Verdonck. The minimum regularized covariance determinant estimator. Statistics and Computing, 30(1):113–128, 2020.
    Google ScholarLocate open access versionFindings
  • [9] M. Badoiu, S. Har-Peled, and P. Indyk. Approximate clustering via core-sets. Annual ACM Symposium on Theory of Computing, pages 250–257, 2002.
    Google ScholarLocate open access versionFindings
  • [10] N. Carlini and D. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. ACM Workshop on Artificial Intelligence and Security, AISec 2017, pages 3–14, 2017.
    Google ScholarLocate open access versionFindings
  • [11] S. Cave, R. Nyrup, K. Vold, and A. Weller. Motivations and Risks of Machine Ethics. Proceedings of the IEEE, 107(3):562–574, mar 2019.
    Google ScholarLocate open access versionFindings
  • [12] R. Chandrasekaran and A. Tamir. Open questions concerning Weiszfeld’s algorithm for the Fermat-Weber location problem. Mathematical Programming, 1989.
    Google ScholarLocate open access versionFindings
  • [13] H. H. Chin, A. Madry, G. Miller, and R. Peng. Runtime Guarantees for Regression Problems. ITCS 2013 - 2013 ACM Conference on Innovations in Theoretical Computer Science, 2011.
    Google ScholarLocate open access versionFindings
  • [14] J. Cohen, E. Rosenfeld, and J. Z. Kolter. Certified adversarial robustness via randomized smoothing. 36th International Conference on Machine Learning, ICML 2019, pages 2323– 2356, 2019.
    Google ScholarLocate open access versionFindings
  • [15] M. B. Cohen, Y. T. Lee, G. Miller, J. Pachocki, and A. Sidford. Geometric median in nearly linear time. Annual ACM Symposium on Theory of Computing, 19-21-June(1):9–21, 2016.
    Google ScholarLocate open access versionFindings
  • [16] C. Croux, G. Haesbroeck, and P. J. Rousseeuw. Location adjustment for the minimum volume ellipsoid estimator. Statistics and Computing, 12(3):191–200, 2002.
    Google ScholarLocate open access versionFindings
  • [17] M. Cuturi, O. Teboul, and J.-P. Vert. Differentiable Ranks and Sorting using Optimal Transport. Neural Information Processing Systems, NeurIPS, 2019.
    Google ScholarLocate open access versionFindings
  • [18] H. Dai, H. Li, T. Tian, H. Xin, L. Wang, Z. Jun, and S. Le. Adversarial attack on graph structured data. 35th International Conference on Machine Learning, ICML 2018, 3:1799– 1808, 2018.
    Google ScholarLocate open access versionFindings
  • [19] P. L. Davies. Asymptotic Behaviour of $S$-Estimates of Multivariate Location Parameters and Dispersion Matrices. The Annals of Statistics, 1987.
    Google ScholarLocate open access versionFindings
  • [20] I. Diakonikolas and D. M. Kane. Recent Advances in Algorithmic High-Dimensional Robust Statistics. arXiv preprint arXiv:1911.05911, 2019.
    Findings
  • [21] I. Diakonikolas, G. Kamath, D. M. Kane, J. Li, A. Moitra, and A. Stewart. Being robust (in high dimensions) can be practical. 34th International Conference on Machine Learning, ICML 2017, 3:1659–1689, 2017.
    Google ScholarLocate open access versionFindings
  • [22] D. Donoho and P. J. Huber. The notion of breakdown point. In A Festschrift For Erich L. Lehmann, pages 157–184. Wadsworth Statist./Probab. Ser., Wadsworth, Belmont, CA, 1983, 1983.
    Google ScholarFindings
  • [23] D. L. Donoho and M. Gasko. Breakdown Properties of Location Estimates Based on Halfspace Depth and Projected Outlyingness. The Annals of Statistics, 1992.
    Google ScholarLocate open access versionFindings
  • [24] N. Entezari, S. A. Al-Sayouri, A. Darvishzadeh, and E. E. Papalexakis. All you need is Low (rank): Defending against adversarial attacks on graphs. International Conference on Web Search and Data Mining, WSDM 2020, pages 169–177, 2020.
    Google ScholarLocate open access versionFindings
  • [25] D. Feldman and M. Langberg. A unified framework for approximating and clustering data. Proceedings of the Symposium on Theory of Computing, pages 569–578, 2011.
    Google ScholarLocate open access versionFindings
  • [26] H. Gao and S. Ji. Graph U-nets. 36th International Conference on Machine Learning, ICML 2019, 2019-June:3651–3660, 2019.
    Google ScholarLocate open access versionFindings
  • [27] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. 34th International Conference on Machine Learning, ICML 2017, 3: 2053–2070, 2017.
    Google ScholarLocate open access versionFindings
  • [28] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. 3rd International Conference on Learning Representations, ICLR 2015, pages 1–11, 2015.
    Google ScholarFindings
  • [29] W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, 2017.
    Google ScholarLocate open access versionFindings
  • [30] S. Har-Peled and A. Kushal. Smaller Coresets for k -Median and k -Means Clustering. Discrete Comput Geom, 37:3–19, 2007.
    Google ScholarLocate open access versionFindings
  • [31] M. Hein and M. Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial Manipulation. In Advances in Neural Information Processing Systems, 2017.
    Google ScholarLocate open access versionFindings
  • [32] P. J. Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 1964.
    Google ScholarLocate open access versionFindings
  • [33] Indyk Piotr. High-Dimensional Computational Geometry. PhD thesis, Stanford University, 2001.
    Google ScholarFindings
  • [34] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. 5th International Conference on Learning Representations, ICLR 2017, pages 1–14, 2017.
    Google ScholarLocate open access versionFindings
  • [35] J. Klicpera, A. Bojchevski, and S. Günnemann. Predict then propagate: Graph neural networks meet personalized PageRank. 7th International Conference on Learning Representations, ICLR 2019, pages 1–15, 2019.
    Google ScholarLocate open access versionFindings
  • [36] J. Klicpera, S. Weißenberger, and S. Günnemann. Diffusion Improves Graph Learning. Neural Information Processing Systems, NeurIPS, 2019.
    Google ScholarLocate open access versionFindings
  • [37] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy, 2019.
    Google ScholarLocate open access versionFindings
  • [38] B. Li, C. Chen, W. Wang, and L. Carin. Certified Adversarial Robustness with Additive Noise. Neural Information Processing Systems, NeurIPS, 2018.
    Google ScholarLocate open access versionFindings
  • [39] H. Lopuhaä and P. Rousseeuw. Breakdown Points of Affine Equivariant Estimators of Multivariate Location and Covariance Matrices. Annals of Statistics, 19, 1991.
    Google ScholarLocate open access versionFindings
  • [40] R. A. Maronna. Robust $M$-Estimators of Multivariate Location and Scatter. The Annals of Statistics, 1976.
    Google ScholarLocate open access versionFindings
  • [41] A. K. McCallum, K. Nigam, J. Rennie, and K. Seymore. Automating the construction of internet portals with machine learning. Information Retrieval, 2000.
    Google ScholarLocate open access versionFindings
  • [42] B. A. Miller, M. Çamurcu, A. J. Gomez, K. Chan, and T. Eliassi-Rad. Topological Effects on Attacks Against Vertex Classification. arXiv preprint arXiv:2003.05822, 2020.
    Findings
  • [43] K. L. Mosier, L. J. Skitka, S. Heers, and M. Burdick. Automation bias: Decision making and performance in high-tech cockpits. International Journal of Aviation Psychology, 1998.
    Google ScholarLocate open access versionFindings
  • [44] J. Newling and F. Fleuret. A sub-quadratic exact medoid algorithm. International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 54, 2017.
    Google ScholarLocate open access versionFindings
  • [45] P. Parrilo and B. Sturmfels. Minimizing polynomial functions. DIMACS Workshop on Algorithmic and Quantitative Aspects of Real Algebraic Geometry in Mathematics and Computer Science, 0000:83–100, 2001.
    Google ScholarLocate open access versionFindings
  • [46] P. J. Rousseeuw. Least Median of Squares Regression. Journal of the American Statistical Association, 1984.
    Google ScholarLocate open access versionFindings
  • [47] P. Sen, G. M. Namata, M. Bilgic, L. Getoor, B. Gallagher, and T. Eliassi-Rad. Collective classification in network data. AI Magazine, 2008.
    Google ScholarLocate open access versionFindings
  • [48] J. W. Tukey. A Survey of Sampling from Contaminated Distributions. Contributions to Probability and Statistics Essays in Honor of Harold Hotelling, 1960.
    Google ScholarFindings
  • [49] P. Velickovic, A. Casanova, P. Liò, G. Cucurull, A. Romero, and Y. Bengio. Graph attention networks. 6th International Conference on Learning Representations, ICLR 2018, pages 1–12, 2018.
    Google ScholarLocate open access versionFindings
  • [50] M. Waniek, T. P. Michalak, M. J. Wooldridge, and T. Rahwan. Hiding individuals and communities in a social network. Nature Human Behaviour, 2(2):139–147, 2018.
    Google ScholarLocate open access versionFindings
  • [51] E. Wong and J. Z. Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In 35th International Conference on Machine Learning, ICML 2018, 2018.
    Google ScholarLocate open access versionFindings
  • [52] H. Wu, C. Wang, Y. Tyshetskiy, A. Docherty, K. Lu, and L. Zhu. Adversarial examples for graph data: Deep insights into attack and defense. IJCAI International Joint Conference on Artificial Intelligence, 2019-Augus:4816–4823, 2019.
    Google ScholarLocate open access versionFindings
  • [53] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, S. Member, and P. S. Yu. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
    Google ScholarLocate open access versionFindings
  • [54] K. Xu, H. Chen, S. Liu, P. Y. Chen, T. W. Weng, M. Hong, and X. Lin. Topology attack and defense for graph neural networks: An optimization perspective. IJCAI International Joint Conference on Artificial Intelligence, 2019-Augus:3961–3967, 2019.
    Google ScholarLocate open access versionFindings
  • [55] K. Xu, S. Jegelka, W. Hu, and J. Leskovec. How powerful are graph neural networks? 7th International Conference on Learning Representations, ICLR 2019, pages 1–17, 2019.
    Google ScholarLocate open access versionFindings
  • [56] Y. Zhang, S. Pal, M. Coates, and D. Ustebay. Bayesian Graph Convolutional Neural Networks for Semi-Supervised Classification. AAAI Conference on Artificial Intelligence, 33:5829–5836, 2019.
    Google ScholarLocate open access versionFindings
  • [57] J. Zhou, G. Cui, Z. Zhang, C. Yang, Z. Liu, L. Wang, and C. Li. Graph Neural Networks: A Review of Methods and Applications. arXiv preprint arXiv:1812.08434, 2018.
    Findings
  • [58] D. Zhu, P. Cui, Z. Zhang, and W. Zhu. Robust graph convolutional networks against adversarial attacks. International Conference on Knowledge Discovery and Data Mining, KDD, pages 1399–1407, 2019.
    Google ScholarLocate open access versionFindings
  • [59] D. Zügner and S. Günnemann. Adversarial attacks on graph neural networks via meta learning. 7th International Conference on Learning Representations, ICLR 2019, pages 1–15, 2019.
    Google ScholarLocate open access versionFindings
  • [60] D. Zügner and S. Günnemann. Certifiable Robustness of Graph Convolutional Networks under Structure Perturbations. International Conference on Knowledge Discovery and Data Mining, KDD, pages 1656–1665, 2020.
    Google ScholarLocate open access versionFindings
  • [61] D. Zügner, A. Akbarnejad, and S. Günnemann. Adversarial attacks on neural networks for graph data. International Conference on Knowledge Discovery and Data Mining, KDD, pages 2847–2856, 2018.
    Google ScholarLocate open access versionFindings
  • [41] PubMed [47]
    Google ScholarFindings
Author
Simon Geisler
Simon Geisler
Daniel Zügner
Daniel Zügner
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科