AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
Our work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions

Equivariance Through Parameter-Sharing.

ICML, (2017)

Cited: 110|Views66
EI
Full Text
Bibtex
Weibo

Abstract

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group $\mathcal{G}$ that acts discretely on the input and output of a standard neural network layer $\phi_{W}: \Re^{M} \to \Re^{N}$, we show that $\phi_{W}$ is equivariant with respect to $\mathcal{G}$-action iff $\mathcal{G}$ e...More

Code:

Data:

0
Introduction
  • The authors propose to study equivariance in deep neural networks through parameter symmetries.
  • If the output is transformed, in a predictable way, as the authors transform the input, the neural layer is equivariant to the action of the group.
  • The authors' goal is to show that parameter-sharing can be used to achieve equivariance to any discrete group action.
Highlights
  • We propose to study equivariance in deep neural networks through parameter symmetries
  • Our goal is to show that parameter-sharing can be used to achieve equivariance to any discrete group action
  • This work is a step towards designing neural network layers with a given equivariance and invariance properties
  • We proposed two parameter-sharing scheme that achieves equivariance wrt any discrete group-action
  • It is essential to be able to draw the line between equivariance/invariance and sensitivity in a function
  • Our work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions
Results
  • Given a group G, and its discrete action through GN,M, the authors are interested in defining parameter-sharing schemes for a parametric class of functions that guarantees their unique GN,M-equivariance.
  • The authors start by looking at a single neural layer and relate its unique GN,M-equivariance to the symmetries of a colored multi-edged bipartite graph that defines parameter-sharing.
  • Using Theorem 3.3 of the section, the authors can prove that these six permutations are the “only” edge-color preserving ones for this structure, resulting in unique equivariance.
  • The implication is that to achieve unique equivariance for a given group-action, the authors need to define the parametersharing using the structure Ω with symmetry group GN,M.
  • Example 3.2 (Nested Subsets and Wreath Product) The permutation-equivariant layer that the authors saw in Example 2.2 is useful for defining neural layers for set structure.
  • Recall The authors' objective is to define parameter-sharing so that φW ∶ RdD → RdD is equivariant to the action of G = Sd ≀ SD – i.e., permutations within sets at two levels.
  • Assuming G-action is semi-regular on both N and M, using representatives {np}1≤p≤P and {mq}1≤q≤Q for orbits in N and M, the authors can rewrite the expression (5) of the structured neural layer for the structure above.
  • Claim 3.5 Under the following conditions the neural layer (5) using the sparse design (10) performs group convolution: I) there is a bijection between the output and G (i.e., M = G) and; II) GN is transitive.
  • This identifies the limitations of group-convolution even in the setting where M = G: When GN is semiregular and not transitive (P > 1), group convolution is not guaranteed to be uniquely equivariant while the sparse parameter-sharing of (10) provides this guarantee.
  • Achieving unique Gequivariance reduces to answering the following question: when could the authors express a permutation group G ≤ SN as the symmetry group Aut(Ω) of a colored multi-edged digraph with N nodes?
  • Example 3.7 (Graph Convolution) Consider the setting where the authors use the adjacency matrix B ∈ {0, 1}N×N of a graph Λ, to identify parameter-sharing in a neural network layer.
Conclusion
  • This work is a step towards designing neural network layers with a given equivariance and invariance properties.
  • The authors proposed two parameter-sharing scheme that achieves equivariance wrt any discrete group-action.
  • The authors' work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions
Funding
  • This research is supported in part by DOE grant DESC0011114 and NSF grant IIS1563887
Reference
  • Agrawal, Pulkit, Carreira, Joao, and Malik, Jitendra. Learning to see by moving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 37– 45, 2015.
    Google ScholarLocate open access versionFindings
  • Anselmi, Fabio, Leibo, Joel Z, Rosasco, Lorenzo, Mutch, Jim, Tacchetti, Andrea, and Poggio, Tomaso. Unsupervised learning of invariant representations in hierarchical architectures. arXiv preprint arXiv:1311.4158, 2013.
    Findings
  • Babai, Laszlo. Automorphism groups, isomorphism, reconstruction (chapter 27 of the handbook of combinatorics). University of Chicago, 1994.
    Google ScholarFindings
  • Bartok, Gabor, Szepesvari, Csaba, and Zilles, Sandra. Models of active learning in group-structured state spaces. Information and Computation, 208(4):364–384, 2010.
    Google ScholarLocate open access versionFindings
  • Bruna, Joan and Mallat, Stephane. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013.
    Google ScholarLocate open access versionFindings
  • Bruna, Joan, Zaremba, Wojciech, Szlam, Arthur, and LeCun, Yann. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
    Findings
  • Bui, Hung Hai, Huynh, Tuyen N, and Riedel, Sebastian. Automorphism groups of graphical models and lifted variational inference. arXiv preprint arXiv:1207.4814, 2012.
    Findings
  • Cohen, Taco S and Welling, Max. Group equivariant convolutional networks. arXiv preprint arXiv:1602.07576, 2016a.
    Findings
  • Cohen, Taco S and Welling, Max. Steerable cnns. arXiv preprint arXiv:1612.08498, 2016b.
    Findings
  • Dieleman, Sander, Willett, Kyle W, and Dambre, Joni. Rotation-invariant convolutional neural networks for galaxy morphology prediction. Monthly notices of the royal astronomical society, 450(2):1441–1459, 2015.
    Google ScholarLocate open access versionFindings
  • Dieleman, Sander, De Fauw, Jeffrey, and Kavukcuoglu, Koray. Exploiting cyclic symmetry in convolutional neural networks. arXiv preprint arXiv:1602.02660, 2016.
    Findings
  • Freeman, William T, Adelson, Edward H, et al. The design and use of steerable filters. IEEE Transactions on Pattern analysis and machine intelligence, 13(9):891–906, 1991.
    Google ScholarLocate open access versionFindings
  • Gens, Robert and Domingos, Pedro M. Deep symmetry networks. In Advances in neural information processing systems, pp. 2537–2545, 2014.
    Google ScholarLocate open access versionFindings
  • Hel-Or, Yacov and Teo, Patrick C. A common framework for steerability, motion estimation, and invariant feature detection. In Circuits and Systems, 1998. ISCAS’98. Proceedings of the 1998 IEEE International Symposium on, volume 5, pp. 337–340. IEEE, 1998.
    Google ScholarLocate open access versionFindings
  • Henaff, Mikael, Bruna, Joan, and LeCun, Yann. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.
    Findings
  • Jaderberg, Max, Simonyan, Karen, Zisserman, Andrew, et al. Spatial transformer networks. In Advances in Neural Information Processing Systems, pp. 2017–2025, 2015.
    Google ScholarLocate open access versionFindings
  • Jayaraman, Dinesh and Grauman, Kristen. Learning image representations equivariant to ego-motion. In Proc. ICCV, 2015.
    Google ScholarLocate open access versionFindings
  • Kersting, Kristian, Ahmadi, Babak, and Natarajan, Sriraam. Counting belief propagation. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 277–284. AUAI Press, 2009.
    Google ScholarLocate open access versionFindings
  • Kipf, Thomas N and Welling, Max. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
    Findings
  • Kondor, Risi. Group theoretical methods in machine learning. Columbia University, 2008.
    Google ScholarFindings
  • Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
    Google ScholarLocate open access versionFindings
  • Lenc, Karel and Vedaldi, Andrea. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991–999, 2015.
    Google ScholarLocate open access versionFindings
  • Niepert, Mathias. Markov chains on orbits of permutation groups. arXiv preprint arXiv:1206.5396, 2012.
    Findings
  • http://colah.github.io/posts/
    Findings
  • 2014-12-Groups-Convolution/, 2014.
    Google ScholarFindings
  • Oyallon, Edouard and Mallat, Stephane. Deep rototranslation scattering for object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2865–2873, 2015.
    Google ScholarLocate open access versionFindings
  • Raedt, Luc De, Kersting, Kristian, Natarajan, Sriraam, and Poole, David. Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 10(2): 1–189, 2016.
    Google ScholarLocate open access versionFindings
  • Ravanbakhsh, Siamak, Schneider, Jeff, and Poczos, Barnabas. Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500, 2016.
    Findings
  • Sifre, Laurent and Mallat, Stephane. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1233–1240, 2013.
    Google ScholarLocate open access versionFindings
  • Wielandt, H. Permutation groups through invariant relations and invariant functions. Dept. of Mathematics, Ohio State University, 1969.
    Google ScholarLocate open access versionFindings
  • Zaheer, Manzil, Kottur, Satwik, Ravanbakhsh, Siamak, Poczos, Barnabas, Salakhutdinov, Ruslan, and Smola, Alexander J. Deep sets. CoRR, abs/1703.06114, 2017. URL http://arxiv.org/abs/1703.06114.
    Findings
0
Your rating :

No Ratings

Tags
Comments
avatar
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn