## AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically

Go Generating

## AI Traceability

AI parses the academic lineage of this thesis

Generate MRT

## AI Insight

AI extracts a summary of this paper

Weibo:
Our work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions

# Equivariance Through Parameter-Sharing.

ICML, (2017)

Cited: 110|Views66
EI
Full Text
Bibtex
Weibo

Abstract

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group $\mathcal{G}$ that acts discretely on the input and output of a standard neural network layer $\phi_{W}: \Re^{M} \to \Re^{N}$, we show that $\phi_{W}$ is equivariant with respect to $\mathcal{G}$-action iff $\mathcal{G}$ e...More

Code:

Data:

0
Introduction
• The authors propose to study equivariance in deep neural networks through parameter symmetries.
• If the output is transformed, in a predictable way, as the authors transform the input, the neural layer is equivariant to the action of the group.
• The authors' goal is to show that parameter-sharing can be used to achieve equivariance to any discrete group action.
Highlights
• We propose to study equivariance in deep neural networks through parameter symmetries
• Our goal is to show that parameter-sharing can be used to achieve equivariance to any discrete group action
• This work is a step towards designing neural network layers with a given equivariance and invariance properties
• We proposed two parameter-sharing scheme that achieves equivariance wrt any discrete group-action
• It is essential to be able to draw the line between equivariance/invariance and sensitivity in a function
• Our work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions
Results
• Given a group G, and its discrete action through GN,M, the authors are interested in defining parameter-sharing schemes for a parametric class of functions that guarantees their unique GN,M-equivariance.
• The authors start by looking at a single neural layer and relate its unique GN,M-equivariance to the symmetries of a colored multi-edged bipartite graph that defines parameter-sharing.
• Using Theorem 3.3 of the section, the authors can prove that these six permutations are the “only” edge-color preserving ones for this structure, resulting in unique equivariance.
• The implication is that to achieve unique equivariance for a given group-action, the authors need to define the parametersharing using the structure Ω with symmetry group GN,M.
• Example 3.2 (Nested Subsets and Wreath Product) The permutation-equivariant layer that the authors saw in Example 2.2 is useful for defining neural layers for set structure.
• Recall The authors' objective is to define parameter-sharing so that φW ∶ RdD → RdD is equivariant to the action of G = Sd ≀ SD – i.e., permutations within sets at two levels.
• Assuming G-action is semi-regular on both N and M, using representatives {np}1≤p≤P and {mq}1≤q≤Q for orbits in N and M, the authors can rewrite the expression (5) of the structured neural layer for the structure above.
• Claim 3.5 Under the following conditions the neural layer (5) using the sparse design (10) performs group convolution: I) there is a bijection between the output and G (i.e., M = G) and; II) GN is transitive.
• This identifies the limitations of group-convolution even in the setting where M = G: When GN is semiregular and not transitive (P > 1), group convolution is not guaranteed to be uniquely equivariant while the sparse parameter-sharing of (10) provides this guarantee.
• Achieving unique Gequivariance reduces to answering the following question: when could the authors express a permutation group G ≤ SN as the symmetry group Aut(Ω) of a colored multi-edged digraph with N nodes?
• Example 3.7 (Graph Convolution) Consider the setting where the authors use the adjacency matrix B ∈ {0, 1}N×N of a graph Λ, to identify parameter-sharing in a neural network layer.
Conclusion
• This work is a step towards designing neural network layers with a given equivariance and invariance properties.
• The authors proposed two parameter-sharing scheme that achieves equivariance wrt any discrete group-action.
• The authors' work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions
Funding
• This research is supported in part by DOE grant DESC0011114 and NSF grant IIS1563887
Reference
• Agrawal, Pulkit, Carreira, Joao, and Malik, Jitendra. Learning to see by moving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 37– 45, 2015.
• Anselmi, Fabio, Leibo, Joel Z, Rosasco, Lorenzo, Mutch, Jim, Tacchetti, Andrea, and Poggio, Tomaso. Unsupervised learning of invariant representations in hierarchical architectures. arXiv preprint arXiv:1311.4158, 2013.
• Babai, Laszlo. Automorphism groups, isomorphism, reconstruction (chapter 27 of the handbook of combinatorics). University of Chicago, 1994.
• Bartok, Gabor, Szepesvari, Csaba, and Zilles, Sandra. Models of active learning in group-structured state spaces. Information and Computation, 208(4):364–384, 2010.
• Bruna, Joan and Mallat, Stephane. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013.
• Bruna, Joan, Zaremba, Wojciech, Szlam, Arthur, and LeCun, Yann. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
• Bui, Hung Hai, Huynh, Tuyen N, and Riedel, Sebastian. Automorphism groups of graphical models and lifted variational inference. arXiv preprint arXiv:1207.4814, 2012.
• Cohen, Taco S and Welling, Max. Group equivariant convolutional networks. arXiv preprint arXiv:1602.07576, 2016a.
• Cohen, Taco S and Welling, Max. Steerable cnns. arXiv preprint arXiv:1612.08498, 2016b.
• Dieleman, Sander, Willett, Kyle W, and Dambre, Joni. Rotation-invariant convolutional neural networks for galaxy morphology prediction. Monthly notices of the royal astronomical society, 450(2):1441–1459, 2015.
• Dieleman, Sander, De Fauw, Jeffrey, and Kavukcuoglu, Koray. Exploiting cyclic symmetry in convolutional neural networks. arXiv preprint arXiv:1602.02660, 2016.
• Freeman, William T, Adelson, Edward H, et al. The design and use of steerable filters. IEEE Transactions on Pattern analysis and machine intelligence, 13(9):891–906, 1991.
• Gens, Robert and Domingos, Pedro M. Deep symmetry networks. In Advances in neural information processing systems, pp. 2537–2545, 2014.
• Hel-Or, Yacov and Teo, Patrick C. A common framework for steerability, motion estimation, and invariant feature detection. In Circuits and Systems, 1998. ISCAS’98. Proceedings of the 1998 IEEE International Symposium on, volume 5, pp. 337–340. IEEE, 1998.
• Henaff, Mikael, Bruna, Joan, and LeCun, Yann. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.
• Jaderberg, Max, Simonyan, Karen, Zisserman, Andrew, et al. Spatial transformer networks. In Advances in Neural Information Processing Systems, pp. 2017–2025, 2015.
• Jayaraman, Dinesh and Grauman, Kristen. Learning image representations equivariant to ego-motion. In Proc. ICCV, 2015.
• Kersting, Kristian, Ahmadi, Babak, and Natarajan, Sriraam. Counting belief propagation. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 277–284. AUAI Press, 2009.
• Kipf, Thomas N and Welling, Max. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
• Kondor, Risi. Group theoretical methods in machine learning. Columbia University, 2008.
• Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
• Lenc, Karel and Vedaldi, Andrea. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991–999, 2015.
• Niepert, Mathias. Markov chains on orbits of permutation groups. arXiv preprint arXiv:1206.5396, 2012.
• http://colah.github.io/posts/
• 2014-12-Groups-Convolution/, 2014.
• Oyallon, Edouard and Mallat, Stephane. Deep rototranslation scattering for object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2865–2873, 2015.
• Raedt, Luc De, Kersting, Kristian, Natarajan, Sriraam, and Poole, David. Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 10(2): 1–189, 2016.
• Ravanbakhsh, Siamak, Schneider, Jeff, and Poczos, Barnabas. Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500, 2016.
• Sifre, Laurent and Mallat, Stephane. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1233–1240, 2013.
• Wielandt, H. Permutation groups through invariant relations and invariant functions. Dept. of Mathematics, Ohio State University, 1969.
• Zaheer, Manzil, Kottur, Satwik, Ravanbakhsh, Siamak, Poczos, Barnabas, Salakhutdinov, Ruslan, and Smola, Alexander J. Deep sets. CoRR, abs/1703.06114, 2017. URL http://arxiv.org/abs/1703.06114.
Author
0