## AI helps you reading Science

## AI Insight

AI extracts a summary of this paper

Weibo:

# Equivariance Through Parameter-Sharing.

ICML, (2017)

EI

Abstract

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group $\mathcal{G}$ that acts discretely on the input and output of a standard neural network layer $\phi_{W}: \Re^{M} \to \Re^{N}$, we show that $\phi_{W}$ is equivariant with respect to $\mathcal{G}$-action iff $\mathcal{G}$ e...More

Code:

Data:

Introduction

- The authors propose to study equivariance in deep neural networks through parameter symmetries.
- If the output is transformed, in a predictable way, as the authors transform the input, the neural layer is equivariant to the action of the group.
- The authors' goal is to show that parameter-sharing can be used to achieve equivariance to any discrete group action.

Highlights

- We propose to study equivariance in deep neural networks through parameter symmetries
- Our goal is to show that parameter-sharing can be used to achieve equivariance to any discrete group action
- This work is a step towards designing neural network layers with a given equivariance and invariance properties
- We proposed two parameter-sharing scheme that achieves equivariance wrt any discrete group-action
- It is essential to be able to draw the line between equivariance/invariance and sensitivity in a function
- Our work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions

Results

- Given a group G, and its discrete action through GN,M, the authors are interested in defining parameter-sharing schemes for a parametric class of functions that guarantees their unique GN,M-equivariance.
- The authors start by looking at a single neural layer and relate its unique GN,M-equivariance to the symmetries of a colored multi-edged bipartite graph that defines parameter-sharing.
- Using Theorem 3.3 of the section, the authors can prove that these six permutations are the “only” edge-color preserving ones for this structure, resulting in unique equivariance.
- The implication is that to achieve unique equivariance for a given group-action, the authors need to define the parametersharing using the structure Ω with symmetry group GN,M.
- Example 3.2 (Nested Subsets and Wreath Product) The permutation-equivariant layer that the authors saw in Example 2.2 is useful for defining neural layers for set structure.
- Recall The authors' objective is to define parameter-sharing so that φW ∶ RdD → RdD is equivariant to the action of G = Sd ≀ SD – i.e., permutations within sets at two levels.
- Assuming G-action is semi-regular on both N and M, using representatives {np}1≤p≤P and {mq}1≤q≤Q for orbits in N and M, the authors can rewrite the expression (5) of the structured neural layer for the structure above.
- Claim 3.5 Under the following conditions the neural layer (5) using the sparse design (10) performs group convolution: I) there is a bijection between the output and G (i.e., M = G) and; II) GN is transitive.
- This identifies the limitations of group-convolution even in the setting where M = G: When GN is semiregular and not transitive (P > 1), group convolution is not guaranteed to be uniquely equivariant while the sparse parameter-sharing of (10) provides this guarantee.
- Achieving unique Gequivariance reduces to answering the following question: when could the authors express a permutation group G ≤ SN as the symmetry group Aut(Ω) of a colored multi-edged digraph with N nodes?
- Example 3.7 (Graph Convolution) Consider the setting where the authors use the adjacency matrix B ∈ {0, 1}N×N of a graph Λ, to identify parameter-sharing in a neural network layer.

Conclusion

- This work is a step towards designing neural network layers with a given equivariance and invariance properties.
- The authors proposed two parameter-sharing scheme that achieves equivariance wrt any discrete group-action.
- The authors' work presents the first results of its kind on guarantees regarding both variance and equivariance with respect to group actions

Funding

- This research is supported in part by DOE grant DESC0011114 and NSF grant IIS1563887

Reference

- Agrawal, Pulkit, Carreira, Joao, and Malik, Jitendra. Learning to see by moving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 37– 45, 2015.
- Anselmi, Fabio, Leibo, Joel Z, Rosasco, Lorenzo, Mutch, Jim, Tacchetti, Andrea, and Poggio, Tomaso. Unsupervised learning of invariant representations in hierarchical architectures. arXiv preprint arXiv:1311.4158, 2013.
- Babai, Laszlo. Automorphism groups, isomorphism, reconstruction (chapter 27 of the handbook of combinatorics). University of Chicago, 1994.
- Bartok, Gabor, Szepesvari, Csaba, and Zilles, Sandra. Models of active learning in group-structured state spaces. Information and Computation, 208(4):364–384, 2010.
- Bruna, Joan and Mallat, Stephane. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013.
- Bruna, Joan, Zaremba, Wojciech, Szlam, Arthur, and LeCun, Yann. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
- Bui, Hung Hai, Huynh, Tuyen N, and Riedel, Sebastian. Automorphism groups of graphical models and lifted variational inference. arXiv preprint arXiv:1207.4814, 2012.
- Cohen, Taco S and Welling, Max. Group equivariant convolutional networks. arXiv preprint arXiv:1602.07576, 2016a.
- Cohen, Taco S and Welling, Max. Steerable cnns. arXiv preprint arXiv:1612.08498, 2016b.
- Dieleman, Sander, Willett, Kyle W, and Dambre, Joni. Rotation-invariant convolutional neural networks for galaxy morphology prediction. Monthly notices of the royal astronomical society, 450(2):1441–1459, 2015.
- Dieleman, Sander, De Fauw, Jeffrey, and Kavukcuoglu, Koray. Exploiting cyclic symmetry in convolutional neural networks. arXiv preprint arXiv:1602.02660, 2016.
- Freeman, William T, Adelson, Edward H, et al. The design and use of steerable filters. IEEE Transactions on Pattern analysis and machine intelligence, 13(9):891–906, 1991.
- Gens, Robert and Domingos, Pedro M. Deep symmetry networks. In Advances in neural information processing systems, pp. 2537–2545, 2014.
- Hel-Or, Yacov and Teo, Patrick C. A common framework for steerability, motion estimation, and invariant feature detection. In Circuits and Systems, 1998. ISCAS’98. Proceedings of the 1998 IEEE International Symposium on, volume 5, pp. 337–340. IEEE, 1998.
- Henaff, Mikael, Bruna, Joan, and LeCun, Yann. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163, 2015.
- Jaderberg, Max, Simonyan, Karen, Zisserman, Andrew, et al. Spatial transformer networks. In Advances in Neural Information Processing Systems, pp. 2017–2025, 2015.
- Jayaraman, Dinesh and Grauman, Kristen. Learning image representations equivariant to ego-motion. In Proc. ICCV, 2015.
- Kersting, Kristian, Ahmadi, Babak, and Natarajan, Sriraam. Counting belief propagation. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pp. 277–284. AUAI Press, 2009.
- Kipf, Thomas N and Welling, Max. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
- Kondor, Risi. Group theoretical methods in machine learning. Columbia University, 2008.
- Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
- Lenc, Karel and Vedaldi, Andrea. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991–999, 2015.
- Niepert, Mathias. Markov chains on orbits of permutation groups. arXiv preprint arXiv:1206.5396, 2012.
- http://colah.github.io/posts/
- 2014-12-Groups-Convolution/, 2014.
- Oyallon, Edouard and Mallat, Stephane. Deep rototranslation scattering for object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2865–2873, 2015.
- Raedt, Luc De, Kersting, Kristian, Natarajan, Sriraam, and Poole, David. Statistical relational artificial intelligence: Logic, probability, and computation. Synthesis Lectures on Artificial Intelligence and Machine Learning, 10(2): 1–189, 2016.
- Ravanbakhsh, Siamak, Schneider, Jeff, and Poczos, Barnabas. Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500, 2016.
- Sifre, Laurent and Mallat, Stephane. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1233–1240, 2013.
- Wielandt, H. Permutation groups through invariant relations and invariant functions. Dept. of Mathematics, Ohio State University, 1969.
- Zaheer, Manzil, Kottur, Satwik, Ravanbakhsh, Siamak, Poczos, Barnabas, Salakhutdinov, Ruslan, and Smola, Alexander J. Deep sets. CoRR, abs/1703.06114, 2017. URL http://arxiv.org/abs/1703.06114.

Tags

Comments

数据免责声明

页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果，我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问，可以通过电子邮件方式联系我们：report@aminer.cn