AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
A new framework for 3D shape matching that is based on entropy regularized optimal transport

Deep Shells: Unsupervised Shape Correspondence with Optimal Transport

NIPS 2020, (2020)

Cited by: 0|Views131
EI
Full Text
Bibtex
Weibo

Abstract

We propose a novel unsupervised learning approach to 3D shape correspondence that builds a multiscale matching pipeline into a deep neural network. This approach is based on smooth shells, the current state-of-the-art axiomatic correspondence method, which requires an a priori stochastic search over the space of initial poses. Our goal ...More
0
Introduction
  • Computing shape correspondence is a fundamental component in understanding the 3D world, and is at the heart of many applications in computer vision and graphics.
  • One straightforward approach is to learn high-level features as local correlation patterns which are explicitly defined as intrinsic patch operators [22, 4, 25, 29, 35]
  • While these charting based techniques have proven to be useful for 3D shapes, they make rather strong assumptions about the structure of the data and are not trivially transferable to different domains like e.g. graphs or general manifolds.
  • A complementary approach is to compute a convolution of signals F : X → RL with some filter in the spectral space where it is a pointwise multiplication with learnable diagonal coefficient matrices Γl ,l [6]: L
Highlights
  • Computing shape correspondence is a fundamental component in understanding the 3D world, and is at the heart of many applications in computer vision and graphics
  • The unprecedented success of convolutional neural networks (CNNs) on tasks like image and natural language processing suggests that there is a big potential of devising similar architectures for nonEuclidean data
  • The inputs to our method are normalized to a fixed square root area of
  • The high frequency Laplacian eigenfunctions on differentiable manifolds are known to grow approximately linear with a constant incline depending on the total surface area
  • A new framework for 3D shape matching that is based on entropy regularized optimal transport
  • We show that the proposed unsupervised method significantly improves over the state-of-the-art on multiple datasets, even in comparison to the most recent supervised methods
  • While most prior learning methods on geometric domains use either extrinsic or intrinsic information to obtain correspondences from local features, our approach operates on both domains jointly in a hierarchical pipeline
Methods
  • Traditional approaches directly compute a matching P : X → Y between two input surfaces X and Y by making use of certain geometric properties and invariances of deformable shapes.
  • The most common choice of basis functions φi are the Laplace-Beltrami eigenfunctions which are provably optimal for representing smooth functions on a manifold [27]
  • This choice allows them to build certain assumptions about the input objects, like near-isometry or area preservation, directly into the representation [26].
  • The original framework has been extended to handle partial shapes [33, 20], compute refined point-to-point maps [32], to allow for orientation preserving maps [31] and to iteratively upsample a coarse map [23]
Results
  • Implementation details The authors implemented the network in PyTorch using Adam optimizer [17].
  • The inputs to the method are normalized to a fixed square root area of.
  • The authors' spectral convolution layer uses 120 filters on the frequency domain represented with 16 cosine basis functions each, see Eq (12).
  • The authors use 200 eigenfunctions for the truncated spectral filters from Eq (11).
  • The high frequency Laplacian eigenfunctions on differentiable manifolds are known to grow approximately linear with a constant incline depending on the total surface area.
Conclusion
  • A new framework for 3D shape matching that is based on entropy regularized optimal transport.
  • While most prior learning methods on geometric domains use either extrinsic or intrinsic information to obtain correspondences from local features, the approach operates on both domains jointly in a hierarchical pipeline.
  • This embedding fully acknowledges the geometric nature of Riemannian manifolds: It is agnostic to the discretization while using both the extrinsic and intrinsic surface geometry.
  • Besides the standard error on individual benchmarks, the method shows compelling generalization results across different datasets
Summary
  • Introduction:

    Computing shape correspondence is a fundamental component in understanding the 3D world, and is at the heart of many applications in computer vision and graphics.
  • One straightforward approach is to learn high-level features as local correlation patterns which are explicitly defined as intrinsic patch operators [22, 4, 25, 29, 35]
  • While these charting based techniques have proven to be useful for 3D shapes, they make rather strong assumptions about the structure of the data and are not trivially transferable to different domains like e.g. graphs or general manifolds.
  • A complementary approach is to compute a convolution of signals F : X → RL with some filter in the spectral space where it is a pointwise multiplication with learnable diagonal coefficient matrices Γl ,l [6]: L
  • Objectives:

    The authors' goal is to replace this costly preprocessing step by directly learning good initializations from the input surfaces.
  • Methods:

    Traditional approaches directly compute a matching P : X → Y between two input surfaces X and Y by making use of certain geometric properties and invariances of deformable shapes.
  • The most common choice of basis functions φi are the Laplace-Beltrami eigenfunctions which are provably optimal for representing smooth functions on a manifold [27]
  • This choice allows them to build certain assumptions about the input objects, like near-isometry or area preservation, directly into the representation [26].
  • The original framework has been extended to handle partial shapes [33, 20], compute refined point-to-point maps [32], to allow for orientation preserving maps [31] and to iteratively upsample a coarse map [23]
  • Results:

    Implementation details The authors implemented the network in PyTorch using Adam optimizer [17].
  • The inputs to the method are normalized to a fixed square root area of.
  • The authors' spectral convolution layer uses 120 filters on the frequency domain represented with 16 cosine basis functions each, see Eq (12).
  • The authors use 200 eigenfunctions for the truncated spectral filters from Eq (11).
  • The high frequency Laplacian eigenfunctions on differentiable manifolds are known to grow approximately linear with a constant incline depending on the total surface area.
  • Conclusion:

    A new framework for 3D shape matching that is based on entropy regularized optimal transport.
  • While most prior learning methods on geometric domains use either extrinsic or intrinsic information to obtain correspondences from local features, the approach operates on both domains jointly in a hierarchical pipeline.
  • This embedding fully acknowledges the geometric nature of Riemannian manifolds: It is agnostic to the discretization while using both the extrinsic and intrinsic surface geometry.
  • Besides the standard error on individual benchmarks, the method shows compelling generalization results across different datasets
Tables
  • Table1: A summary of our quantitative experiments. For each result, we show the mean geodesic error in % of the shape diameter. The table is subdivided into three sections with the current stateof-the-art axiomatic, supervised and unsupervised learning approaches. The odd columns show the results on the test set of FAUST remeshed trained on FAUST remeshed, SCAPE remeshed and both datasets respectively. Analogously, the results on SCAPE are in the even columns
  • Table2: A runtime comparison corresponding to the experiments in the first two columns of Table 1
Download tables as Excel
Funding
  • This work was supported by the Collaborative Research Center SFB-TRR 109 ’Discretization in Geometry and Dynamics’, the ERC Consolidator Grant ”3D Reloaded”, the Humboldt Foundation through the Sofja Kovalevskaja Award and the Helmholtz Association under the joint research school "Munich School for Data Science - MUDS"
Study subjects and analysis
pairs: 512
Therefore, the remeshed versions are an improvement over the classical FAUST and SCAPE datasets which contain templates with the same number of points and connectivity. We split both datasets into training sets of 80 and 51 shapes respectively and 20 test shapes each and randomly shuffle the 802 and 512 pairs during training. Although both FAUST and SCAPE contain humans, the

Reference
  • Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM transactions on graphics (TOG), volume 24, pages 408–416. ACM, 2005.
    Google ScholarLocate open access versionFindings
  • Matthieu Aubry, Ulrich Schlickewei, and Daniel Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. IEEE International Conference on Computer Vision (ICCV) Workshop on Dynamic Shape Capture and Analysis, 2011.
    Google ScholarLocate open access versionFindings
  • Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Piscataway, NJ, USA, June 2014. IEEE.
    Google ScholarLocate open access versionFindings
  • Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael Bronstein. Learning shape correspondence with anisotropic convolutional neural networks. In Advances in neural information processing systems, pages 3189–3197, 2016.
    Google ScholarLocate open access versionFindings
  • Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18–42, 2017.
    Google ScholarLocate open access versionFindings
  • Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
    Findings
  • Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in neural information processing systems, pages 2292–2300, 2013.
    Google ScholarLocate open access versionFindings
  • Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pages 3844– 3852, 2016.
    Google ScholarLocate open access versionFindings
  • Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Robust feature learning for shape correspondence. arXiv preprint arXiv:2003.14286, 2020.
    Findings
  • Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. Smooth shells: Multi-scale shape registration with functional maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12265–12274, 2020.
    Google ScholarLocate open access versionFindings
  • Joel Franklin and Jens Lorenz. On the scaling of multidimensional matrices. Linear Algebra and its applications, 114:717–735, 1989.
    Google ScholarLocate open access versionFindings
  • Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. 3d-coded: 3d correspondences by deep deformation. In The European Conference on Computer Vision (ECCV), September 2018.
    Google ScholarLocate open access versionFindings
  • Oshri Halimi, Or Litany, Emanuele Rodola, Alex M Bronstein, and Ron Kimmel. Unsupervised learning of dense shape correspondence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4370–4379, 2019.
    Google ScholarLocate open access versionFindings
  • Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks on graph-structured data. arxiv (2015). arXiv preprint arXiv:1506.05163, 2015.
    Findings
  • Kai Hormann and Günther Greiner. Mips: An efficient global parametrization method. Technical report, Erlangen-Nuernberg University (Germany) Computer Graphics Group, 2000.
    Google ScholarFindings
  • Vladimir G Kim, Yaron Lipman, and Thomas A Funkhouser. Blended intrinsic maps. Transactions on Graphics (TOG), 30(4), 2011.
    Google ScholarLocate open access versionFindings
  • Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
    Findings
  • Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
    Findings
  • Or Litany, Tal Remez, Emanuele Rodolà, Alex Bronstein, and Michael Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In Proceedings of the IEEE International Conference on Computer Vision, pages 5659–5667, 2017.
    Google ScholarLocate open access versionFindings
  • Or Litany, Emanuele Rodolà, Alex M Bronstein, Michael M Bronstein, and Daniel Cremers. Non-rigid puzzles. Computer Graphics Forum (CGF), Proceedings of Symposium on Geometry Processing (SGP), 35(5), 2016.
    Google ScholarLocate open access versionFindings
  • British Museum London. Scan of discobolus at the british museum. https://www.myminifactory.com/object/3d-print-discobolus-at-the-british-museum-london-7896, 2020. Accessed:202005-30.
    Findings
  • Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pages 37–45, 2015.
    Google ScholarLocate open access versionFindings
  • Simone Melzi, Jing Ren, Emanuele Rodolà, Abhishek Sharma, Peter Wonka, and Maks Ovsjanikov. Zoomout: Spectral upsampling for efficient shape correspondence. ACM Transactions on Graphics (TOG), 38(6):155, 2019.
    Google ScholarLocate open access versionFindings
  • Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with gumbel-sinkhorn networks. arXiv preprint arXiv:1802.08665, 2018.
    Findings
  • Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5115–5124, 2017.
    Google ScholarLocate open access versionFindings
  • Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Functional maps: a flexible representation of maps between shapes. ACM Transactions on Graphics (TOG), 31(4):30, 2012.
    Google ScholarLocate open access versionFindings
  • Beresford N Parlett. The symmetric eigenvalue problem, volume 20. siam, 1998.
    Google ScholarFindings
  • Ulrich Pinkall and Konrad Polthier. Computing discrete minimal surfaces and their conjugates. EXPERI- MENTAL MATHEMATICS, 2:15–36, 1993.
    Google ScholarLocate open access versionFindings
  • Adrien Poulenard and Maks Ovsjanikov. Multi-directional geodesic neural networks via equivariant convolution. ACM Transactions on Graphics (TOG), 37(6):1–14, 2018.
    Google ScholarLocate open access versionFindings
  • Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
    Google ScholarLocate open access versionFindings
  • Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov. Continuous and orientation-preserving correspondences via functional maps. ACM Trans. Graph., 37(6):248:1–248:16, December 2018.
    Google ScholarLocate open access versionFindings
  • Emanuele Rodolà, Michael Moeller, and Daniel Cremers. Regularized pointwise map recovery from functional correspondence. In Computer Graphics Forum, volume 36, pages 700–711. Wiley Online Library, 2017.
    Google ScholarLocate open access versionFindings
  • Emanuele Rodolà, Luca Cosmo, Michael Bronstein, Andrea Torsello, and Daniel Cremers. Partial functional correspondence. Computer Graphics Forum (CGF), 2016.
    Google ScholarFindings
  • Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov. Unsupervised deep learning for structured shape matching. In Proceedings of the IEEE International Conference on Computer Vision, pages 1617– 1627, 2019.
    Google ScholarLocate open access versionFindings
  • Klaus Hildebrandt Ruben Wiersma, Elmar Eisemann. Cnns on surfaces using rotation-equivariant features. Transactions on Graphics, 39(4), July 2020.
    Google ScholarLocate open access versionFindings
  • Yusuf Sahillioglu. Recent advances in shape correspondence. The Visual Computer, pages 1–17, 2019.
    Google ScholarLocate open access versionFindings
  • Joaquim Salvi, Carles Matabosch, David Fofi, and Josep Forest. A review of recent range image registration methods with accuracy evaluation. Image Vision Comput., 25(5):578–596, 2007.
    Google ScholarLocate open access versionFindings
  • Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4938–4947, 2020.
    Google ScholarLocate open access versionFindings
  • Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383–1392. Wiley Online Library, 2009.
    Google ScholarLocate open access versionFindings
  • Federico Tombari, Samuele Salti, and Luigi Di Stefano. Unique signatures of histograms for local surface description. In Proceedings of European Conference on Computer Vision (ECCV), 16(9):356–369, 2010.
    Google ScholarLocate open access versionFindings
  • Oliver van Kaick, Hao Zhang, Ghassan Hamarneh, and Daniel Cohen-Or. A survey on shape correspondence. Computer Graphics Forum, 30(6):1681–1707, 2011.
    Google ScholarLocate open access versionFindings
  • Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 109–117, 2017.
    Google ScholarLocate open access versionFindings
  • Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
    Google ScholarLocate open access versionFindings
  • Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11824–11833, 2020.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments
小科