FORML: A Riemannian Hessian-free Method for Meta-learning on Stiefel Manifolds
arxiv(2024)
摘要
Meta-learning problem is usually formulated as a bi-level optimization in
which the task-specific and the meta-parameters are updated in the inner and
outer loops of optimization, respectively. However, performing the optimization
in the Riemannian space, where the parameters and meta-parameters are located
on Riemannian manifolds is computationally intensive. Unlike the Euclidean
methods, the Riemannian backpropagation needs computing the second-order
derivatives that include backward computations through the Riemannian operators
such as retraction and orthogonal projection. This paper introduces a
Hessian-free approach that uses a first-order approximation of derivatives on
the Stiefel manifold. Our method significantly reduces the computational load
and memory footprint. We show how using a Stiefel fully-connected layer that
enforces orthogonality constraint on the parameters of the last classification
layer as the head of the backbone network, strengthens the representation reuse
of the gradient-based meta-learning methods. Our experimental results across
various few-shot learning datasets, demonstrate the superiority of our proposed
method compared to the state-of-the-art methods, especially MAML, its Euclidean
counterpart.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要