Hessian-Inverse-Free Stochastic Gradient Method for Stochastic Multi-Objective Bilevel Optimization

Stochastic multi-objective bilevel optimization (SMOBO) has emerged as a powerful framework for large-scale machine learning problems, where stochastic gradient methods are naturally employed for their simplicity and efficiency. However, existing stochastic gradient methods for SMOBO typically require computing the expensive inverse Hessian matrices, leading to high computational cost. To tackle these issues, we propose a fully Hessian-inverse-free stochastic gradient method for SMOBO, which is computationally efficient due to the use of a decoupling approach. Specifically, we employ a momentum-like gradient estimate to replace the stochastic gradients with their moving average, thereby mitigating the errors induced by dynamic weight updates. Without assuming bounded function values, we demonstrate that the proposed method achieves not only an optimal non-asymptotic convergence rate of 𝒪(K^-1/2) for SMOBO, but also an improved bias bound for the stochastic multi-gradient with a rate of 𝒪(K^-1/4) . Moreover, we derive an optimal sample complexity of 𝒪(ϵ ^-2) to find an ϵ -Pareto stationary point. Finally, preliminary numerical experiments validate the competitiveness of the proposed method.