Tight Query Complexity Lower Bounds for PCA via Finite Sample Deformed Wigner Law

We prove a query complexity lower bound for approximating the top r dimensional eigenspace of a matrix. We consider an oracle model where, given a symmetric matrix 𝐌∈ℝ^d × d, an algorithm 𝖠𝗅𝗀 is allowed to make 𝖳 exact queries of the form 𝗐^(i) = 𝐌𝗏^(i) for i in {1,...,𝖳}, where 𝗏^(i) is drawn from a distribution which depends arbitrarily on the past queries and measurements {𝗏^(j),𝗐^(i)}_1 ≤ j ≤ i-1. We show that for every 𝚐𝚊𝚙∈ (0,1/2], there exists a distribution over matrices 𝐌 for which 1) gap_r(𝐌) = Ω(𝚐𝚊𝚙) (where gap_r(𝐌) is the normalized gap between the r and r+1-st largest-magnitude eigenvector of 𝐌), and 2) any algorithm 𝖠𝗅𝗀 which takes fewer than const×r log d/√(𝚐𝚊𝚙) queries fails (with overwhelming probability) to identity a matrix 𝖵∈ℝ^d × r with orthonormal columns for which ⟨𝖵, 𝐌𝖵⟩≥ (1 - const×𝚐𝚊𝚙)∑_i=1^r λ_i(𝐌). Our bound requires only that d is a small polynomial in 1/𝚐𝚊𝚙 and r, and matches the upper bounds of Musco and Musco '15. Moreover, it establishes a strict separation between convex optimization and randomized, "strict-saddle" non-convex optimization of which PCA is a canonical example: in the former, first-order methods can have dimension-free iteration complexity, whereas in PCA, the iteration complexity of gradient-based methods must necessarily grow with the dimension.