Faster SVD-truncated regularized least-squares

We develop a fast algorithm for computing the “SVD-truncated” regularized solution to the least-squares problem: minx ∥Ax - b∥2. Let Ak of rank k be the best rank k matrix computed via the SVD of A. Then, the SVD-truncated regularized solution is: xk = Ak†b. If A is m × n, then, it takes O(mnmin{m, n}) time to compute xk using the SVD of A. We give an approximation algorithm for xk which constructs a rank k approximation Ãk and computes x̃k = Ãk† in roughly O(nnz(A)k log n) time. Our algorithm uses a randomized variant of the subspace iteration method. We show that, with high probability: ∥Ax̃k - b∥2 ≈ ∥Axk - b∥2 and ∥xk - x̃k∥2 ≈ 0.