On Convergence of Model Parallel Proximal Gradient Algorithm for Stale Synchronous Parallel System.

With ever growing data volume and model size, an error-tolerant, communication efficient, yet versatile parallel algorithm has become a vital part for the success of many large-scale applications. In this work we propose msPG, an extension of the flexible proximal gradient algorithm to the model parallel and stale synchronous setting. The worker machines of msPG operate asynchronously as long as they are not too far apart, and they communicate efficiently through a dedicated parameter server. Theoretically, we provide a rigorous analysis of the various convergence properties of msPG, and a salient feature of our analysis is its seamless generality that allows both nonsmooth and nonconvex functions. Under mild conditions, we prove the whole iterate sequence of msPG converges to a critical point (which is optimal under convexity assumptions). We further provide an economical implementation of msPG, completely bypassing the need of keeping a local full model. We confirm our theoretical findings through numerical experiments.