Extreme Stochastic Variational Inference: Distributed and Asynchronous

We propose extreme stochastic variational inference (ESVI), which allows multiple processors to simultaneously and asynchronously perform variational inference updates. Moreover, by using a classic owner computes paradigm, our algorithm can be made lock-free. ESVI exhibits data and model parallelism, that is, each processor only needs access to a subset of the data and a subset of the parameters. In our experiments we show that our new algorithm outperforms a straightforward strategy for parallelizing variational inference, which requires bulk synchronization after every iteration.