On the Use of Low-discrepancy Sequences in the Training of Neural Networks.

The quasi-Monte Carlo methods use specially designed deterministic sequences with improved uniformity properties compared with random numbers, in order to achieve higher rates of convergence. Usually certain measures like the discrepancy are used in order to quantify these uniformity properties. The usefulness of certain families of sequences with low discrepancy, like the Sobol and Halton sequences, has been established in problems with high practical value as in Mathematical Finance. Multiple studies have been done about applying these sequences also in the domains of optimisation and machine learning. Currently many types of neural networks are used extensively to achieve break-through results in Machine Learning and Artificial Intelligence. The process of training these networks requires substantial computational resources, usually provided by using powerful GPUs or specially designed hardware. In this work we study different approaches to employ efficiently low-discrepancy sequences at various places in the training process where their uniformity properties can speed-up or improve the training process. We demonstrate the advantage of using Sobol low-discrepancy sequences in benchmark problems and we discuss various practical issues that arise in order to achieve acceptable performance in real-life problems.