Efficient Supernet Training Using Path Parallelism

Compared to conventional neural networks, training a supernet for Neural Architecture Search (NAS) is very time consuming. Although current works have demonstrated that parallel computing can significantly speed up the training process, almost all of their parallelism still follow the conventional data- and model-based paradigms, which actually face performance issues in both computation and inter-node communication of the supernet training. To further improve the performance of current methods, we discover the unique path-parallelism that exists in supernet training, and proposed a novel training approach designed specifically for supernet. In detail, we focus on analyzing path correlations between subnets in a supernet and exploiting effective path-merging methods to reduce redundant computations and communications raised by concurrent subnets. Moreover, we also try to combine the proposed path parallelism with traditional intra-subnet parallelism to perform multi-level parallelization to further optimize the parallel performance. We present the detailed design and implementation of our method, and our experimental results show that our proposed approach can achieve up to 3.2x end-to-end speedup over conventional parallel training solutions, and 1.46x–5.78x speedup compared to the state-of-art supernet training frameworks.