Understanding and Enhancing Dynamic Sparse Training Through the Lens of Stochasticity

Training deep neural networks (DNNs) under high sparsity remains challenging, as model performance often degrades significantly once sparsity exceeds a critical threshold. Dynamic Sparse Training (DST) has emerged as an effective paradigm that dynamically evolves network connectivity during training, enabling sparse models to achieve better performance than static sparse training. Despite its empirical success, the optimization mechanisms underlying DST remain insufficiently understood. In this work, we investigate the training dynamics of DST and provide empirical evidence that connectivity evolution is associated with transient stochastic perturbations in the optimization trajectory. These perturbations are consistent with enhanced exploration of the loss landscape and may help sparse models move away from sharp or suboptimal regions. Motivated by this observation, we propose Trajectory-Regularized Stochasticity Injection (TRSI), a lightweight mechanism that enhances stochastic exploration while stabilizing training through trajectory regularization. TRSI introduces magnitude-aware stochastic perturbations to active parameters and incorporates trajectory-based regularization to guide optimization. Extensive experiments on CIFAR-100, TinyImageNet, and ImageNet demonstrate that TRSI consistently improves existing DST methods across multiple architectures and sparsity levels, with notable gains under extreme sparsity. These results highlight the importance of stochastic exploration in sparse training and provide a simple yet effective approach for improving dynamic sparse training.