Structured Pruning for Vision Transformer Via Multiple Linear Regression

Vision Transformers (ViTs) have emerged as a dominant backbone for various visual tasks, yet their high computational cost limits deployment in real-time applications. While structured pruning offers a viable solution by removing entire neurons to enable hardware-friendly acceleration, it often leads to significant performance degradation due to coarse-grained pruning. This paper proposes a novel recovery algorithm for structured pruning in ViTs, leveraging multivariate linear regression to approximate pruned neurons using remaining ones, thereby mitigating accuracy loss. Unlike traditional methods requiring labeled data and retraining, our approach utilizes unlabeled data and statistical relationships between neurons for restoration. Specifically, we introduce a regression-based recovery mechanism where the regression matrix is absorbed into the model parameters, ensuring no additional inference overhead. Furthermore, we design a neuron selection strategy based on estimated recovery error to optimize pruning decisions, balancing computational efficiency and restoration quality. Experiments demonstrate that our method effectively preserves model performance while maintaining hardware compatibility, offering a practical solution for deploying lightweight ViTs in resource-constrained scenarios.