SP^3: Enhancing Structured Pruning Via PCA Projection

Structured pruning is a widely used technique for reducing the size ofpre-trained language models (PLMs), but current methods often overlook thepotential of compressing the hidden dimension (d) in PLMs, a dimension criticalto model size and efficiency. This paper introduces a novel structured pruningapproach, Structured Pruning with PCA Projection (SP3), targeting the effectivereduction of d by projecting features into a space defined by principalcomponents before masking. Extensive experiments on benchmarks (GLUE and SQuAD)show that SP3 can reduce d by 70over 96accuracy at the same compression ratio. SP3 has also proven effective withother models, including OPT and Llama. Our data and code are available at ananonymous repo.