ProRes-LoRA: Bridging the Rank Gap Via Progressive Orthogonal Residual Decomposition for Heterogeneous Federated Fine-tuning

Federated fine-tuning of Large Language Models (LLMs) faces significant challenges due to system heterogeneity across participating clients. While existing heterogeneous Low-Rank Adaptation (LoRA) methods attempt to address resource disparities by allocating variable ranks, they typically rely on zero-padding and truncation strategies. These approaches often suffer from aggregation noise and computational inefficiency. To overcome these limitations, we propose ProRes-LoRA, a Progressive Orthogonal Residual Decomposition framework. Unlike traditional dimension alignment approaches, ProRes-LoRA reformulates heterogeneous aggregation as a sequence of homogeneous residual learning subproblems. The training process is decoupled into two stages: "universal skeleton knowledge" and "refined texture learning". This mechanism effectively eliminates the optimization instability induced by sparse gradients in zero-padding and ensures feature space non-redundancy via orthogonal regularization. Extensive experiments demonstrate that ProRes-LoRA matches or exceeds the performance of HetLoRA while significantly reducing communication and computational overhead, achieving a 65.5% reduction compared to HetLoRA and 93.7% against FLoRA.