NeuroPatch: Lightweight Diffusion Model Repair for Backdoor Attack Mitigation Based on Neuron-Level Patching

Diffusion models, as generative models, achieve remarkable results across diverse applications. Despite their excellent performance, diffusion models are vulnerable to backdoor attacks, as recent studies show. Although a few methods attempt to mitigate backdoors, they require adjusting the entire model, incurring high costs and risking degradation in generative performance. To address this, we propose NEUROPATCH, a framework that patches only a minimal subset of backdoor-related erroneous neurons without modifying the neural net work itself, thereby significantly reducing intervention costs and preserving model performance. Specifically, NEUROPATCH first approximates the distribution of clean samples to detect backdoor-infected inputs, then ranks neurons based on the activation deviations induced by these abnormal samples. Finally, it patches the most error-inducing neurons by attaching lightweight corrective controllers to adjust their outputs, without changing the model parameters. We evaluate NEUROPATCH on 168 diffusion models across two datasets and three main stream backdoor attacks. Extensive results demonstrate both the effectiveness and efficiency of our approach: NEUROPATCH achieves, on average, up to 24 & times; faster repair speed compared to state-of-the-art methods, while slightly improving backdoor mitigation performance. Ablation studies further confirm that ranking neurons by their error-inducing scores is a key factor driving the effectiveness of our framework. Our code is available at: https://github.com/Mospic/NeuroPatch.