Partially Recentralization Softmax Loss for Vision-Language Models Robustness
CoRR(2024)
摘要
As Large Language Models make a breakthrough in natural language processing
tasks (NLP), multimodal technique becomes extremely popular. However, it has
been shown that multimodal NLP are vulnerable to adversarial attacks, where the
outputs of a model can be dramatically changed by a perturbation to the input.
While several defense techniques have been proposed both in computer vision and
NLP models, the multimodal robustness of models have not been fully explored.
In this paper, we study the adversarial robustness provided by modifying loss
function of pre-trained multimodal models, by restricting top K softmax
outputs. Based on the evaluation and scoring, our experiments show that after a
fine-tuning, adversarial robustness of pre-trained models can be significantly
improved, against popular attacks. Further research should be studying, such as
output diversity, generalization and the robustness-performance trade-off of
this kind of loss functions. Our code will be available after this paper is
accepted
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要