Training-free Deep Concept Injection Enables Language Models for Crossmodal Tasks

Recently, enabling pretrained language models (PLMs) to perform zero-shot crossmodal tasks has been extensively studied. A popular approach is to learn a projection network that projects visual features into the input text embedding space of a PLM, as well as feed-forward adaptation layers, with the weights of the PLM frozen. However, is it really necessary to learn such additional layers? In this paper, we make the first attempt to demonstrate that the PLM is able to perform zero-shot crossmodal tasks without any training, when the observed visual concepts are injected as both additional input text tokens and augmentation in the intermediate features within each feed-forward network for the PLM. Specifically, inputting observed visual concepts as text tokens helps to inject them through the self-attention layers in the PLM; to augment the intermediate features in a way that is compatible with the PLM, we propose to construct adaptation layers based on the intermediate representation of concepts (obtained by solely inputting them to the PLM). These two complementary injection mechanisms form the proposed Deep Concept Injection, which comprehensively enables the PLM to perceive instantly as learning process is no longer needed. Extensive empirical analysis on zero-shot video question answering and visual question answering shows Deep Concept Injection achieves competitive or even better results, compared to state-of-the-art methods requires crossmodal training.