Task Formulation Matters when Learning Continuously: A Case Study in Visual Question Answering

Continual learning is a promising alternative 001 to the current pretrain-and-finetune paradigm: 002 It aims to learn a model on a sequence of tasks 003 without forgetting knowledge from preceding 004 tasks. We investigate continual learning for Vi-005 sual Question Answering and show that perfor-006 mance highly depends on task design, order, 007 and similarity – where tasks may be formu-008 lated according to either modality. Our results 009 suggest that incremental learning of language 010 reasoning skills (such as questions about color, 011 count etc.) is more difficult than incrementally 012 learning visual categories. We show that this 013 difficulty is related to task similarity, where 014 heterogeneous tasks lead to more severe for-015 getting. We also demonstrate that naive fine-016 tuning of pretrained models is insufficient, and 017 recent continual learning approaches can re-018 duce forgetting by more than 20%. We pro-019 pose a simple yet effective P SEUDO -R EPLAY 020 algorithm, which improves results while using 021 less memory compared to standard replay. Fi-022 nally, to measure gradual forgetting we intro-023 duce a new metric that takes into account the 024 semantic similarity of predicted answers. 025