CAAN: Context-Aware attention network for visual question answering

•Introduce contextual information into the VQA task for the first time and propose a context-aware model CAAN.•Employ the positional relationship information between the image regions and the image to obtain a context-enhanced visual representation.•First introduce question contextual information to enhance the question feature representation in VQA.•Reach a significant performance improvement or comparable performance compared with some other state-of-the-art VQA models.