Cross-modal knowledge reasoning for knowledge-based visual question answering

•Using multiple knowledge graphs from the visual, semantic and factual views to depict the multimodal knowledge.•A memory-based recurrent model for multi-step knowledge reasoning over graphstructured multimodal knowledge.•Good interpretability to reveal the knowledge selection mode from different modalities.•Significant improvement over state-of-the-art approaches on three benchmark datasets.