A novel strategy to balance the results of cross-modal hashing.

•This paper reveals the problem of unbalanced semantic information of different feature representations in cross-modal retrieval and explores the semantic augmentation for cross-modal retrieval.•A semantic augmentation strategy based on the intermediate semantic space is proposed to augment the semantic information of the modality data with weak semantics.•Extensive experiments on four datasets using typical cross-modal hashing methods, as well as real-valued, partial-paired, semi-paired, and completely unpaired cross-modal retrieval approaches are conducted to evaluate the effectiveness of semantic augmentation, and the results show that the gap between cross-modal retrieval results can be decreased substantially.