Towards local visual modeling for image captioning

•Local visual modeling with grid features for image captioning.•Locality-Sensitive Attention (LSA) is deployed for the intra-layer interaction via local visual modeling.•Locality-Sensitive Fusion (LSF) is used for inter-layer information fusion.•Locality-Sensitive Transformer Network (LSTNet) outperforms SOTA captioning models on MS-COCO.•The generalization of LSTNet is also verified on the Flickr8k and Flickr30k datasets.