Towards a large-scale person search by vietnamese natural language: dataset and methods

Person search by natural language description is a challenging problem because of demands for modelling and learning visual-text semantic embedding. While several works have been dedicated to person search by English description, very few attempts have been made for other languages. This paper presents the first work towards person search by Vietnamese description. The contribution of the paper is threefold. First, the first and large-scale dataset for person search by Vietnamese natural language named 3000VnPersonSearch is built. Second, inspired by dual-path architecture (Zheng et al. ACM Trans Multimed Comput Commun Appl (TOMM) 16(2):1–23, 2020 ), in which single loss for intra-modal and triple loss for cross-modal learning of text and image data distribution were considered, in this paper, we employ this architecture for Vietnamese description-based person search. However, as Vietnamese language is under-resource, the existing word embedding model is still modest compared to that of English. Therefore, instead of using word2vec model as in Zheng et al. ( ACM Trans Multimed Comput Commun Appl (TOMM) 16(2):1–23, 2020 ), we modify the initialization process of the first convolution layer of the text-CNN path. In addition, we investigate in detail two online triplet mining strategies that are batch all and batch hard triplet. Extensive experiments have been conducted on benchmark datasets as well as on 3000VnPersonSearch. Experimental results show that the proposed method obtains 2.42% of improvement over the baseline method on CUHK-PEDES dataset and achieved state of the art results on VnPersonSearch dataset with a significant margin in comparison with the method in Pham et al. ( 2020 ). Finally, in order to illustrate the practical usage of person search by Vietnamese description language, a web-based application of person search is implemented and deployed.