Owner name entity recognition in websites based on heterogeneous and dynamic graph transformer

Identifying owners of devices on the Internet can enable numerous network security applications. For example, accurate Owner Name Entity Recognition (ONER) of websites is critical to find influenced owners in light of new security threats. In this situation, as a specific task of Multimodal Named Entity Recognition (MNER), ONER is essential and helpful for network security. Currently, most existing MNER models only use texts and images, so they cannot effectively utilize the multimodal data of devices to achieve ONER accurately. Also, most of the existing MNER models separately use information in each modality and between modalities. Thus, the fusion is inconsistent, so the effect is not satisfied. Therefore, the paper proposes HDGT: A heterogeneous and Dynamic Graph Transformer, to improve the performance of ONER. The core components in HDGT to realize MNER are a dynamic graph and two-stream mechanism, which could learn the relationship between different modalities during training and the graph’s structure well. The paper manually labels a multimodal dataset containing texts, images, and domains to prove the performance of HDGT. Also, the paper conducts experiments on existing and public MNER datasets. The results show that HDGT achieves 84.88% F1 scores on the recognition of owner entities, 75.21% F1 on Twitter2015, and 87.03% F1 on Twitter2017, which outperforms other existing MNER models.