Language Model for Text Analytic in Cybersecurity

domain. Using our proposed methods for tokenization and model weights adjustment, SecureBERT is not only able to preserve the understanding of general English as most "off-the-shelf" language models can do, but also effective when applied to text that has cybersecurity implications. We conducted comparative evaluations of SecureBERT using the industry-standard Masked Language Model (MLM) test to demonstrate its efficacy in processing cybersecurity text, as well as two downstream tasks to prove how well it retains general English language understanding. NOTE : This is the initial draft of this work and it may contain errors and typos. The revised.complete version has already been submitted to a venue. Following the publication of the paper, the full version would be updated and the corpus and the pre-trained model will be made available on GitHub.