Using Language Models to Pre-train Features for Optimizing Information Technology Operations Management Tasks

Information Technology (IT) Operations management is a vexing problem for most companies that rely on IT systems for mission-critical business applications. While IT operators are increasingly lever-aging analytical tools powered by artificial intelligence (AI), the volume, the variety and the complexity of data generated in the IT Operations domain poses significant challenges in managing the applications. In this work, we present an approach to leveraging language models to pre-train features for optimizing IT Operations management tasks such as anomaly prediction from logs. Specifically, using log-based anomaly prediction as the task, we show that the machine learning models built using language models (embeddings) trained with IT Operations domain data as features outperform those AI models built using language models with general-purpose data as features. Furthermore, we present our empirical results outlining the influence of factors such as the type of language models, the type of input data, and the diversity of input data, on the prediction accuracy of our log anomaly prediction model when language models trained from IT Operations domain data are used as features. We also present the run-time inference performance of log anomaly prediction models built using language models as features in an IT Operations production environment.