Learning Dynamic Representations in Large Language Models for Evolving Data Streams

In the world of Large Language Modeling, incremental learning plays an important role in evolving data such as streaming text. We introduce an incremental learning approach for dynamic contextualized word embeddings in the setting of streaming data. We call the embeddings generated by our model as Incremental Dynamic Contextualized Word Embeddings (iDCWE). Our model introduces the incremental BERT (iBERT) (BERT stands for Bidirectional Encoder Representations from Transformers) to create a dynamic and incremental model to perform incremental training. Our model further captures the semantic drift of words using dynamic graphs. Our paper is the first in the line of research on (incremental) dynamic modeling of streaming text which we also refer to as Neural Dynamic Language Modeling. The performance of our model on the benchmark datasets is on par and even often outperforms the dynamic contextualized word embeddings which was the first paper to combine contextualization with dynamic word embeddings, to the best of our knowledge. Moreover, the compute time efficiency of our model is better than that of the aforementioned paper.