Linguistic Intelligence in Large Language Models for Telecommunications

Large Language Models (LLMs) have emerged as a significant advancement in thefield of Natural Language Processing (NLP), demonstrating remarkablecapabilities in language generation and other language-centric tasks. Despitetheir evaluation across a multitude of analytical and reasoning tasks invarious scientific domains, a comprehensive exploration of their knowledge andunderstanding within the realm of natural language tasks in thetelecommunications domain is still needed. This study, therefore, seeks toevaluate the knowledge and understanding capabilities of LLMs within thisdomain. To achieve this, we conduct an exhaustive zero-shot evaluation of fourprominent LLMs-Llama-2, Falcon, Mistral, and Zephyr. These models require fewerresources than ChatGPT, making them suitable for resource-constrainedenvironments. Their performance is compared with state-of-the-art, fine-tunedmodels. To the best of our knowledge, this is the first work to extensivelyevaluate and compare the understanding of LLMs across multiple language-centrictasks in this domain. Our evaluation reveals that zero-shot LLMs can achieveperformance levels comparable to the current state-of-the-art fine-tunedmodels. This indicates that pretraining on extensive text corpora equips LLMswith a degree of specialization, even within the telecommunications domain. Wealso observe that no single LLM consistently outperforms others, and theperformance of different LLMs can fluctuate. Although their performance lagsbehind fine-tuned models, our findings underscore the potential of LLMs as avaluable resource for understanding various aspects of this field that lacklarge annotated data.