Towards model-based anomaly detection in network communication protocols

Over the last few years many techniques have been applied to find and mitigate vulnerabilities, misuses, cyber-attacks and other cyber-security flaws. One of the approaches, which we consider in this paper, is a model-based technique applied to network communication protocols. This idea is not brand new, and model-based techniques have been successfully used to verify and validate the standard models of communication protocols. However, the implementation of network protocols varies from one system to another, and in many cases they miss standards or recommendations. Attackers know these flaws very often and try to use them before everybody else finds them, what can be called “zero-day exploit of communication protocol.” To address this issue, a combination of the best features of model-based and anomaly detection techniques could be applied. Treating discovered anomalies as a signature of a cyber-attack or any other malicious activity and focusing on the investigation of them could significantly increase the success rate of the defense against them. In this paper we considered some significant inputs from the research community to model-based anomaly detection in network communication protocols. Then we prepared a synthetic brief of the theories and methods for modelling network protocols as state-machines. Next we examined the application of it in a cyber-security area. Finally we proposed some key directions that actual research should follow to bring some breakthrough results as soon as possible.