Controlling the events occurring in the network traffic and detecting malicious activities are of great importance for the security and sustainability of the system. For this reason, it is necessary to accurately detect different types of attacks that may occur in network traffic. On the other hand, it is very important from the security aspect to be able to distinguish the types of attacks that have not been seen before. In this paper, a two-step procedure is proposed that can both correctly classify known attack types and distinguish unknown attack types. In the first stage, incoming traffic is classified by supervised learning with an autoencoder. In the second stage, the reliability of this classification is checked with the help of the autoencoder and the Extreme Value Theorem (EVT). According to the reliability value, incoming traffic is classified as unknown class. The simulation results were obtained with the IDS 2017 data set, which is widely used in the literature.

Network traffic characterization has become an important topic with the development of evasion techniques. Preventing malicious activities is vital in terms of network performance. On the other hand, defining which applications are run through the network traffic has become a significant issue with the diversification of applications. In this work, a payload-based flow analysis tool that provides both application classification and intrusion detection is proposed. Payload features that characterize network flows efficiently are used to classify network traffic and detect malicious attacks. Application classification performance analysis is performed on a publicly available up-to-date dataset containing traces from most popular applications such as Spotify, WhatsApp, etc. Attack detection performance is evaluated on IDS 2012 and IDS 2017 datasets containing different kinds of attack traces.

Network attacks become more complicated with the improvement of technology. Traditional statistical methods may be insufficient in detecting constantly evolving network attack. For this reason, the usage of payload-based deep packet inspection methods is very significant in detecting attack flows before they damage the system. In the proposed method, features are extracted from the byte distributions in the payload and these features are provided to characterize the flows more deeply by using N-Gram analysis methods. The proposed procedure has been tested on IDS 2012 and 2017 datasets, which are widely used in the literature.

Traffic classification has a crucial place in today’s technology in terms of detecting events on the network. With the developing encryption and evasion techniques, the toughness of traffic classification is increasing. It is seen that traditional methods have low success rate on current datasets. In this paper, a classification algorithm based on statistical and payload based features with the use of machine learning techniques is proposed. The proposed algorithm has been tested on a dataset containing up-to-date network traffic. The obtained results show that the algorithm has high accuracy.