Silent Consensus: Probabilistic Packet Sampling For Lightweight Network Monitoring

Artificial intelligence based methods for operations of IT-systems (AIOps) support the process of maintaining and operating large IT infrastructures on different levels, e.g. anomaly detection, root cause analysis, or initiation of self-stabilizing activities. The foundation for the deployment of such methods are extensive and reliable metric data on the current state of the overall system. In particular, network information expressing the core parameters latency, throughput, and bandwidth have crucial impact on modern IoT and edge computing environments. Collecting the data is a challenging problem, as the communication is limited to existent network protocols, and adding new features requires a major infrastructure adaptation. The usage of additional monitoring protocols increases the CPU/network overhead and should be avoided as well. Therefore, we propose a two step approach for measuring latency between adjacent hops without manipulating or generating any network traffic. Inspired by audio and image compression algorithms, we developed a probabilistic method named silent consensus, where we keep the precision within a desired interval while reducing the overhead significantly. This method identifies the same packets on a sequence of network hops solely by observing the regular traffic. A linear regression helps to predict packets that are likely to appear after a fixed temporal offset based on a constrained set of historic observations. A correction of the predicted entity increases the probability for consensus between the involved hops. An extensive experimental evaluation proves that the approach delivers the expected foundation for further analysis of the network streams and the overall system.