Gangs of the internet: Towards automatic discovery of peer-to-peer communities

Internet Service Providers and network administrators currently lack effective means for discovering and tracking peer-to-peer (P2P) applications on their networks. This ability would be very useful in various ways such as enforcing security policies on the use of P2P applications (e.g. banning file-sharing networks such as Bit Torrent), mitigating malicious P2P networks (i.e. botnets), or allocating network resources appropriately to improve network performance. To provide this ability, in this work we propose a method to discover P2P networks (both benign and malicious) from network flow records captured at the boundary of a tier-1 Internet backbone provider. The basic idea is that flows belonging to P2P applications can be modeled as observations from a mixed membership statistical model, with P2P applications acting as latent variables. Hence the communication patterns of hosts (who-talks-to-whom), as measured at the edge of a large network, can be decomposed into constituent application-layer P2P communities without any human effort in selecting specific features. This allows for automatic identification and isolation of P2P communities of interest, including those that take deliberate measures to remain hidden, as well as new or evolving ones such as P2P Botnets. In large scale experiments on flow records from a portion of IPv4 space of size /8, we demonstrate that the proposed method is able to detect a number of well known P2P networks, as well as a few evolving malicious P2P botnets.