SDGen: A Scalable, Reproducible and Flexible Approach to Generate Real World Cyber Security Datasets

Real world cyber security datasets are essential for developing and evaluating new techniques to counter cyber attacks. Ideally, these datasets should represent modern network infrastructures with upto-date cyber attacks. However, existing datasets commonly used by researchers are either synthetic, unscalable or easily outdated due to the dynamic network infrastructure and evolving nature of cyber attacks. In this paper, we introduce a security dataset generator (SDGen) which focuses on a scalable, reproducible and flexible approach to generate real world datasets for detection and response against cyber attacks. We implement SDGen within a virtual environment using DetectionLab, ELK (Elasticsearch, Logstash, Kibana) stack with Beats and AttackIQ (a security control validation platform). This implementation in fact provides a proof-of-concept (POC) of SDGen to demonstrate the dataset generation of an organisation being compromised by several types of Ransomware. We showcase that our proposed dataset generator, SDGen, provides scalability, reproducibility and flexibility in generating cyber security datasets by modifying the configurations in DetectionLab, Vagrant-Files and launching different types of attacks in AttackIQ.