AI-HydRa: Advanced hybrid approach using random forest and deep learning for malware classification

The extremely diffused architecture of the Internet enables the propagation of malware and presents a significant challenge for the development of defenses against such malware propagation. Although machine learning-based malware detection models can improve approaches in response to this problem, their detection rates vary according to their features and classification methods. Single machine learning approaches for malware detection can vary in effectiveness according to the suitability of their classifiers despite the use of an appropriate training dataset. Some classifiers result in high detection rates with a malicious training dataset but have low detection rates with a benign training dataset, and false positive rates are particularly dependent on the use of appropriate classifiers. In this paper, we propose a machine learning-based hybrid decision model that can achieve a high detection rate with a low false positive rate. This hybrid model combines a random forest and a deep learning model using 12 hidden layers to determine malware and benign files, respectively. This model also includes certain proposed voting rules to make final decisions. In an experiment involving 6,395 atypical samples, this hybrid decision model achieved a higher detection rate (85.1% and standard deviation of 0.006) than that of the prior model (65.5%) without voting rules.