Propagation, detection and containment of mobile malware

Today's enterprise systems and networks are frequent targets of malicious attacks, such as worms, viruses, spyware and intrusions that can disrupt, or even disable critical services. Recent trends suggest that by combining spyware as a malicious payload with worms as a delivery mechanism, malicious programs can potentially be used for industrial espionage and identity theft. The problem is compounded further by the increasing convergence of wired, wireless and cellular networks, since virus writers can now write malware that can crossover from one network segment to another, exploiting services and vulnerabilities specific to each network. This dissertation makes four primary contributions. First, it builds more accurate malware propagation models for emerging hybrid malware (i.e., malware that use multiple propagation vectors such as Bluetooth, Email, Peer-to-Peer, Instant Messaging, etc.), addressing key propagation factors such as heterogeneity of nodes, services and user mobility within the network. Second, it develops a proactive containment framework based on group-behavior of hosts against such malicious agents in an enterprise setting. The majority of today's anti-virus solutions are reactive, i.e., these are activated only after a malicious activity has been detected at a node in the network. In contrast, proactive containment has the potential of closing the vulnerable services ahead of infection, and thereby halting the spread of the malware. Third, we study (1) the current-generation mobile viruses and worms that target SMS/MMS messaging and Bluetooth on handsets, and the corresponding exploits, and (2) their potential impact in a large SMS provider network using real-life SMS network data. Finally, we propose a new behavioral approach for detecting emerging malware targeting mobile handsets. Our approach is based on the concept of generalized behavioral patterns instead of traditional signature-based detection. The signature-based methods are not scalable for deployment in mobile devices due to limited resources available on today's typical handsets. Further, we demonstrate that the behavioral approach not only has a compact footprint, but also can detect new classes of malware that combine some features from existing classes of malware.