Advanced binary analysis techniques for malware defense

As malware has become one of the most efficient vehicles for monetarily driven cybercrimes, the past several years have seen an exponentially growing number of malicious codes emerging in the wild. In the face of this emerging threat, various efforts have been made to develop binary analysis techniques for malware defense, such as malware behavior analysis in emulated environments and information flow tracking using dynamic taint analysis. However, the development of such binary analysis techniques is challenging for several reasons. Most of all, malware authors attempt to frustrate analysis by applying various anti-analysis techniques. Another challenge to analysis arises from the inaccuracy of the existing analysis techniques. In this work, we discuss two major challenges in binary analysis for malware defense: (1) anti-analysis techniques and (2) taint propagation problems in dynamic taint analysis. We present advanced binary analysis techniques for malware defense and demonstrate that our techniques ameliorate the problems in realistic environments. We present practical methods for addressing two categories of anti-analysis problems: code packing and emulation resistance. To extract hidden code from packed malware samples, we propose a fully dynamic approach that captures an intrinsic nature of packed executables and is thus capable of dealing with arbitrary code-packing techniques without a priori knowledge of them. To remedy emulation resistance, we propose an automated technique that dynamically modifies the execution of a whole system emulator to fool a malware sample's anti-emulation checks. We demonstrate the effectiveness of our techniques by implementing and evaluating them with real malware samples collected in the wild. We also show that dynamic taint analysis can achieve better accuracy in analyzing sensitive information flows by improving control-flow propagation that causes under-tainting problems. We propose DTA++, an enhancement technique that additionally propagates taint through a minimum subset of control-flow dependencies. We implement and evaluate DTA++ with off-the-shelf Windows word processor applications, showing that it efficiently locates and resolves culprit implicit flows while introducing very little over-tainting.