Spamming botnets present a critical challenge in the control of spam messages due to the sheer volume and wide spread of the botnet members. In this paper we advocate the approach for recipient mail servers to filter messages directly delivered from remote end-user (EU) machines, given that the majority of spamming bots are EU machines. We develop a Support Vector Machine (SVM) based classifier to separate EU machines from legitimate mail server (LMS) machines, using a set of machine features that cannot be easily manipulated by spammers. We investigate the efficacy and performance of the SVM-based classifier using a number of real-world data sets. Our performance studies show that the SVM-based classifier is indeed a feasible and effective approach in distinguishing EU machines from LMS machines. For example, training and testing on an aggregated data set containing both EU machines and LMS machines, the SVM-based classifier can achieve a 99.27% detection accuracy, with very small false positive rate (0.44%) and false negative rate (1.1%), significantly outperforming eight DNS-based blacklists widely used today.
Phishing sites have become a common approach to steal sensitive information, such as usernames, passwords and credit card details of the internet users. We propose a semisupervised machine learning approach to detect phishing URLs from a set of phishing and spam URLs. Spam emails are the source of these URLs. In reality, the number of phishing URLs received through these spam emails is fewer compared to other URLs. Our study is targeted to detect phishing URLs in a realistic scenario of a highly imbalanced data set containing phishing and spam URLs with 1:654 ratio. To train a learning algorithm labeled URLs are needed, where manual labeling is a common approach. Given that it is not feasible to manually label all the URLs from large data sets, we propose reducing manual intervention by labeling only 10% of the URLs manually and using a semisupervised learning algorithm. We compare the proposed approach with a supervised learning approach. Evaluation results show that our proposal is competitive if it is applied in combination with appropriate feature selection and undersampling techniques.
IP blacklists are a well-regarded anti-spam mechanism that capture global spamming patterns. These properties make such lists a practical ground-truth by which to study email spam behaviors. Observing one blacklist for nearly a year-and-a-half, we collected data on roughly half a billion listing events. In this paper, that data serves two purposes. First, we conduct a measurement study on the dynamics of blacklists and email spam at-large. The magnitude/duration of the data enables scrutiny of long-term trends, at scale. Further, these statistics help parameterize our second task: the mining of blacklist history for temporal association rules. That is, we search for IP addresses with correlated histories. Strong correlations would suggest group members are not independent entities and likely share botnet membership. Unfortunately, we find that statistically significant groupings are rare. This result is reinforced when rules are evaluated in terms of their ability to: (1) identify shared botnet members, using ground-truth from botnet infiltrations and sinkholes, and (2) predict future blacklisting events. In both cases, performance improvements over a control classifier are nominal. This outcome forces us to re-examine the appropriateness of blacklist data for this task, and suggest refinements to our mining model that may allow it to better capture the dynamics by which botnets operate.
Traditional content-based e-mail spam filtering takes into account content of e-mail messages and apply machine learning techniques to infer patterns that discriminate spams from hams. In particular, the use of content-based spam filtering unleashed an unending arms race between spammers and filter developers, given the spammers' ability to continuously change spam message content in ways that might circumvent the current filters. In this paper, we propose to expand the horizons of content-based filters by taking into consideration the content of the Web pages linked by e-mail messages. We describe a methodology for extracting pages linked by URLs in spam messages and we characterize the relationship between those pages and the messages. We then use a machine learning technique (a lazy associative classifier) to extract classification rules from the web pages that are relevant to spam detection. We demonstrate that the use of information from linked pages can nicely complement current spam classification techniques, as portrayed by SpamAssassin. Our study shows that the pages linked by spams are a very promising battleground.
E-mail sender authentication is a promising way of verifying the sources of e-mail messages. Since today's primary e-mail sender authentication mechanisms are designed as fully decentralized architecture, it is crucial for e-mail operators to know how other organizations are using and misusing them. This paper addresses the question "How is the DNS Sender Policy Framework (SPF), which is the most popular e-mail sender authentication mechanism, used and misused in the wild?" To the best of our knowledge, this is the first extensive study addressing the fundamental question. This work targets both legitimate and spamming domain names and correlates them with multiple data sets, including the e-mail delivery logs collected from medium-scale enterprise networks and various IP reputation lists. We first present the adoption and usage of DNS SPF from both global and local viewpoints. Next, we present empirically why and how spammers leverage the SPF mechanism in an attempt to pass a simple SPF authentication test. We also present that non-negligible volume of legitimate messages originating from legitimate senders will be rejected or marked as potential spam with the SPF policy set by owners of legitimate domains. Our findings will help provide (1) e-mail operators with useful insights for setting adequate sender or receiver policies and (2) researchers with the detailed measurement data for understanding the feasibility, fundamental limitations, and potential extensions to e-mail sender authentication mechanisms.
