Password-based authentication remains the main method of user authentication in computer systems. In case of a leak of the user database, the obfuscated storage of passwords is the last remaining protection of credentials. The strength of a password determines how hard it is to crack a password hash for uncovering the plain text password. Internet users often ignore recommended password guidelines and choose weak passwords that are easy to guess. In addition, service providers do not warn users that their chosen passwords are not secure enough. In this work we present a semi-automatic password cracking algorithm that orders and executes user-chosen password cracking attacks based on their efficiency. With our new approach, we are able to accelerate the cracking of password hashes and to demonstrate that weak passwords are still a serious security risk. The intention of this work is to point out that the usage of weak passwords holds great dangers for both the user and the service provider.

The relevance of identity data leaks on the Internet is more present than ever. Almost every week we read about leakage of databases with more than a million users in the news. Smaller but not less dangerous leaks happen even multiple times a day. The public availability of such leaked data is a major threat to the victims, but also creates the opportunity to learn not only about security of service providers but also the behavior of users when choosing passwords. Our goal is to analyze this data and generate knowledge that can be used to increase security awareness and security, respectively. This paper presents a novel approach to the processing and analysis of a vast majority of bigger and smaller leaks. We evolved from a semi-manual to a fully automated process that requires a minimum of human interaction. Our contribution is the concept and a prototype implementation of a leak processing workflow that includes the extraction of digital identities from structured and unstructured leak-files, the identification of hash routines and a quality control to ensure leak authenticity. By making use of parallel and distributed programming, we are able to make leaks almost immediately available for analysis and notification after they have been published. Based on the data collected, this paper reveals how easy it is for criminals to collect lots of passwords, which are plain text or only weakly hashed. We publish those results and hope to increase not only security awareness of Internet users but also security on a technical level on the service provider side.

The last years have shown an increasing sophistication of attacks against enterprises. Traditional security solutions like firewalls, anti-virus systems and generally Intrusion Detection Systems (IDSs) are no longer sufficient to protect an enterprise against these advanced attacks. One popular approach to tackle this issue is to collect and analyze events generated across the IT landscape of an enterprise. This task is achieved by the utilization of Security Information and Event Management (SIEM) systems. However, the majority of the currently existing SIEM solutions is not capable of handling the massive volume of data and the diversity of event representations. Even if these solutions can collect the data at a central place, they are neither able to extract all relevant information from the events nor correlate events across various sources. Hence, only rather simple attacks are detected, whereas complex attacks, consisting of multiple stages, remain undetected. Undoubtedly, security operators of large enterprises are faced with a typical Big Data problem. In this thesis, we propose and implement a prototypical SIEM system named Real-Time Event Analysis and Monitoring System (REAMS) that addresses the Big Data challenges of event data with common paradigms, such as data normalization, multi-threading, in-memory storage, and distributed processing. In particular, a mostly stream-based event processing workflow is proposed that collects, normalizes, persists and analyzes events in near real-time. In this regard, we have made various contributions in the SIEM context. First, we propose a high-performance normalization algorithm that is highly parallelized across threads and distributed across nodes. Second, we are persisting into an in-memory database for fast querying and correlation in the context of attack detection. Third, we propose various analysis layers, such as anomaly- and signature-based detection, that run on top of the normalized and correlated events. As a result, we demonstrate our capabilities to detect previously known as well as unknown attack patterns. Lastly, we have investigated the integration of cyber threat intelligence (CTI) into the analytical process, for instance, for correlating monitored user accounts with previously collected public identity leaks to identify possible compromised user accounts. In summary, we show that a SIEM system can indeed monitor a large enterprise environment with a massive load of incoming events. As a result, complex attacks spanning across the whole network can be uncovered and mitigated, which is an advancement in comparison to existing SIEM systems on the market.

The analysis of security-related event logs is an important step for the investigation of cyber-attacks. It allows tracing malicious activities and lets a security operator find out what has happened. However, since IT landscapes are growing in size and diversity, the amount of events and their highly different representations are becoming a Big Data challenge. Unfortunately, current solutions for the analysis of security-related events, so called Security Information and Event Management (SIEM) systems, are not able to keep up with the load. In this work, we propose a distributed SIEM platform that makes use of highly efficient distributed normalization and persists event data into an in-memory database. We implement the normalization on common distribution frameworks, i.e. Spark, Storm, Trident and Heron, and compare their performance with our custom-built distribution solution. Additionally, different tuning options are introduced and their speed advantage is presented. In the end, we show how the writing into an in-memory database can be tuned to achieve optimal persistence speed. Using the proposed approach, we are able to not only fully normalize, but also persist more than 20 billion events per day with relatively small client hardware. Therefore, we are confident that our approach can handle the load of events in even very large IT landscapes.

After almost two decades of development, modern Security Information and Event Management (SIEM) systems still face issues with normalisation of heterogeneous data sources, high number of false positive alerts and long analysis times, especially in large-scale networks with high volumes of security events. In this paper, we present our own prototype of SIEM system, which is capable of dealing with these issues. For efficient data processing, our system employs in-memory data storage (SAP HANA) and our own technologies from the previous work, such as the Object Log Format (OLF) and high-speed event normalisation. We analyse normalised data using a combination of three different approaches for security analysis: misuse detection, query-based analytics, and anomaly detection. Compared to the previous work, we have significantly improved our unsupervised anomaly detection algorithms. Most importantly, we have developed a novel hybrid outlier detection algorithm that returns ranked clusters of anomalies. It lets an operator of a SIEM system to concentrate on the several top-ranked anomalies, instead of digging through an unsorted bundle of suspicious events. We propose to use anomaly detection in a combination with signatures and queries, applied on the same data, rather than as a full replacement for misuse detection. In this case, the majority of attacks will be captured with misuse detection, whereas anomaly detection will highlight previously unknown behaviour or attacks. We also propose that only the most suspicious event clusters need to be checked by an operator, whereas other anomalies, including false positive alerts, do not need to be explicitly checked if they have a lower ranking. We have proved our concepts and algorithms on a dataset of 160 million events from a network segment of a big multinational company and suggest that our approach and methods are highly relevant for modern SIEM systems.

The number of attacks that use sophisticated and complex methods increased lately. The main objective of these attacks is to largely infiltrate the target network and to stay undetected. Therefore, the attackers often use valid credentials and standard administrative tools to hide between legitimate user actions and to hinder detection. Most existing security systems, which use standard signature-based or anomaly-based approaches, are not able to identify this type of malicious activities. Furthermore, it is also most often not feasible to analyze user behavior manually, due to the complexity of this task and the high amount of different user actions. Thus, it is necessary to develop new automated approaches to identify suspicious user behavior. In this paper, we propose to use neural networks to analyze user behavior and to identify suspicious actions. Due to the fact that neural networks require suitable datasets to learn the difference between suspicious and benign actions, we describe a behavioral simulation system to generate reasonable datasets. These datasets use different behavioral features to describe log-on and log-off activities of users. To identify suitable neural network models for user behavior analysis, we evaluate and compare 16,275 different feed-forward neural networks with three different datasets and 75 recurrent neural networks with one dataset. The results show that the used dataset and the complexity of a model are crucial to achieve a high accuracy. Appropriate models, which also consider context behavior information, are able to automatically classify before unseen user actions with an accuracy of up to 98 %.

Nowadays, identity breaches are happening almost on a daily basis. Just recently, hundreds of millions of identities were leaked from services like LinkedIn, MySpace and VKontakte. Undoubtedly, these breaches constitute a major threat because victims might fall to identity theft. As part of our warning service for victims of these breaches, we have gathered and normalized most of the publicly available breaches and could assess nearly one billion credentials. Apart from our security awareness service, the large amount of real world credentials allows to create comprehensive and realistic password statistics. In this paper, we introduce multiple comprehensive statistics on the use of passwords based on the gathered data. We especially focus on the often mentioned, but rarely researched, issue of password reuse and reveal the regional differences in password selection. We are confident that the analysis of such a large amount of real-life credentials is novel to existing studies on passwords, which were limited to thousands or a few million credentials, at most. For the first time, a realistic view on password reuse can be given.

Nowadays, attacks against single computer systems or whole infrastructures pose a significant risk. Although deployed security systems are often able to prevent and detect standard attacks in a reliable way, it is not uncommon that more sophisticated attackers are capable to bypass these systems and stay undetected. To support the prevention and detection of attacks, the sharing of cyber threat intelligence information becomes increasingly important. Unfortunately, the currently available threat intelligence formats, such as YARA or STIX (Structured Threat Information eXpression), cannot be used to describe complex patterns that are needed to share relevant attack details about more sophisticated attacks. In this paper, we propose an extension for the standardized STIX format that allows the description of complex patterns. With this extension it is possible to tag attributes of an object and use these attributes to describe precise relations between different objects. To evaluate the proposed STIX extension we analyzed the API calls of the credential dumping tool Mimikatz and created a pattern based on these calls. This pattern precisely describes the performed API calls of Mimikatz to access the LSASS (Local Security Authority Subsystem Service) process, which is responsible for authentication procedures in Windows. Due to the specified relations, it is possible to detect the execution of Mimikatz in a reliable way.

The relevance of identity data leaks on the Internet is more present than ever. Almost every month we read about leakage of databases with more than a million users in the news. Smaller but not less dangerous leaks happen even multiple times a day. The public availability of such leaked data is a major threat to the victims, but also creates the opportunity to learn not only about security of service providers but also the behavior of users when choosing passwords. Our goal is to analyze this data and generate knowledge that can be used to increase security awareness and security, respectively. This paper presents a novel approach to automatic analysis of a vast majority of bigger and smaller leaks. Our contribution is the concept and a prototype implementation of a parser, composed of a syntactic and a semantic module, and a data analyzer for identity leaks. In this context, we deal with the two major challenges of a huge amount of different formats and the recognition of leaks' unknown data types. Based on the data collected, this paper reveals how easy it is for criminals to collect lots of passwords, which are plain text or only weakly hashed.

Advanced persistent threats (APTs) pose a significant risk to nearly every infrastructure. Due to the sophistication of these attacks, they are able to bypass existing security systems and largely infiltrate the target network. The prevention and detection of APT campaigns is also challenging, because of the fact that the attackers constantly change and evolve their advanced techniques and methods to stay undetected. In this paper we analyze 22 different APT reports and give an overview of the used techniques and methods. The analysis is focused on the three main phases of APT campaigns that allow to identify the relevant characteristics of such attacks. For each phase we describe the most commonly used techniques and methods. Through this analysis we could reveal different relevant characteristics of APT campaigns, for example that the usage of 0-day exploit is not common for APT attacks. Furthermore, the analysis shows that the dumping of credentials is a relevant step in the lateral movement phase for most APT campaigns. Based on the identified characteristics, we also propose concrete prevention and detection approaches that make it possible to identify crucial malicious activities that are performed during APT campaigns.

Modern security information and event management systems should be capable to store and process high amount of events or log messages in different formats and from different sources. This requirement often prevents such systems from usage of computational heavy algorithms for security analysis. To deal with this issue, we built our system based on an in‐memory database with an integrated machine learning library, namely, SAP HANA. Three approaches, that is, (1) deep normalisation of log messages, (2) storing data in the main memory and (3) running data analysis directly in the database, allow us to increase processing speed in such a way that machine learning analysis of security events becomes possible nearly in real time. Besides that, we developed a universal anomaly detection algorithm, which uses vector space model to represent and cluster textual log messages. Together with deep normalisation approach, this algorithm solves the problem of correlation for heterogenous security events containing many text fields. To prove our concepts, we measured the processing speed for the developed system on the data generated using Active Directory testbed, compared it with classical system architecture based on PostgreSQL database and showed the efficiency of our approach for high‐speed analysis of security events. Copyright © 2016 John Wiley & Sons, Ltd.

The amount of identity data leaks in recent times is drastically increasing. Not only smaller web services, but also established technology companies are affected. However, it is not commonly known, that incidents covered by media are just the tip of the iceberg. Accordingly, more detailed investigation of not just publicly accessible parts of the web but also deep web is imperative to gain greater insight into the large number of data leaks. This paper presents methods and experiences of our deep web analysis. We give insight in commonly used platforms for data exposure, formats of identity related data leaks, and the methods of our analysis. On one hand a lack of security implementations among Internet service providers exists and on the other hand users still tend to generate and reuse weak passwords. By publishing our results we aim to increase awareness on both sides and the establishment of counter measures.

Modern Security Information and Event Management systems should be capable to store and process high amount of events or log messages in different formats and from different sources. This requirement often prevents such systems from usage of computational-heavy algorithms for security analysis. To deal with this issue, we built our system based on an in-memory data base with an integrated machine learning library, namely SAP HANA. Three approaches, i.e. (1) deep normalisation of log messages (2) storing data in the main memory and (3) running data analysis directly in the database, allow us to increase processing speed in such a way, that machine learning analysis of security events becomes possible nearly in real-time. To prove our concepts, we measured the processing speed for the developed system on the data generated using Active Directory tested and showed the efficiency of our approach for high-speed analysis of security events.

Looking at recent cyber-attacks in the news, a growing complexity and sophistication of attack techniques can be observed. Many of these attacks are performed in multiple steps to reach the core of the targeted network. Existing signature detection solutions are focused on the detection of a single step of an attack, but they do not see the big picture. Furthermore, current signature languages cannot integrate valuable external threat intelligence, which would simplify the creation of complex signatures and enables the detection of malicious activities seen by other targets. We extend an existing multi-step signature language to support attack detection on normalized log events, which were collected from various applications and devices. Additionally, the extended language supports the integration of external threat intelligence and allows us to reference current threat indicators. With this approach, we can create generic signatures that stay up-to-date. Using our language, we could detect various login brute-force attempts on multiple applications with only one generic signature.

When looking at media reports nowadays, major security breaches of big companies and governments seem to be a normal situation. An important step for the investigation or even prevention of these breaches is to normalize and analyze security-related log events from various systems in the target network. However, the number of log events produced in big IT landscapes can grow up to multiple billions per day. Current log management solutions, e.g., Security Information and Event Management (SIEM), cannot even closely normalize such huge amounts of data and therefore disable the tracking of attacks in real-time, which means that the log data remains unusable for attack analysis. In this paper, we present an approach to fully normalize event logs in high-speed by making use of established high-performance inter-thread messaging in conjunction with a hierarchical knowledge-base of log formats and parallel processing on multiple low-end systems. Using our approach, we are able to process more than 250,000 events/sec on relatively low-profile machines and can therefore easily handle more than 20 billion events/day, which is enough to handle average and peek loads of log events from big enterprise networks.

An important technique for attack detection in complex company networks is the analysis of log data from various network components. As networks are growing, the number of produced log events increases dramatically, sometimes even to multiple billion events per day. The analysis of such big data highly relies on a full normalization of the log data in realtime. Until now, the important issue of full normalization of a large number of log events is only insufficiently handled by many software solutions and not well covered in existing research work. In this paper, we propose and evaluate multiple approaches for handling the normalization of a large number of typical logs better and more efficient. The main idea is to organize the normalization in multiple levels by using a hierarchical knowledge base KB of normalization rules. In the end, we achieve a performance gain of about 1000x with our presented approaches, in comparison to a naive approach typically used in existing normalization solutions. Considering this improvement, big log data can now be handled much faster and can be used to find and mitigate attacks in realtime.

As computer networks grow in size and complexity, monitoring them becomes more challenging. In order to meet the needs of IT administrators maintaining such networks, various Network Monitoring Systems (NMS) have been developed. Most NMSs rely solely on active scanning techniques in order to detect the topology of the networks they monitor. We propose a passive scanning solution using the logs produced by the systems within the networks. Additionally, we demonstrate how passive monitoring can be used to develop a holistic knowledge graph of the network landscape.

The differences in log file formats employed in a variety of services and applications remain to be a problem for security analysts and developers of intrusion detection systems. The proposed solution, i.e. the usage of common log formats, has a limited utilisation within existing systems for security management. In our paper, we reveal the reasons for this limitation and show disadvantages of existing common log formats for normalisation of security events. To deal with it we have created a new log format that fits for intrusion detection purposes and can be extended easily. Based on our developing intrusion detection system, we demonstrate advantages of offered format. However, taking previous work into account, we would like to propose a new format as an extension to existing common log formats, rather than a standalone specification.

Understanding events produced by IT systems is a vital part of effectively managing and maintaining medium and large sized computer networks. As a result, Security Information and Management (SIEM) systems have become an indispensable part of modern networks. One of the main challenges facing SIEM systems is the ability to parse and extract relevant information from the processed events in near real-time. The more diverse the processed events are, the more complicated it becomes to extract the necessary information. Deep event correlation tasks perform very differently over structured data than they do over unstructured data. As an example, a text search for an IP address can take very long, but can return instantly if the IP address is extracted and stored in an appropriate field of its own. Dealing with dozens of different formats using the same event parsing module quickly becomes infeasible. As a result, the parsing of events has largely been outsourced to regular expressions. Although very affective at extracting information from events, they lack sophisticated logic operations. This paper presents a novel method of event processing using an embedded ECMAScript engine to effectively outsource the logic operations needed for deeper event processing.

A huge amount of information about real-time events are being generated in every second in a running IT-Infrastructure and recorded by the system logs, application logs, as well as the output from the deployed security or management methods, e.g., IDS alerts, firewall logs, scanning reports, etc. To rapidly gather, process, correlate, and analyze the massive event information is a challenging task. High performance security analytics is proposed to address this challenge by which the real-time event information can be normalized, centralized, and correlated to help identify the current running state of the target environment. As an example of next generation Security Information and Event Management (SIEM) platform, Security Analytics Lab (SAL) has been designed and implemented based on the newly emerged In-Memory data management technique, which makes it possible to efficiently organize, access, and process different types of event information through a consistent central storage and interface. In this paper, the multi-core architecture is introduced on the event correlation module of SAL platform by which the correlation tasks can be executed in parallel by different computing resources. The k-means algorithm is implemented as an example of possible event clustering and correlation algorithms. Several experiments are conducted and analyzed to show that the performance of analytics can be significantly improved by applying multi-core architecture in the event correlation procedure.