System-call monitoring has become the basis for many hostbased intrusion detection as well as policy enforcement techniques. Mimicry attacks attempt to evade system-call monitoring IDS by executing innocuous-looking sequences of system calls that accomplish the attacker’s goals. Mimicry attacks may execute a sequence of dozens of system calls in order to evade detection. Finding such a sequence is difficult, so researchers have focused on tools for automating mimicry attacks and extending them to gray-box IDS 1. In this paper, we describe an alternative approach for building mimicry attacks using only skills and technologies that hackers possess today, making this attack a more immediate and realistic threat. These attacks, which we call persistent interposition attacks, are not as powerful as traditional mimicry attacks — an adversary cannot obtain a root shell using a persistent interposition attack — but are sufficient to accomplish the goals of today’s cyber-criminals. Persistent interposition attacks are stealthier than standard mimicry attacks and are amenable to covert information-harvesting attacks, features that are likely to be attractive to profitmotivated criminals. Persistent interposition attacks are not IDS specific — they can evade a large class of systemcall-monitoring intrusion-detection systems, which we call I/O-data-oblivious. I/O-data-oblivious monitors have perfect knowledge of the values of all system call arguments as well as their relationships, with the exception of data buffer arguments to read and write. Many of today’s black-box and gray-box IDS are I/O-data-oblivious and hence vulnerable to persistent interposition attacks.

Abstract. Many host-based anomaly detection systems monitor process execution at the granularity of system calls. Other recently proposed schemes instead verify the destinations of control-flow transfers to prevent the execution of attack code. This paper formally analyzes and compares real systems based on these two anomaly detection philosophies in terms of their attack detection capabilities, and proves and disproves several intuitions. We prove that for any system-call sequence model, under the same (static or dynamic) program analysis technique, there always exists a more precise control-flow sequence based model. While hybrid approaches combining system calls and control flows intuitively seem advantageous, especially when binary analysis constructs incomplete models, we prove that they have no fundamental advantage over simpler control-flow models. Finally, we utilize the ideas in our framework to make external monitoring feasible at the precise control-flow level. Our experiments show that external control-flow monitoring imposes performance overhead comparable to previous system call based approaches while detecting synthetic and real world attacks as effectively as an inlined monitor.

Heap and stack buffer overflows are still among the most common attack vectors in intrusion attempts. In this paper, we ask a simple question that is surprisingly difficult to answer: which bytes contributed to the overflow? By careful observation of all scenarios that may occur in overflows, we identified the information that needs to be tracked to pinpoint the offending bytes. There are many reasons why this is a hard problem. For instance, by the time an overflow is detected some of the bytes may already have been overwritten, creating gaps. Additionally, it is hard to tell the offending bytes apart from unrelated network data. In our solution, we tag data from the network with an age stamp whenever it is written to a buffer. Doing so allows us to distinguish between different bytes and ignore gaps, and provide precise analysis of the offending bytes. By tracing these bytes to protocol fields, we obtain accurate signatures that cater to polymorphic attacks.

age stamps

Mimicry attacks have been the focus of detector research where the objective of the attacker is to generate an attack that evades detection while achieving the attacker’s goals. If such an attack can be found, it implies that the target detector is vulnerable against mimicry attacks. In this work, we emphasize that there are two components of a buffer overflow attack: the preamble and the exploit. Although the attacker can modify the exploit component easily, the attacker may not be able to prevent preamble from generating anomalous behavior since during preamble stage, the attacker does not have full control. Previous work on mimicry attacks considered an attack to completely evade detection, if the exploit raises no alarms. On the other hand, in this work, we investigate the source of anomalies in both the preamble and the exploit components against two anomaly detectors that monitor four vulnerable UNIX applications. Our experiment results show that preamble can be a source of anomalies, particularly if it is lengthy and anomalous. Keywords Information hiding, mimicry attacks, anomaly detection, vulnerability testing 1

Abstract—Many host-based anomaly detection techniques have been proposed to detect code-injection attacks on servers. The vast majority, however, are susceptible to “mimicry ” attacks in which the injected code masquerades as the original server software, including returning the correct service responses, while conducting its attack. “Behavioral distance, ” by which two diverse replicas processing the same inputs are continually monitored to detect divergence in their low-level (system-call) behaviors and hence potentially the compromise of one of them, has been proposed for detecting mimicry attacks. In this paper, we present a novel approach to behavioral distance measurement using a new type of Hidden Markov Model, and present an architecture realizing this new approach. We evaluate the detection capability of this approach using synthetic workloads and recorded workloads of production web and game servers, and show that it detects intrusions with substantially greater accuracy than a prior proposal on measuring behavioral distance. We also detail the design and implementation of a new architecture, which takes advantage of virtualization to measure behavioral distance. We apply our architecture to implement intrusion-tolerant web and game servers, and through trace-driven simulations demonstrate that it experiences moderate performance costs even when thresholds are set to detect stealthy mimicry attacks. Index Terms—Intrusion detection, replicated system, output voting, system call, behavioral distance. Ç

Many host-based anomaly detection systems monitor a process by observing the system calls it makes, and comparing these calls to a model of normal behavior for the program that the process is executing. In this thesis we explore two novel approaches for constructing the normal behavior model for anomaly detection. We introduce execution graph, which is the first model that both requires no static analysis of the program source or binary, and conforms to the control flow graph of the program. When used as the model in an anomaly detection system monitoring system calls, it (i) accepts only system call sequences that are consistent with the control flow graph of the program; (ii) is maximal given a set of training data, meaning that any extensions to the execution graph could permit some intrusions to go undetected. We formalize and prove these claims, and evaluate the performance of an anomaly detector using execution graphs. Behavioral distance compares the behavior of a process to the behavior of another process that is executing on the same input but that either runs on a different operating system or runs a different program that has similar functionality. Assuming their diversity renders

Computer security systems protect computers and networks from unauthorized use by external agents and insiders. The similarities between computer security and the problem of protecting a body against damage from externally and internally generated threats are compelling and were recognized as early as 1972 when the term computer virus was coined. The connection to immunology was made explicit in the mid 1990s, leading to a variety of prototypes, commercial products, attacks, and analyses. The paper reviews one thread of this active research area, focusing on system-call monitoring and its application to anomaly intrusion detection and response. The paper discusses the biological principles illustrated by the method, followed by a brief review of how system call monitoring was used in anomaly intrusion detection and the results that were obtained. Proposed attacks against the method are discussed, along with several important branches of research that have arisen since the original papers were published. These include other data modeling methods, extensions to the original system call method, and rate limiting responses. Finally, the significance of this body of work and areas of possible future investigation are outlined in the conclusion. 1

Smartphones have come to resemble PCs in software complexity. Moreover, as they are often used for privacy-sensitive tasks, they are becoming attractive targets for attackers. Unfortunately, they are quite different from PCs in terms of resources, so that PC-oriented security solutions are not always applicable. Worse, common security solutions (such as on-access file scanners, system call profilers, etc.) protect against a very limited set of attacks. Comprehensive measures require a far wider and more expensive set of checks- some of which are much beyond the capacity of a phone. We propose an alternative solution, where security checks are applied on remote security servers which host exact replicas of the phones in virtual environments. The servers are not subject to the same constraints, allowing us to apply multiple detection techniques simultaneously (including ones that are very heavy-weight). Moreover, as the full execution trace is preserved on the security server, attackers cannot hide their traces. This property ensures that attacks for which a detection method exists at the security server- even if the detection method is installed at a later time- are detectable eventually. This is a stronger guarantee than existing solutions can give. It allows administrators to trade server resources for security by specifying how far back in time they want to be able to start looking for intrusions with a new detection method. We implemented the security model for Android phones and show that it is both practical and scalable: we generate about 2KiB/s and 64B/s of trace data under high-load and idle operation respectively, and are able to support more than a hundred replicas on a single server. 1

Abstract—In a masquerade attack, an adversary who has stolen a legitimate user’s credentials attempts to impersonate him to carry out malicious actions. Automatic detection of such attacks is often undertaken constructing models of normal behaviour of each user and then measuring significant departures from them. One potential vulnerability of this approach is that anomaly detection algorithms are generally susceptible of being deceived. In this paper, we first investigate how a resourceful masquerader can successfully evade detection while still accomplishing his goals. We then propose an algorithm based on the Kullback-Leibler divergence which attempts to identify if a sufficiently anomalous attack is present within an apparently normal request. Our experimental results indicate that the proposed scheme achieves considerably better detection quality than adversarial-unaware approaches. Index Terms—Anomaly detection; insider threats; masqueraders; mimicry attacks; Kullback-Leibler divergence. I.