Buffer overflow attacks are still a serious threat to the security of software systems. One of the most important classes of buffer overflow attacks is code-injection attacks, in which malicious code is injected into a memory area of vulnerable software and eventually executed. In this paper, we propose a simple and effective method for preventing code-injection attacks. The basic idea is to adopt a security-enhanced convention of system call invocations in which system call IDs and arguments are “encrypted ” before being passed to the kernel and then “decrypted ” at the beginning of in-kernel procedures. This is achieved with a modified standard library and a kernel module. In environments where the method is applied, injected code is likely to fail in executing system calls because their IDs and arguments are likely to be decrypted into meaningless values. We implemented the method on a Linux/IA-32 machine and measured the performance of real applications including GCC, L ATEX, wget, and Apache. Experimental results showed that the incurred performance overhead ranged from 0.1 % to 15.0%. 1

This thesis has been submitted in partial fulfillment of requirements of an advanced degree at The University of Arizona and is deposited in the University Library to be made available to borrowers under rules of the Library. Brief quotations from this thesis are allowable without special permission, provided that accurate acknowledgement of source is made. Requests for permission for extended quotation from or reproduction of this manuscript in whole or on part may be granted by the head of the major department or the Dean of the Graduate College when in his or her judgment the proposed use of the material is in the interests of scholarship. In all other instance, however, permission must be obtained from the author.

Abstract—Much of modern software development consists of assembling together existing software components and writing the glue code that integrates them into a unified application. The term COTS-Based System (CBS) is often used to describe such applications, for which the components assembled are understood to be Commercial-Off-The-Shelf (COTS) components written by a multitude of independent third parties. The manner of assembly in CBS includes full-source components that are integrated at compile-time, pure-binary libraries incorporated at loadtime, and plugins that are loaded into the application at execution time by the user. Because components have access to system resources, applications may crash due to faulty components or may be compromised by malicious components. In this paper, we ask the question: can hardware support the development and deployment of CBS by providing applications with a trusted platform for managing components and their interactions? We present an architecture that places each CBS component in a hardware-enforced container. The hardware then detects improper usage of system resources (unauthorized memory accesses or denial-of-service) and enables applications to undertake a hardware-supervised recovery procedure. Furthermore, the hardware also maintains a violation record to enable developers to recreate the violation for the purpose of debugging and further development. Taken together, the purpose of the architecture we propose is to enable executing untrusted CBS code on trusted hardware. I.