•   When: Monday, July 17, 2017 from 11:00 AM to 12:00 PM
•   Speakers: Mohamed Elsabagh
•   Location: ENGR 4201
•   Export to iCal

Software systems are the backbone of modern life as they drive all computerized technologies. The ever-increasing size and complexity of today's systems makes them very challenging to properly design and test, resulting in an incomprehensible attack surface that leads to poor --- or altogether missing --- security countermeasures. Owing to implementation and testing deficiencies, security defenses are deployed at the network and host perimeters to increase cyber coverage against attacks. Unfortunately, the current poor state of systems security bespeaks that perimeter security is not effective, especially with the proliferation of mobile, cloud, and IoT services where the perimeter cannot be clearly defined.

In this dissertation, I offer novel techniques to protect applications against attacks by retrofitting them with runtime shielding layers that defend from within the application itself rather than presuming that malicious inputs are blocked at the perimeter. These layers enable the application to detect and react to errant behavior by making it aware of its benign behavior and legitimate execution flows. I present automatic techniques that defend against the two most common classes of attacks on software systems: Denial-of-Service (DoS) attacks and Code-Reuse Attacks (CRA). The presented techniques directly embed security defenses into program binaries without requiring side information such as source code, debug symbols, annotations, or attack signatures.

The first part of the dissertation presents Radmin and Cogo as two novel systems for early detection of resource DoS attacks at the application level. These are attacks that can result in program termination (crashing) by exploiting specific design and implementation weaknesses that force the program to over-consume or starve for resources. I present Probabilistic Finite Automata (PFA) based algorithms that save valuable system resources by detecting application-level DoS attacks in their early stages. I demonstrate and contrast the effectiveness of Radmin and Cogo on large-scale servers against comprehensive synthetic and real-world attacks.

The second part of the dissertation presents novel systems to defend against Return-Oriented Programming (ROP) attacks and Function-Reuse Attacks (FRA). I present EigenROP as a system for transparent detection of ROP attacks by monitoring runtime program characteristics such as memory locality and reuse distances. Then, I present VCI as a static binary rewriting system that retrofits C++ binaries with protections against FRA. EigenROP and VCI are significantly more precise than state-of-the-art binary solutions and can defeat sophisticated attacks.

The solutions presented in this dissertation raise defenses to a new level, making a strong case for automatic runtime hardening as a promising approach towards effectuating resilient systems that remain constructively functioning under attacks or even after sustaining some damage.