ISA 673, Operating Systems' Security - Sping 2012

[Class Schedule] [Projects] [OS Programming Resources]

Instructor: Angelos Stavrou
Lecture: Wednesdays 7:20 - 10:00pm
Innovation Hall, room 222
Office Hours: Wednesday 4:30 - 6:30pm and by appointment
Office: Research I, Rm 437
Email: astavrou(_)

Teaching Assistant: Brian Schulte
Office Hours: TBD

Course Description:

This course covers both fundamentals and advanced topics in operating system (OS) security. We will study OS level mechanisms and policies and how they relate to mitigating and defending against real-world attacks on computer systems, including self-propagating worms, large-scale botnets, and advanced malware. Basic OS security techniques such as logging, system call auditing, address space randomization, memory protection, virtual machine introspection (VMI) will be discussed. Recent advanced techniques such as host-based intrusion detections, system randomization, vulnerability fingerprinting, and virtualization will also be introduced.

Topics Covered:

  1. Introduction
    1. Operating Systems (OS)
    2. Types of Threats
    3. Basic OS Security Mechanisms
  2. Understanding the Threats
    1. Malware Taxonomy
    2. Viruses
    3. Worms
    4. Rootkits
    5. Defense -- An Overview
  3. Logging, Auditing, and Recovery
    1. Log Generation
    2. Log Auditing
    3. Log-based Recovery
  4. OS-level Memory Protection
    1. Review of OS Memory Management
    2. NX Bit
    3. Randomization
  5. Virtualization Technology and Applications
    1. Virtualization Taxonomy
    2. Security Applications
    3. Virtual Machine Introspection
  6. Vulnerability Analysis
    1. Vulnerability Classification
    2. Defense against Known Vulnerabilities
    3. Defense against Unknown (0-day) Vulnerabilities
  7. Malware Capture and Analysis (Honeypots and Honeyfarm)
    1. Honeypot Taxonomy
    2. Recent Honeypot Advances
    3. Deployment and Liabilities
  8. Advanced Topic: Malware
    1. Polymorphic Malware
    2. Malware Packers and Javascript Encoders
    3. Analyzing Malware with PIN & IDA Pro
  9. Advanced Topic: Rootkits
    1. Rootkit Basics
    2. Advanced Rootkit Techniques
    3. Rootkit Defenses
  10. Advanced Topic: Botnets


CS571 and ISA 562; or permission of instructor. The coursework will include substantial programming projects; in order to be able to complete the projects, the students must be comfortable with C/C++ or Java.


Professional Linux Kernel Architecture, Wolfgang Mauerer, John Wiley and Sons, New York, NY, 2008.
Available by: [Willey] [Amazon]

Understanding the Linux Kernel, Third Edition Daniel P. Bovet Marco Cesati ISBN-10: 0596005652 ISBN-13: 978-0596005658 O'Reilly Media Available by: [Online for GMU] [O' Reilly] [Amazon]

Modern Operating Systems, 3/E Andrew S. Tanenbaum. ISBN-10: 0136006639 ISBN-13: 9780136006633 Prentice Hall Available by: [GMU Bookstore] [Prentice Hall] [Amazon]

  • Class Projects: 80%
  • Class Presentations: 15%
  • Class Participation: 5%
  • No Midterm or Final

The students must achieve a total score of at least 90 (out of 100) to be considered for an A. This class is an upper-level class and is geared towards understanding the fundamental concepts behind Security for Computer systems. The students will be expected to participate in large projects under the guidance of the instructor.

Computer Accounts:

All students should have accounts on the central Mason Unix system (also known as
and on IT&E Unix cluster (Instructions and related links are here). Please read the FAQ if you have any questions. Students can work in IT&E computer labs for programming projects during the specified hours.

Please read the University's Academic Honesty Page and GMU Honor Code.

Disability Statement

If you have a documented learning disability or other condition that may affect academic performance you should:
1) Make sure this documentation is on file with the Office of Disability Services.
All academic accommodations must be arranged through the ODS.
2) Talk with me to discuss your accommodation needs.

Other Usefull Resources
Writing Center: A114 Robinson Hall; (703) 993-1200;
University Libraries: “Ask a Librarian”
Counseling and Phychological Services (CAPS): (703) 993-2380;
University Policies: The University Catalog,, is the central resource
for university policies affecting student, faculty, and staff conduct in university affairs.


- Android Kernel Projects (Malware + Functionality)
- Malware Related Projects
- Surveilance Related Projects
- Power Management Related Projects
- Defense/Protection Related Projects
- Logging and Forensics Related Projects

(See Detailed Project Descriptions [Here])

Class Schedule (Tentative)

Week & Date
Course Lectures & Readings (Tentative)

Week 1, Jan. 25

Introduction and Class Mechanics [Lecture pdf]

Week 2, Feb. 1

Understanding the Security Threats

Managed Code Rootkits [PDF] [HTML] [Java Rootkits]
Erez Metula, 2BSecure, BHUSA, 2009.

Back|track Linux Distribution [HTML]
Durzosploit [HTML]

Week 3, Feb. 8

Memory Exploitation & Protection

Practical Windows XP/2003 Heap Exploitation [PDF] [Demo]
John McDonald Chris Valasek IBM ISS X-Force Research, BHUSA, 2009.

Heaps About Heaps [PDF]
Moore, Brett, SyScan 2008.

Understanding and bypassing Windows Heap Protection [PDF]
Waisman, Nicolas, SyScan 2007.

Heap Feng Shui in JavaScript [HTML]

NOZZLE: A Defense Against Heap-spraying Code Injection Attacks [PDF]

LAB Android Introduction [PDF]
Google Android SDK [HTML]
Developer's Guide [HTML]

Week 4, Feb. 15


Week 5, Feb. 22

Android Programming Model
Google I/O 2009 [Video]
Android Emulator [HTML]
Android Debug Bridge [HTML]

LAB Debbugging the Android Applications

Fundamentals [HTML]
Notepad Tutorials [HTML]
Sample Applications:
Bluetooth Chat [Link] (P2P for two users, can we extend it?)
Android Global Time [Link]

Semantically Rich Application-Centric Security in Android
Machigar Ongtang, Stephen McLaughlin, William Enck, and Patrick McDaniel.
(ACSAC), December 2009.[PDF]

Week 6, Feb. 29

Project I Discussion & Assignments

Open Lab - Kernel and Application Development for Android

Android Programming Model using HTML for the UI [HTML]

Android Kernel Programming How To [HTML]

IBM's Tapping into Android Sensors' Page [HTML]

Week 7, Mar. 7

Exploits for that target the Internet Browsers

Internet Explorer turns your personal computer into a public File Server [PDF]
Jorge Luis Alvarez Medina, Blackhat 2010

Neat, New, and Ridiculous Flash Hacks [PDF]
Mike Baily, Blackhat 2010

Week 8, Mar. 15
Spring Break, No Classes
Week 9, Mar. 21

Security using Virtualization Technologies [PDF]

The Role of Virtualization in Embedded Systems [PDF]
Gernot Heiser

Project I Discussions (Cont)

Week 10, Mar. 28

Analysis of Current OS and Application Vulnerabilities

Android Application Debugging [HTML]
Android Kernel Debbugging [PDF]
Creating a keyboard logger using Common Tasks and How to Do Them in Android [HTML]

Week 11, Apr. 4
Understanding Malware / Malware Capture and Analysis (Honeypots and HoneyClients) [PDF]

Creating a keyboard logger using Common Tasks and How to Do Them in Android [HTML]
Week 12, Apr. 11
Malware Packers and Javascript Encoders - Malware Analysis

The Art of Unpacking
Mark Vincent Yason, IBM Internet Security Systems, Blackhat 2007
Week 13, Apr. 18
Malware Packers and Javascript Encoders - Malware Analysis II

Circumventing Automated JavaScript Analysis [PPT]
Billy Hoffman, HP Web Security Research Group, Blackhat 2009

Unpacking and decrypting malware [PDF]
Jarkko Turkulainen, F-Secure Corporation, Blackhat 2009
Week 14, Apr. 25
Malware Defenses for Smartphones

Google Android: A State-of-the-Art Review of Security Mechanisms [PDF]
A. Shabtai et al.
Week 15, May 2
Final Lab - Open Discussion and Presentation Preparation
Week 16, May 9 Team Project Presentations I
Week 17, May 16 Team Project Presentations II