ISA 673, Operating Systems' Security - Sping 2010

[Class Schedule (see Class Website)]

Instructor: Angelos Stavrou
Lecture: Wednesdays 7:20 - 10:00pm
Innovation Hall, room 223
Office Hours: Wednesday 5:00 - 7:00pm and by appointment
Office: Research I, Rm 437
Email: astavrou(_)

Teaching Assistant: Sharath Hiremagalore
Research I, Rm 439
Office Hours: Thursday, 4:00 – 6:00pm
Email: shiremag(_)

Course Description:

This course covers both fundamentals and advanced topics in operating system (OS) security. We will study OS level mechanisms and policies and how they relate to mitigating and defending against real-world attacks on computer systems, including self-propagating worms, large-scale botnets, and advanced malware. Basic OS security techniques such as logging, system call auditing, address space randomization, memory protection, virtual machine introspection (VMI) will be discussed. Recent advanced techniques such as host-based instrusion detectionsystem randomization, vulnerability fingerprinting, and virtualization will also be introduced.

Topics Covered:

  1. Introduction
    1. Operating Systems (OS)
    2. Types of Threats
    3. Basic OS Security Mechanisms
  2. Understanding the Threats
    1. Malware Taxonomy
    2. Viruses
    3. Worms
    4. Rootkits
    5. Defense -- An Overview
  3. Logging, Auditing, and Recovery
    1. Log Generation
    2. Log Auditing
    3. Log-based Recovery
  4. OS-level Memory Protection
    1. Review of OS Memory Management
    2. NX Bit
    3. Randomizatio
  5. Virtualization Technology and Applications
    1. Virtualization Taxonomy
    2. Security Applications
    3. Virtual Machine Introspection
  6. Vulnerability Analysis
    1. Vulnerability Classification
    2. Defense against Known Vulnerabilities
    3. Defense against Unknown (0-day) Vulnerabilities
  7. Malware Capture and Analysis (Honeypots and Honeyfarm)
    1. Honeypot Taxonomy
    2. Recent Honeypot Advances
    3. Deployment and Liabilit
  8. Advanced Topic -- Malware
    1. Polymorphic Malware
    2. Malware Packers and Javascript Encoders
    3. Analyzing Malware with PIN & IDA Pro
  9. Advanced Topic -- Rootkits
    1. Rootkit Basics
    2. Advanced Rootkit Techniques
    3. Rootkit Defense
  10. Advanced Topic -- Botnets


CS571 and ISA 562; or permission of instructor. The coursework will include substantial programming projects; in order to be able to complete the projects, the students must be comfortable with C/C++.


Professional Linux Kernel Architecture, Wolfgang Mauerer, John Wiley and Sons, New York, NY, 2008.
Available by: [Willey] [Amazon]

Understanding the Linux Kernel, Third Edition Daniel P. Bovet Marco Cesati ISBN-10: 0596005652 ISBN-13: 978-0596005658 O'Reilly Media Available by: [Online for GMU] [O' Reilly] [Amazon]

Modern Operating Systems, 3/E Andrew S. Tanenbaum. ISBN-10: 0136006639 ISBN-13: 9780136006633 Prentice Hall Available by: [GMU Bookstore] [Prentice Hall] [Amazon]


The students must achieve a total score of at least 90 (out of 100) to be considered for an A. This class is an upper-level class and is geared towards understanding the fundamental concepts behind Security for Computer systems. The students will be expected to participate in large projects under the guidance of the instructor.

Computer Accounts:

All students should have accounts on the central Mason Unix system (also known as
and on IT&E Unix cluster (Instructions and related links are here). Please read the FAQ if you have any questions. Students can work in IT&E computer labs for programming projects during the specified hours.

Please read the University's Academic Honesty Page and GMU Honor Code.

Disability Statement

If you have a documented learning disability or other condition that may affect academic performance you should:
1) Make sure this documentation is on file with the Office of Disability Services.
All academic accommodations must be arranged through the ODS.
2) Talk with me to discuss your accommodation needs.

Other Usefull Resources
Writing Center: A114 Robinson Hall; (703) 993-1200;
University Libraries: “Ask a Librarian”
Counseling and Phychological Services (CAPS): (703) 993-2380;
University Policies: The University Catalog,, is the central resource
for university policies affecting student, faculty, and staff conduct in university affairs.



Class Schedule

Week & Date
Course Lectures & Readings (Tentative)

Week 1, Jan. 20

Introduction and Class Mechanics [Lecture pdf]

Week 2, Jan. 27
Understanding the Security Threats [Lecture pdf]
Week 3, Feb. 3

Operating System Logging Auditing & Recovery [Lecture pdf]

Week 4, Feb. 10
Laboratory I: Debbugging the Linux Kernel
Week 5, Feb. 17
OS-level Memory Protection - Assignment of Project I
Week 6, Feb. 24
Virtualization Technology and Applications
Week 7, Mar. 3
Laboratory II: Security using Virtualization Technologies (XEN, VMWare)
Week 8, Mar. 10
Spring Break, No Classes
Week 9, Mar. 17
Team Project Presentations
Week 10, Mar. 24
Analysis of Current OS and Application Vulnerabilities - Assignment of Project II
Week 11, Mar. 31
Understanding Malware / Malware Capture and Analysis (Honeypots and Honeyfarm)
Week 12, Apr. 7
Laboratory III: Malware Packers and Javascript Encoders - Malware Analysis
Week 13, Apr. 14
Beyond Application Penetration - Rootkits
Week 14, Apr. 21
Virtual Machine Introspection
Week 15, Apr. 28
Network Analysis of Botnets
Week 16, May 5 Team Project Presentations