ISA 673, Operating Systems' Security - Fall 2018

[Class Schedule] [Projects] [OS Programming Resources]

Instructor: Angelos Stavrou
Lecture: Wednesdays 7:20 - 10:00pm
Art and Design Building 2026
Office Hours: Wednesday 4:30 - 6:30pm and by appointment
Office: Research Hall, Rm 437
Email: astavrou(_)gmu.edu

Teaching Assistant: TBD
Office: TBD

Office Hours: TBD
Email: TBD

Course Description:

This course will cover both fundamentals and advanced topics in operating system (OS) security. We will study OS level mechanisms and policies and how they relate to mitigating and defending against real-world attacks on computer systems, including large-scale network threats, and advanced malware. Basic OS security techniques such as logging, system call auditing, address space randomization, memory protection, virtual machine introspection (VMI) will be discussed. Recent advanced techniques such as host-based intrusion detection, system randomization, malware behavioral fingerprinting, mobile security, and virtualization will also be discussed.

Topics Covered:

  1. Introduction
    1. Operating Systems (OS)
    2. Types of Threats
    3. Basic OS Security Mechanisms: File System Security, ACLs
  2. Understanding the Threats
    1. Malware Taxonomy
    2. Defenses -- An Overview
  3. Logging, Auditing, and Recovery
    1. Log Generation
    2. Log Auditing
    3. Log-based Recovery
  4. OS-level Memory Protection
    1. Review of OS Memory Management
    2. NX Bit
    3. Randomization
    4. Other Defenses
  5. Virtualization Technology and Applications
    1. Virtualization Taxonomy
    2. Security Applications
    3. Virtual Machine Introspection
  6. Vulnerability Analysis
    1. Vulnerability Classification
    2. Defense against Known Vulnerabilities
    3. Defense against Unknown (0-day) Vulnerabilities
  7. Malware Capture and Analysis 
    1. Honeypot Taxonomy
    2. Recent Honeypot Advances
    3. Deployment and Liabilities
  8. Advanced Topic: Malware
    1. Polymorphic Malware
    2. Malware Packers and Javascript Encoders
    3. Analyzing Malware with PIN & IDA Pro
  9. Advanced Topic: Rootkits
    1. Rootkit Basics
    2. Advanced Rootkit Techniques
    3. Rootkit Defenses
  10. Advanced Topic: Botnets


CS571 and ISA 562; or permission of instructor. The coursework will include substantial programming projects; in order to be able to complete the projects, the students must be comfortable with C/C++ or Java.


Professional Linux Kernel Architecture, Wolfgang Mauerer, John Wiley and Sons, New York, NY, 2008.
Available by: [Willey] [Amazon]

Understanding the Linux Kernel, Third Edition Daniel P. Bovet Marco Cesati ISBN-10: 0596005652 ISBN-13: 978-0596005658 O'Reilly Media Available by: [Online for GMU] [O' Reilly] [Amazon]

Modern Operating Systems, 3/E Andrew S. Tanenbaum. ISBN-10: 0136006639 ISBN-13: 9780136006633 Prentice Hall Available by: [GMU Bookstore] [Prentice Hall] [Amazon]

  • Midterm: 30%
  • Homework: 30%
  • Class Projects: 35%
  • Class Participation: 5%

The students must achieve a total score of at least 90 (out of 100) to be considered for an A. The students will be expected to participate in large projects under the guidance of the instructor.

Computer Accounts:

Our classroom is a computer classroom. We will frequently be using the internet as a means to enhance our discussions. We will also be using the computers for our in-class writing assignments. Please be respectful of your peers and your instructor and do not engage in activities that are unrelated to the class. Such disruptions show a lack of professionalism and may affect your participation grade.

All students should have accounts on the central Mason Unix system mason.gmu.edu (also known as osf1.gmu.edu)
and on IT&E Unix cluster zeus.ite.gmu.edu (Instructions and related links are here). Please read the FAQ if you have any questions. Students can work in IT&E computer labs for programming projects during the specified hours.

Honor Code:
The integrity of the University community is affected by the individual choices made by each of us. Mason has an Honor Code with clear guidelines regarding academic integrity. Three fundamental and rather simple principles to follow at all times are that: (1) all work submitted be your own; (2) when using the work or ideas of others, including fellow students, give full credit through accurate citations; and (3) if you are uncertain about the ground rules on a particular assignment, ask for clarification. No grade is important enough to justify academic misconduct. Plagiarism means using the exact words, opinions, or factual information from another person without giving the person credit.
As in many classes, a number of projects in this class are designed to be completed within your study group. With collaborative work, names of all the participants should appear on the work. Collaborative projects may be divided up so that individual group members complete portions of the whole, provided that group members take sufficient steps to ensure that the pieces conceptually fit together in the end product. Other projects are designed to be undertaken independently. In the latter case, you may discuss your ideas with others and conference with peers on drafts of the work; however, it is not appropriate to give your paper to someone else to revise. You are responsible for making certain that there is no question that the work you hand in is your own. If only your name appears on an assignment, your professor has the right to expect that you have done the work yourself, fully and independently.

Please read the GMU Honor Code and the CS Department Honor Code

Disability Statement

If you have a documented learning disability or other condition that may affect academic performance you should:
1) Make sure this documentation is on file with the Office of Disability Services:
(SUB I, Rm. 4205; 993-2474; http://ds.gmu.edu).

2) Talk with me to discuss your accommodation needs.

Other Useful Resources

Writing Center: A114 Robinson Hall; (703) 993-1200; http://writingcenter.gmu.edu
University Libraries: “Ask a Librarian” http://library.gmu.edu/mudge/IM/IMRef.html
Counseling and Phychological Services (CAPS): (703) 993-2380; http://caps.gmu.edu
University Policies: The University Catalog, http://catalog.gmu.edu, is the central resource
for university policies affecting student, faculty, and staff conduct in university affairs.

Sample Projects

- Android Kernel Projects (Malware + Functionality)
- Malware Related Projects
- Surveillance Related Projects
- Power Management Related Projects
- Defense/Protection Related Projects
- Logging and Forensics Related Projects

Class Schedule (Tentative)

Week & Date
Course Lectures & Readings (Tentative)

Week 1, Aug. 29

Introduction and Class Mechanics

Home -  Publications - Teaching - CV - Contact

Last updated:
Please feel free to send your comments and suggestions to Angelos Stavrou.
© 2018 Angelos Stavrou, Computer Science Department, George Mason University.