ISA 673 Operating Systems Security

Fall 2017

Time and Place: Wednesday 7:20~10:00pm at Art and Design Building L008

Instructor: Dr. Xinyuan (Frank) Wang

Office: Room 5331, Nguyen Engineering Building

Office phone: (703) 993-9461

Office hours: Wednesday 5:30-7:00pm or by appointment

Email: xwangc at gmu dot edu

Teaching Assistant: None

Office hours: None

Email: None


Course Description

The objective of this course is to provide an in-depth introduction to the security issues - including vulnerabilities, threats, exploits and defense mechanisms in operating systems. Topics covered include: logging, auditing, address space randomization, memory protection, virtual machine introspection (VMI), malware and malware immunization. The course emphasizes on real world vulnerabilities (e.g., buffer overflow), threats, exploits (e.g., gaining remote shell) and defense (e.g., malware immunization). In addition, the course brings the state of the art of operating system security to students and expose them to open problems (e.g., rootkit, malware defense) in operating systems security.

Course Outcomes

  • First hand experience in operating systems security.
  • A knowledge of the state of the art of operating systems security.
  • A deep understanding on security vulnerabilities, exploits and defense, as well as the technical challenges and fundamental limitations of existing operating system security solutions.
  • Preparation to become either a capable practitioner or independent researcher in operating systems security.

Course Prerequisite

CS 571 and ISA 562; or permission by the instructor.

The students are expected to have good understanding on Unix/BSD/Linux operating system internals (e.g. system call internals, run-time memory organization, assembly language of x86). Proficiency in C programming is essential in order to be successful in the course projects.

Textbook and Readings

There is NO textbook for this course. The course is based on current research papers and technical reports!

Reference Books

Class Schedule (click to see)

Class schedule is tentative and subject to change. Please check frequently.


There will be NO written exams :)

However, there are 3 or 4security lab assignments to be done individually, a research assignment with presentation and a open lab project to be done by small teams.

Course Lab Assignments


Research Assignment


Course Project


Class Discussion Participation


The final grade is computed according to the following rules:

·        A+: >= 95%; A: [90%, 95%); A-: [85%, 90%)

·        B+: [80%, 85%);  B: [75%, 80%); B-: [70%, 75%)

·        C+: [66%, 70%); C: [63%, 66%); C-: [60%, 63%)

·        D+: [56%, 60%); D:[53%, 56%); D-: [50%, 53%)

·        F: < 50%.

Academic Integrity

All students are required to follow all university, school and department policies regarding academic integrity. Violator of the Honor Code will result in a grade of F for the course, as well as any penalties imposed by the university and/or the CS department.

Disability Statement

If you have a documented learning disability or other condition that may affect academic performance you should: 1) make sure this documentation is on file with the Office of Disability Services (SUB I, Rm. 222; 703-993-2474; to determine the accommodations you need; and 2) talk with me to discuss your accommodation needs. All academic accommodations must be arranged through the ODS.


This course contains materials provided by Dr. Xuxian Jiang (N. C. State University) .