ISA 673, Operating Systems' Security - Sping 2013

[Class Schedule] [Projects] [OS Programming Resources]

Instructor: Angelos Stavrou
Lecture: Wednesdays 7:20 - 10:00pm
Nguyen Engineering Building 5358
Office Hours: Wednesday 4:30 - 6:30pm and by appointment
Office: Research Hall, Rm 437
Email: astavrou(_)gmu.edu

Teaching Assistant: Rahul Murmuria
Research Hall, Rm 439
Office Hours: Thursday 4-6PM
Email: rmurmuri_gmu.edu

Course Description:

This course covers both fundamentals and advanced topics in operating system (OS) security. We will study OS level mechanisms and policies and how they relate to mitigating and defending against real-world attacks on computer systems, including self-propagating worms, large-scale botnets, and advanced malware. Basic OS security techniques such as logging, system call auditing, address space randomization, memory protection, virtual machine introspection (VMI) will be discussed. Recent advanced techniques such as host-based intrusion detections, system randomization, vulnerability fingerprinting, and virtualization will also be introduced.

Topics Covered:

  1. Introduction
    1. Operating Systems (OS)
    2. Types of Threats
    3. Basic OS Security Mechanisms
  2. Understanding the Threats
    1. Malware Taxonomy
    2. Viruses
    3. Worms
    4. Rootkits
    5. Defense -- An Overview
  3. Logging, Auditing, and Recovery
    1. Log Generation
    2. Log Auditing
    3. Log-based Recovery
  4. OS-level Memory Protection
    1. Review of OS Memory Management
    2. NX Bit
    3. Randomization
  5. Virtualization Technology and Applications
    1. Virtualization Taxonomy
    2. Security Applications
    3. Virtual Machine Introspection
  6. Vulnerability Analysis
    1. Vulnerability Classification
    2. Defense against Known Vulnerabilities
    3. Defense against Unknown (0-day) Vulnerabilities
  7. Malware Capture and Analysis (Honeypots and Honeyfarm)
    1. Honeypot Taxonomy
    2. Recent Honeypot Advances
    3. Deployment and Liabilities
  8. Advanced Topic: Malware
    1. Polymorphic Malware
    2. Malware Packers and Javascript Encoders
    3. Analyzing Malware with PIN & IDA Pro
  9. Advanced Topic: Rootkits
    1. Rootkit Basics
    2. Advanced Rootkit Techniques
    3. Rootkit Defenses
  10. Advanced Topic: Botnets


CS571 and ISA 562; or permission of instructor. The coursework will include substantial programming projects; in order to be able to complete the projects, the students must be comfortable with C/C++ or Java.


Professional Linux Kernel Architecture, Wolfgang Mauerer, John Wiley and Sons, New York, NY, 2008.
Available by: [Willey] [Amazon]

Understanding the Linux Kernel, Third Edition Daniel P. Bovet Marco Cesati ISBN-10: 0596005652 ISBN-13: 978-0596005658 O'Reilly Media Available by: [Online for GMU] [O' Reilly] [Amazon]

Modern Operating Systems, 3/E Andrew S. Tanenbaum. ISBN-10: 0136006639 ISBN-13: 9780136006633 Prentice Hall Available by: [GMU Bookstore] [Prentice Hall] [Amazon]

  • Class Projects: 80%
  • Class Presentations: 15%
  • Class Participation: 5%
  • No Midterm or Final

The students must achieve a total score of at least 90 (out of 100) to be considered for an A. This class is an upper-level class and is geared towards understanding the fundamental concepts behind Security for Computer systems. The students will be expected to participate in large projects under the guidance of the instructor.

Computer Accounts:

All students should have accounts on the central Mason Unix system mason.gmu.edu (also known as osf1.gmu.edu)
and on IT&E Unix cluster zeus.ite.gmu.edu (Instructions and related links are here). Please read the FAQ if you have any questions. Students can work in IT&E computer labs for programming projects during the specified hours.

Please read the University's Academic Honesty Page and GMU Honor Code.

Disability Statement

If you have a documented learning disability or other condition that may affect academic performance you should:
1) Make sure this documentation is on file with the Office of Disability Services.
All academic accommodations must be arranged through the ODS. http://ods.gmu.edu
2) Talk with me to discuss your accommodation needs.

Other Usefull Resources
Writing Center: A114 Robinson Hall; (703) 993-1200; http://writingcenter.gmu.edu
University Libraries: “Ask a Librarian” http://library.gmu.edu/mudge/IM/IMRef.html
Counseling and Phychological Services (CAPS): (703) 993-2380; http://caps.gmu.edu
University Policies: The University Catalog, http://catalog.gmu.edu, is the central resource
for university policies affecting student, faculty, and staff conduct in university affairs.


- Android Kernel Projects (Malware + Functionality)
- Malware Related Projects
- Surveilance Related Projects
- Power Management Related Projects
- Defense/Protection Related Projects
- Logging and Forensics Related Projects

(See Detailed Project Descriptions [Here])

Class Schedule (Tentative)

Week & Date
Course Lectures & Readings (Tentative)

Week 1, Jan. 23

Introduction and Class Mechanics [Lecture pdf]

Week 2, Jan. 30

Understanding the Security Threats

Managed Code Rootkits [PDF] [HTML] [Java Rootkits]
Erez Metula, 2BSecure, BHUSA, 2009.

Back|track Linux Distribution [HTML]
Durzosploit [HTML]

Week 3, Feb. 6

Memory Exploitation & Protection

Practical Windows XP/2003 Heap Exploitation [PDF] [Demo]
John McDonald Chris Valasek IBM ISS X-Force Research, BHUSA, 2009.

Heaps About Heaps [PDF]
Moore, Brett, SyScan 2008.

Understanding and bypassing Windows Heap Protection [PDF]
Waisman, Nicolas, SyScan 2007.

Heap Feng Shui in JavaScript [HTML]

NOZZLE: A Defense Against Heap-spraying Code Injection Attacks [PDF]

Week 4, Feb. 13

Project I Discussion & Assignments

LAB Android Introduction [PDF]
Google Android SDK [HTML]
Developer's Guide [HTML]

Android Programming Model
Android Emulator [HTML]
Android Debug Bridge [HTML]

Introductory LAB Excercises [HTML]

Week 5, Feb. 20

Malware Packers and Javascript Encoders - Malware Analysis

The Art of Unpacking
Mark Vincent Yason, IBM Internet Security Systems, Blackhat 2007

Malware Reverse Engineering Unpacking Binary Packers [PDF]
VMware Image (1.1GB) [Link]

Week 6, Feb. 27

Guest Lecture Dr. Arnur Tokhtabayev

Malware Reverse Engineering: Anti-Debuggers [PDF]

Android API, HAL and SDK [PDF]

LAB Debbugging the Android Applications
Fundamentals [HTML]
Notepad Tutorials [HTML]
Sample Applications:
Bluetooth Chat [Link] (P2P for two users, can we extend it?)
Android Global Time [Link]

Introductory LAB Excercises (cont) [HTML]

Week 7, Mar. 6

Android Permission Model
Semantically Rich Application-Centric Security in Android
Machigar Ongtang, Stephen McLaughlin, William Enck, and Patrick McDaniel.
(ACSAC), December 2009.[PDF]

Understanding Malware / Malware Capture and Analysis (Honeypots and HoneyClients) [PDF]

Creating a keyboard logger using Common Tasks and How to Do Them in Android [HTML]
Open Lab - Kernel and Application Development for Android

Android Programming Model using HTML for the UI [HTML]

Android Kernel Programming How To [HTML]

IBM's Tapping into Android Sensors' Page [HTML]

Week 8, Mar. 13
Spring Break, No Classes
Week 9, Mar. 20

Security using Virtualization Technologies [PDF]

The Role of Virtualization in Embedded Systems [PDF]
Gernot Heiser

Project I Discussions (Cont)

Week 10, Mar. 27

Analysis of Current OS and Application Vulnerabilities

Android Application Debugging [HTML]
Android Kernel Debbugging [PDF]
Creating a keyboard logger using Common Tasks and How to Do Them in Android [HTML]

Week 11, Apr. 3

Malware Defenses for Smartphones

Google Android: A State-of-the-Art Review of Security Mechanisms [PDF]
A. Shabtai et al.

Securing Mobile Devices: Present and Future from Mcafee

Test Report: Anti-Malware solutions for Android from AVTest

Week 12, Apr. 10
Open Discussion and Presentation Preparation
Week 13, Apr. 17
Malware Packers and Javascript Encoders - Malware Analysis II

Circumventing Automated JavaScript Analysis [PPT]
Billy Hoffman, HP Web Security Research Group, Blackhat 2009

Unpacking and decrypting malware [PDF]
Jarkko Turkulainen, F-Secure Corporation, Blackhat 2009
Week 14, Apr. 24

Mobile Device Managers (MDMs)

Magic Quadrant for Mobile Device Management Software by Gartner

Mobility Capability Package - National Security Agency

Week 15, May 1
Team Project Presentations I
Week 16, May 8 Team Project Presentations II

Home -  Publications - Teaching - CV - Contact

Last updated:
Please feel free to send your comments and suggestions to Angelos Stavrou.
© 2010 Angelos Stavrou, Computer Science Department, George Mason University.