CS 695-002
Natural Language Processing (Special Topics)
Instructor
Antonios Anastasopoulos (antonis [at] gmu [dot] edu)
Office Hours: TBD.
Meets
Mondays, 4:30 to 7:10 PM, Online.
Course Web Page
TBD -- We will also communicate through Piazza.
Course Description
Computers process massive amounts of information every day in the form of human language. Although they do not understand it, they can learn how to do things like answer questions about it, or translate it into other languages. Neural networks provide powerful new tools for modeling language, and have been used both to improve the state-of-the-art in a number of tasks and to tackle new problems that were not easy in the past. This class will start with a brief overview of NLP and Neural Networks, then spend the majority of the class demonstrating how to apply neural networks to natural language problems.
Each section will introduce a particular problem or phenomenon in natural language, describe why it is difficult to model, and demonstrate recent models that were designed to tackle this problem. In the process of doing so, the class will cover different techniques that are useful in creating neural network models, including handling variably sized and structured sentences, semi-supervised and unsupervised learning, structured prediction, and multilingual modeling.
The class will include assignments culminating in a final project.
Online Classroom Specifics
I'll expect you to attend the class, as we will be having the quizzes synchronously, beyond any presentations and discussions). I'll be using Zoom for video-conferencing. I won't require you to have your video on, but it will be greatly appreciated (noone likes to just talk to a screen). I will try to adjust to your needs and to do so I will poll the class for your preferences in the beginning of the semester as well as throughout -- we're in this together!
Prerequisites
Ideally, (a) Algorithms and Data Structures, (b) Artificial Intelligence or Data Mining, and (c) Probability and Statistics (STAT 344) or equivalent.
Students should be experienced with writing substantial programs in Python. The ideal set of prerequisites includes Algorithms and Data Structures, Artificial Intelligence or Data Mining, and linear algebra and calculus, but please contact the instructor if you have questions about the necessary background.
Class Format
As the class aims to provide skills necessary to familiarize the students with, and to do cutting-edge NLP research, the classes and assignments will be at least partially implementation-focused. In general each class will take the following format:
- Reading: Before the class, you will be given a reading assignment that you should read before coming to class that day. This will usually be an important paper that tackles a specific NLP problem.
- Quiz: At the beginning of class, there will be a short quiz that tests your knowledge of the reading assignment. (These quizzes should be easy if the reading assignment has been completed and understood.)
- Summary/Elaboration/Questions: The instructor will summarize the important points of the reading material, elaborate on details that were not included in the reading while fielding any questions. Finally, new material on cutting-edge methods, or a deep look into one salient method will be covered.
- Code Walk: In some classes we will walk through some demonstration code that implements a simple version of the main concepts presented in the reading material.
Grading
Your final grade will be dependent on the in-class individual quizzes (20%) and the assignments/project (80%). There will be no final exam.
Quizzes: Worth 20% of the grade. Your lowest 2 quiz grades will be dropped. If you are sick or traveling on business (e.g. to a conference, for a job interview, or delayed in return due to visa issues), send a doctor's note or evidence of the reason for being away to the instructor within a week of the absence, and you will be excused. I expect excused quizzes to be relatively rare, and if you'll be away for more than, e.g. 2 classes over the semester, please consult in advance. (Of course, given the current situation, the requirements might be adjusted as the semester unfolds; when in doubt, just email the instructor)
Assignments: There will be 4 assignments, worth respectively 10%, 10%, 20%, 40% of the grade. Brief assignment details (more to follow in the class webpage):
- Assignment 1: Text Classifier and Initial Interest Survey
You will be asked to implement a text classifier (or a baseline translation system) from scratch.
There will be an initial questionnaire regarding what task you are interested in tackling for the final project.
- Project Milestone 1: Project Proposal and Literature Survey
This is Checkpoint 1 of your project: it involves a proposal of a project topic and a literature survey regarding this topic. In the survey, explain the task that you would like to tackle in concrete terms, and also cover all of the relevant recent research on the topic. Note that you are still allowed to change topics if you think of something different after assignment 2, but you will need to confirm with the instructor first. Between assignments 1 and 2 make sure to discuss your project ideas with the instructor -- suggested projects will also be posted in the class page.
- Project Milestone 2: State-of-the-art Reimplementation
Checkpoint 2 will involve reproducing the evaluation numbers of a state-of-the-art baseline model for the task of interest with code that you have implemented (mostly from scratch, dependent on the project). In other words, you must get the same numbers as the previous paper on the same dataset.
- Project Milestone 3: Final Project Report
The final project work will be expected to be a novel research contribution that either (1) introduces new techniques for one of the existing tasks in the assignment utilizing one of the more advanced techniques introduced in the class, or (2) tackles a new NLP task (potentially with a neural network model that is motivated by the unique problems posed by the application domain), or (3) presents a novel, meaningful analysis of existing methods and their potential failures.
Late Day Policy: In case there are unforeseen circumstances that don't let you turn in your assignment on time, 5 late days total over the first three assignments will be allowed (late days may not be applied to the final project, assignment 4). Note that the third assignment is harder than the first one, so it'd be a good idea to try to save your late days for the third assignment if possible. Assignments that are late beyond the allowed late days will be graded down one half-grade per day late.
Readings
For each topic/class the instructor will provide a list of papers as suggested readings. One paper will be required reading and will be tested with a quiz (see above).
Students should be able to understand the course content just by following the lecture and by doing the readings. However, the following textbooks serve as good references.
- Jurafsky and Martin, Speech and Language Processing, 2nd edition [amazon]
*Make sure you get the purple 2nd edition of J+M, not the white 1st edition.
- Jurafsky and Martin, Speech and Language Processing, 3rd edition [online]
- Goldberg, Neural Network Methods in Natural Language Processing [publisher] [online primer pdf]
Tentative Schedule
We will try to cover a lot of ground on s in the first weeks in order to lay the foundations for the projects, but then we will focus more on specific NLP tasks and Linguistics phenomena.
| Week |
Date |
Topic |
Homework Due |
| 1 |
8/24 |
Introduction (NLP, Neural Networks) |
|
| 2 |
8/31 |
Language Models, Smoothing, and Recurrent Neural Networks |
|
| 3 |
9/7 |
NO CLASS [Labor Day] |
|
| 4 |
9/14 |
Topic Models, Distributional Semantics and Word Vectors |
Assignment 1 Due 9/18 |
| 5 |
9/21 |
Contextual Representations and Text Classification |
|
| 6 |
9/28 |
Alignment, Translation and Encoder-Decoder Models |
|
| 7 |
10/5 |
Attention, Self-Attention, and Variations thereof |
Assignment 2 Due 10/9 |
| 8 |
10/12 |
Unsupervised Learning from lots of text |
|
| 9 |
10/19 |
Morphology and Syntax |
|
| 10 |
10/26 |
POS Tagging, Structured Prediction I |
|
| 11 |
11/2 |
Entity Recognition, Structured Prediction II |
Assignment 3 Due 11/6 |
| 12 |
11/9 |
Parsing and the CKY Algorithm |
|
| 13 |
11/16 |
Summarization |
|
| 14 |
11/23 |
Question Answering and Dataset Biases |
|
| 15 |
11/30 |
Languages of the World, Multimodal, Multitask, and Multilingual Models |
|
| 16 |
12/7 |
Discourse and Dialog |
|
| 17 |
12/14 |
Project Presentations |
Project Report (Assignment 4) Due |
Honor Code
The class enforces the GMU Honor Code, and the more specific honor code policy special to the Department of Computer Science. You will be expected to adhere to this code and policy.
Note to Students
Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, global pandemics, feeling down, difficulty concentrating and/or lack of motivation. All of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of having a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful. GMU services are available, and treatment does work. You can learn more about confidential mental health services available on campus at: https://caps.gmu.edu/. Support is always available (24/7) from Counseling and Psychological Services: 703-527-4077.
Disabilities
If you have a documented learning disability or other condition which may affect academic performance, make sure this documentation is on file with the Office of Disability Services and come talk to me about accommodations. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Services, I encourage you to contact them at ods@gmu.edu.