We are entitled to
our own opinions,
but not our own facts.
Syllabus Schedule Papers Project Jeff’s page Birgitta’s page
Last update
January 2019
Experimentation in Computing
Course Syllabus
Spring 2019
SWE 763 Software Engineering Experimentation & CS 795 Computer Science Experimentation (Mason)
IT0912F Software Engineering Experimentation (Skövde)
Collaboration empowered with Piazza
Professors  
Jeff Offutt
offutt +++ gmu.edu
http://www.cs.gmu.edu/~offutt/
ENGR 4430, (+1) 703-993-1654
Class: M 4:30-7:10, Buchanan D023
Birgitta Lindström
birgitta.lindstrom +++ his.se
http://www.his.se/en/about-us/Facts-and-figures/staff/birgitta_lindstrom/
PA520E, (+46) 0500-448368
Experimentation in Computing is being taught as a unique and innovative international collaborative learning opportunity. Lectures and discussions will be asynchronous and the class will virtually merge students from universities in the US and in Sweden.
COURSE STRUCTURE
This course will have three parts: (1) We will study concepts of designing experiments and the peculiar complexities of experimenting in computing; (2) Students and professors will read and discuss experimental research papers from the literature; and (3) Students will carry out a semester-long experimental project, write a paper, and present the results conference-style.

ASYNCHRONOUS DELIVERY
Most of this class will be taught asynchronously. The first meeting will be face-to-face in classrooms. Part 1 will feature recorded lectures that are deployed online. The paper discussions in part 2 will be carried out on the online discussion board Piazza. The students’ papers in part 3 will be posted online and the final presentations will be face-to-face in classrooms.

INTERNATIONAL VIRTUAL CLASSROOM
We will leverage online educational opportunities to virtually merge students from George Mason University and University of Skövde. All groups will use the same discussion board (Piazza), see the same lectures, and read and discuss the same papers on the same schedule. Student presentations will be synchronized with the professor and broadcast to the other groups via DE technology. All lectures and discussions will be in English.

OVERVIEW OF CONTENTS
This course will take a horizontal look at several different research areas in computing, with a focus on empirical validation. Most research in computing must be validated empirically. This course discusses how to design and carry out experiments in computing, with a focus on defining experimental hypotheses, choosing appropriate dependent and independent variables, and designing an experiment that will answer the right question without bias.

MATERIALS
Lectures and papers.

DISCUSSION BOARD USE
Experimentation in Computing will use the Piazza bulletin board for lectures and discussions. Information for accessing our class will be provided during the first meeting. Lectures will be posted on Piazza and joining the discussions is mandatory.

COURSE LEVEL
update 8 January: If you are early in your research career, this course will help you get started with the basics of experimental research, and may well lead to one of your first publications or your eventual PhD research. If you have already started your PhD research, this course should integrate seamlessly with your ongoing research project and help you do the experimental part of your research correctly. If you are an advanced MS student, this course will help you understand how experimental research works and help you decide you are interested in attempting a PhD.

LEARNING OBJECTIVES
  1. Understanding of the scientific process; particularly using the experimental method.
  2. Knowledge of how empirical studies are carried out in computing and software engineering.
  3. Understanding when experimentation is required in computing, and what kinds of questions can be answered using experimentation.
  4. Knowledge of how to control variables and eliminate bias in experimentation.
On completion of the course students will able to design and carry out experiments in ways appropriate for a given problem, and will have skills in analyzing and presenting experimental data. Students will be ready to carry out the empirical portions of their PhD dissertations.

LEARNING PRACTICE
This course will examine and critique experimental techniques to evaluate techniques and processes in computing fields. We will read and discuss papers in the current literature, specifically focusing on the experiments as opposed to the background or the implications of the results.

PAPERS
The first set of papers (section I, papers i through vi) are about experimentation in general. For these, each student must submit a simple summary and evaluation of half of the first set of papers (you can choose which half). Your summary and evaluation should be about a page long and must be posted on the discussion board. Due dates for the summaries of the first set of papers are Friday evening the week the paper is scheduled.

Corrected 28 January: The second set of readings (section II, A, B, C, and D) are short readings about dissemination that we will refer to through the semester.

The third set of papers (section III, papers 1 through 14) are published research papers that present experimental results. Each is assigned to a specific week on the schedule page. Two people will be assigned to write and post a summary evaluation for each experimental result paper. The summaries should: (1) describe the problem in general terms, (2) paraphrase the experimental hypothesis, (3) summarize and critique the design, (4) discuss the conduct of the experiment, (5) explain whether the hypothesis was proved or disproved, and (6) critique the presentation of the paper. Summaries of the second set of papers are due on the bulletin board on Mondays by close of business in Fairfax (6:00 PM EST, midnight in Sweden).

One student will be designated as “dissenter” for each paper and is required to disagree with the posted summaries and provide reasonable arguments. The dissenter must post the counter-arguments on Tuesday by midnight in Sweden.

All students must join in the discussion for each paper throughout the week. All postings, including dissents, must be courteous and professional! Imagine the authors of the paper reading your comments.

PROJECT
Each student will design, conduct, write up, and present a small-scale experiment. A paper will be delivered and the results presented orally to the class. I will suggest a number of projects or students can develop their own related to their research topic. I strongly recommend that GMU students seek help on writing from the GMU writing center, especially if English is not your first language. Their online resources are especially helpful. Numerous papers from past semesters have been published in refereed conferences and led to PhD dissertations.

GRADING STANDARDS
The course has four major components to the grading. At GMU, these percentages will be used to compute a score from 1 to 100, which will then be used to compute a letter grade (A+, A, A-, B+, B, B-, C, F).

I. (25%) Participation via the bulletin board is required. All students are expected to participate energetically in the discussions. Discussion postings must have substantial content. Postings such as “I agree with Mats” are welcome but will not earn credit. Postings such as “Mats thought the number of subjects were too few, but I believe five is enough for this study because ...” will earn credit (even if the professors disagree with the analysis). Students who do not participate in the discussions on a weekly basis will not receive credit for this portion of the grade. Shyness or lack of English proficiency are not valid reasons for staying out of the discussions.

II. (20%) Each student will post summaries and evaluations, and dissensions, as described above under PAPERS. You can “bid” on papers by sending the professors an email and we will try to honor all requests when assigning papers to students. Depending on the numbers, you will only have two or three papers, and it is imperative that you post your summaries on time. The entire class is depending on you.

III. (50%) The experimental project will constitute half the grade, and both the paper and the presentation will be graded. Each student will write and post a short description of the proposed experiment early in the semester and the class will make suggestions for improvements.

IV. (5%) A draft of each paper will be reviewed by three people; a professor and two classmates. We will assign papers for reviews late in the semester.

NECESSARY BACKGROUND
A strong background in computing will be necessary. Although many of the papers will be quite specific in topic, extensive background in these topics will not be necessary—we will extract the experimental techniques and procedures from the papers. Although experimental data must be analyzed appropriately, simple analysis often suffices for this type of research, so this course is not a statistical methods course.

CS 700 and STAT 554 (for Mason students)
SWE 763 is a perfect complement to STAT 554. STAT 554 focuses on data analysis (statistics), while SWE 763 focuses on how to gather data that will answer a hypothesis. CS 700 also covers most of the statistical analysis techniques that would be used in computing experiments.
Top