Editorial:
Running a Conference Program Meeting in the 21st Century

I am drafting this editorial from the second IEEE International Conference on Software Testing, Verification and Validation (ICST 2009), where I am serving as program co-chair. STVR has committed to publishing a special issue each year of the best papers published in this important new testing conference. Since this editorial came due during the conference, I decided to write about conference management.

In the 1990s, technical program committee (TPC) meetings were almost always in person, part of the TPC meeting for one or two days to choose papers to accept. In recent years, online technical program committee meetings have become more common. I have seen both versions done well, and I have seen significant mistakes with both versions. Here I present some of my observations and thoughts on why this trend is occurring and how to run online TPC meetings well.

The two models offer significant differences in terms of travel, duration and communication mode. In-person meetings require most people to fly and spend two or three days in a hotel. Online TPC meetings require no travel. In-person meetings usually last one or two days; more time escalates the monetary and time costs beyond feasibility. Online TPC meetings usually last two or three weeks. In-person meetings are synchronous, where every attendee is engaged in the discussion at the same time, face-to-face. Online TPC meetings are asynchronous, where messages are submitted and later read. These differences allow the relative advantages and disadvantages to be analyzed.

In-person TPC meetings are incredibly expensive, both in terms of time and money. These costs were less important in the 1980s, when most TPCs were largely drawn from one continent. However, we now have truly global communities, and flying from my home in Virginia to a meeting in Hong Kong, or from Tokyo to London, is simply too expensive. My observation is that most in-person TPC meetings have 30% attendance or less. The short duration of an in-person meeting means that there is very little time to think during the discussion. The program chairs (PCs) must enforce deadlines of a few minutes per paper, so every decision becomes rushed. There is very little time for extra reading of the paper, thoughtful reflection on the results, or additional reviews to be requested. The result is that more weight is given to TPC members who (1) have lots of travel funds and (2) are more aggressive in a discussion.

An in-person TPC meeting, on the other hand, requires little or no travel. Thus it saves money (as well as the environment). This increases participation; I have seen 90% or more of TPC members join online discussions. The longer time duration and asynchronous discussion can lead to more reflection and thoughtful comments as well as more egalitarian discussions. Comments are recorded, so a side-effect is that offhand inappropriate remarks are less common and much less influential.

However, online meetings come with some risks. While we have more time to participate, it is easy for TPC members to make the meeting low priority, and not join actively in the discussion. It is also easy for TPC members to look only at papers they reviewed; instead of everyone being forced to synchronously focus on a single paper at a time, online meetings allow us to pick and choose which papers to asynchronously focus on. We also lose some depth of communication without facial expressions and body language.

So … can we achieve the benefits of an online TPC meeting without losing the benefits of an in-person meeting? If so, how? I have seen this model applied successfully as well as unsuccessfully. My experience has convinced me that the most important thing is to have continuous, proactive, energetic intervention from the program chairs. Specific best practices include:

• Do not assign too many papers to the TPC members. Nobody can afford the time to write more than about 10 high quality reviews for a conference. If TPC members are assigned 15, 20 or 30 papers, the quality of the reviews will unavoidably be low, and over time, so will the quality of the conference.
• After reviews are in, mark the papers initially to have lots of "open for discussion," and few "probable rejects" and "probable accepts." Making lots of initial decisions will depress the discussion, whereas TPC members will actively discuss papers with no default decision. (At ICST, we took the rule that three positive reviews resulted in a default decision of "probable accept" and three negative reviews resulted in a default decision of "probable reject;" this left the majority of the papers in the middle.)
• Allow TPC names to be associated with comments during the discussion. People are more comfortable talking when they know who they are talking with. Hiding names will depress the discussion.
• Assign a moderator for each paper. Choosing the most negative reviewer will bias the discussion against the paper; choosing the most positive reviewer will bias the discussion in favor of the paper.
• Explicitly ask TPC members to join discussions on papers they did not review.
• Divide responsibility for monitoring papers among the program chairs. But do not take complete ownership. Each PC should feel comfortable monitoring the discussion on any paper.
• Monitor the discussion on every paper and prod the participants when the discussions go stagnant. This is crucially important to the success of an online meeting!
• If the reviewers are deadlocked, invite additional reviewers. ICST often uses steering committee members for this, but other members of the TPC can also help.
• A subtle point is that average opinions often trend downwards during discussions. Reviewers tend to be conservative, and would rather be wrong in voting against a good paper than be wrong in voting for a poor paper. The PCs can watch for this tendency, and add comments to give the group a reality check.
• The PCs must read every review and every comment, and be prepared to ask reviewers to rethink or rewrite reviews.

The high cost of travel and the continuing growth of the global nature of research means we will continue to have online TPC meetings. Although opportunities for mistakes are plentiful, if done well, an online meeting can be a rich and productive process. Most importantly, it allows decisions to be made on the basis of technical merit.

Jeff Offutt
offutt@gmu.edu
8 April 2009