I am drafting this editorial from the second IEEE International Conference on Software Testing, Verification and Validation (ICST 2009), where I am serving as program co-chair. STVR has committed to publishing a special issue each year of the best papers published in this important new testing conference. Since this editorial came due during the conference, I decided to write about conference management.
In the 1990s, technical program committee (TPC) meetings were almost always in person, part of the TPC meeting for one or two days to choose papers to accept. In recent years, online technical program committee meetings have become more common. I have seen both versions done well, and I have seen significant mistakes with both versions. Here I present some of my observations and thoughts on why this trend is occurring and how to run online TPC meetings well.
The two models offer significant differences in terms of travel, duration and communication mode. In-person meetings require most people to fly and spend two or three days in a hotel. Online TPC meetings require no travel. In-person meetings usually last one or two days; more time escalates the monetary and time costs beyond feasibility. Online TPC meetings usually last two or three weeks. In-person meetings are synchronous, where every attendee is engaged in the discussion at the same time, face-to-face. Online TPC meetings are asynchronous, where messages are submitted and later read. These differences allow the relative advantages and disadvantages to be analyzed.
In-person TPC meetings are incredibly expensive, both in terms of time and money. These costs were less important in the 1980s, when most TPCs were largely drawn from one continent. However, we now have truly global communities, and flying from my home in Virginia to a meeting in Hong Kong, or from Tokyo to London, is simply too expensive. My observation is that most in-person TPC meetings have 30% attendance or less. The short duration of an in-person meeting means that there is very little time to think during the discussion. The program chairs (PCs) must enforce deadlines of a few minutes per paper, so every decision becomes rushed. There is very little time for extra reading of the paper, thoughtful reflection on the results, or additional reviews to be requested. The result is that more weight is given to TPC members who (1) have lots of travel funds and (2) are more aggressive in a discussion.
An in-person TPC meeting, on the other hand, requires little or no travel. Thus it saves money (as well as the environment). This increases participation; I have seen 90% or more of TPC members join online discussions. The longer time duration and asynchronous discussion can lead to more reflection and thoughtful comments as well as more egalitarian discussions. Comments are recorded, so a side-effect is that offhand inappropriate remarks are less common and much less influential.
However, online meetings come with some risks. While we have more time to participate, it is easy for TPC members to make the meeting low priority, and not join actively in the discussion. It is also easy for TPC members to look only at papers they reviewed; instead of everyone being forced to synchronously focus on a single paper at a time, online meetings allow us to pick and choose which papers to asynchronously focus on. We also lose some depth of communication without facial expressions and body language.
So … can we achieve the benefits of an online TPC meeting without losing the benefits of an in-person meeting? If so, how? I have seen this model applied successfully as well as unsuccessfully. My experience has convinced me that the most important thing is to have continuous, proactive, energetic intervention from the program chairs. Specific best practices include:
The high cost of travel and the continuing growth of the global nature of research means we will continue to have online TPC meetings. Although opportunities for mistakes are plentiful, if done well, an online meeting can be a rich and productive process. Most importantly, it allows decisions to be made on the basis of technical merit.
8 April 2009