This issue contains two papers. Automatic Property based Testing and Path Validation of XQuery Programs, by Jesús M Almendros-Jiménez and Antonio Becerra-Terón, presents a tool that automatically generates tests as XML strings for XQuery programs. (Recommended by Guilio Antoniol.) The second paper, Test Suite Completeness and Black Box Testing, Adilson Luiz Bonifácio and Arnaldo Vieira Moura, is a theoretical study of the issue of completeness in test suites. (Recommended by Hassan Ural.) We did not ask the authors of either paper to cut their papers short.
A common conundrum when reviewing conference papers is what to do with papers when the reviewers split, or with papers that were not reviewed quite favorably enough to be accepted, but not quite negatively enough to be rejected. We are a field that strives for consensus, but what to do when we there is no consensus?
When this happens, someone always proposes to accept the paper as a short paper. It is not quite “good enough” as is, so maybe it would be fine if the authors just cut the bad parts. Or maybe by taking a shorter version than submitted, it makes it clear to all that this is a marginal paper. “Okay, we can publish it, but only if we hold our noses so the smell won't be too bad.” Really, what is the point?
After 30 years of writing papers, reviewing papers, editing papers, and helping to organize conferences in all capacities, I strongly believe accepting “short versions” of papers is always a bad idea.
Why? First, the authors wrote a full-length paper (let’s say 10 pages), not a short paper (let’s say 6 pages). A 6-page paper would be a different paper, and is it really possible to adequately describe the research in 6 pages? As my colleague Lionel Briand said, that takes a borderline paper and makes it worse.
The paper is either a valuable contribution to the conference or it is not—cutting the number of pages does not change that.
Second, it is grossly unfair to authors. The authors wrote a 10-page paper, and are asked to rewrite their research as a much shorter paper. That’s very difficult and because the revision period is always limited, it imposes an unreasonable time burden.
Third, it is deceptive. Unless the call for papers says “some papers may be accepted with the condition that they are cut by 40%,” this change introduces new rules in the middle of a process. Short papers are sometimes requested when there isn’t even a short paper category—meaning the TPC created a new category on the fly. What if someone else wanted to contribute to the conference with a short paper, but didn’t because the option wasn’t advertised?
Fourth, it is unfair to the readers. The new, reduced, paper is almost never re-reviewed, and it is by definition very different from the paper that was submitted and reviewed. Instead of a thorough paper that three experts edited, reviewed, and vetted, readers get a cryptic paper that has not been reviewed. That is, this fundamentally corrupts the peer review process.
Finally, it puts the authors in a very difficult situation. If they publish a shortened version of their paper, what will hiring and promotion committees think? In my university, we ask applicants to put short papers into a separate category and essentially ignore them. Worse, can they take the 6-page version, add the other content back in, and publish it as a “real” paper? Is that double-publishing? Is that a copyright problem? Or they could simply refuse by withdrawing the paper. That's hard in a business where “every marble counts.”
For all these reasons, I will always argue against accepting shortened papers. And my first option when asked to cut my paper will be to withdraw.
Another option with a marginal paper, or a split decision paper, is to shepherd it. In that model, someone from the TPC is assigned to check the revised paper to make sure it was actually improved. Sometimes this process can help make a paper better. Most of the time, however, the shepherd doesn’t do much, and the revised paper is the same as it would have been without the shepherd. I don’t object to the shepherding option, but do not see many advantages either.
Is there a third option? Just accept it! It is just a conference. Obviously, someone on the TPC thought the paper was interesting, or it wouldn’t be controversial during the discussion. So it won’t embarrass the conference or hurt its reputation. The authors have a chance to make it better, and maybe it will add to the conference. If not, so what? Ultimately, if nobody cites the paper in the future, it doesn’t matter.
So I implore you. If you are a member of a TPC, please do not suggest “accepting as a short paper.” If someone else suggests it, please explain that is a bad idea. If you are PC, please do not accept such a suggestion. If you are an author, please have the courage to just say no.
Jeff Offutt
George Mason University
offutt@gmu.edu
3 February 2017