This paper contains two novel papers about test data generation. Event tree algorithms to generate test sequences for composite Web services, by Andre Takeshi Endo and Adenilso Simao, introduces a new algorithm to generate tests from event sequence models. (Recommended by Hong Zhu.) Coincidental correctness in the Defects4J benchmark, by Rawad Abou Assi, Chadi Trad, Marwan Maalouf, and Wes Masri, evaluates the Defects4J benchmark empirically to find incidents of coincidental correctness. (Recommended by Mauro Pezze.) Neither is based on a prior conference paper and so are journal first papers.
This editorial discusses whether to publish in a journal or a conference. It’s a hard decision for scientists, especially young scientists. The tradeoffs are real and the choice is not always clear. Let’s explore advantages and disadvantages of each model.
The most obvious advantage of publishing in a conference is speed: Papers are published sooner. Deadlines are inflexible and backdated a few months from the conference date. In 2019, ICST’s submission deadline was 12 October 2018 and the conference starts on 22 April 2019—about 6.5 months. ICSE’s submission deadline was 24 August 2018 and the conference starts on 25 May 2019—almost 10 months. Journals usually return first reviews within about 6 months (STVR averages just under 4 months). Then a paper must be revised, reviewed again, and sometimes a third time, before eventually being published. That takes 18 to 36 months, two-thirds of which the paper is being revised by the authors. That’s much faster, especially for young scientists with only a few years to establish their careers.
Another advantage of conferences is the social aspect. Not only do we get to present our results to peers, they get to ask questions face-to-face. Conferences also offer numerous opportunities to socialize with other scientists, making connections that can be invaluable throughout our careers.
Of course another advantage is that travel is fun! Conferences have let me visit many fascinating parts of the world, often using grant funds.
This is related to a disadvantage of conferences: They are expensive! And the expense has been steadily growing as major publishers use conferences as revenue streams to replace journal subscriptions. ICST costs $996 USD this year if you’re not an IEEE member and another $210 USD if you want to go to the reception, banquet, and excursions (significant social events). This is in addition to airfare (often over $1500 USD) and the hotel ($140 USD per night at ICST). Altogether, an international conference can easily cost $3500 USD. These costs have turned research conferences into elitist activities only available to rich scientists.
Time is another cost. Major conferences like ICST, ICSE, and FSE take us away for most of a week, with jetlag having the potential to reduce our effectiveness both during and after the conference. The day-to-day work of teaching classes, advising students, and administrative duties doesn’t go away while we’re gone. It piles up, waiting impatiently for us to return.
But the biggest disadvantage of conferences is the reviewing. Nobody could doubt that conference reviews are shorter, use less expertise, and do less to improve the papers than journal reviews. Technical program committee members often have 10, 15, or sometimes more than 20 papers to review within a few weeks. We simply do not have the time to fully evaluate every aspect of a conference paper.
Following directly from this, conference revisions are often minimal. Sure, make easy fixes, but the paper is accepted and we have little motivation to carry out major rewrites. Some conferences try to “shepherd” papers that need significant revision, but this shepherding is usually nominal. I have thrice had papers shepherded. Two times I never heard from whoever was assigned to shepherd my paper, and the third got no feedback from the shepherd, so submitted final versions without shepherds’ approval. Twice I have been asked to shepherd a paper, but the authors never sent me an update and the conference published their papers anyway (one was still significantly flawed).
Journals also have several advantages. First, journal papers can be longer. Some research can be explained in 8-10 pages plus references, but much research needs additional words to explain complicated ideas fully. In fact, the best and deepest research is usually too complicated for a short conference paper. So if we try to squeeze all of our complex ideas into 10 pages, we lose depth, complexity, and novelty, and favor small incremental research steps.
Journal reviews are also better, often much better. They are certainly longer! I just pulled up reviews of my last five journal and last five conference submissions. The journal reviews average more than three times longer than the conference reviews! Editors are better able to select experts who can really understand the material. Reviewers spend more time and think harder about the work. The back and forth between author, editor, and reviewers allow the process to become a collaboration, where reviewers help the authors find the best possible paper within the original submission.
Better reviews and more revisions mean journal papers are often much better than conference papers. The writing is better, the evaluations are more likely to be sound, and the results have more credibility.
Perhaps the length and opportunity to improve the quality of papers directly leads to my next point: Journal papers have more impact. Of my 10 most cited papers on Google Scholar, eight are in journals. Out of curiosity, I looked up the Google Scholar pages for 10 leading software testing researchers (all with impact factors above 45) and checked their top 10 publications. Seven scientists have a majority of their top 10 papers in journals, two are five and five, and one has only two of the top 10 papers in journals. Overall, 68% of these high-impact papers were published in journals.
Before I summarize, let me emphasize that I love conferences. I love travel. I love spending time with my colleagues; making new friends and renewing old friendships. But as far as raw quality of research? Journal papers are far better. The inherent difficulties of reviewing many papers in a short time frame means that conferences make more mistakes by rejecting good papers and publishing bad papers. Conferences are also inherently elitist. Papers are only published if the author registers, so conferences have turned into a “pay-to-publish” enterprise, a model that is widely criticized when used by journals. These costs have made research conferences only available to rich scientists. Why do conferences have so few attendees from South America, Africa, or much of Asia? Scientists cannot afford to spend two or three months’ salary on a conference.
I am not advocating doing away with conferences. They serve several valuable purposes. I am advocating that we have fewer conferences. Combine some. Eliminate some. And for goodness sake, let’s stop taking them so seriously. Driving acceptance rates down to 15%, 10%, or 5% does not make a conference better, only poorer by rejecting more good papers. I wish we could take conference papers less seriously (like most academic fields), and lean toward including more good papers even if it means a few more weak papers. And send your best work to journals! Is your goal to travel or to have more impact on the field?
Jeff Offutt
George Mason University
offutt@gmu.edu
2 March 2019