This should be provided to all reviewers for Software Testing, Verification and Reliability.

A Short Guide to Reviewing Papers for STVR
To review a paper for STVR, simply answer the following five questions:
  1. What are the major results?
  2. Are they correct?
  3. Are they new?
  4. Are they clearly presented?
  5. Are they worth publishing?

I suggest the following criteria for reviews.

ACCEPT

If the major results in the paper are correct, new, clearly presented, and worth publishing, the paper should be accepted as is.

MINOR REVISION

If the major results are correct and new, but not clearly presented and the needed changes are clear enough to be made without further reviewing, a minor revision should be recommended.

MAJOR REVISION

If the major results are correct (or almost) and new, but either further research is needed to be convincing or the changes needed to clear up the presentation are so significant that they must be rechecked, a major revision should be recommended.

REJECT

If the results are incorrect, not new, not significant or cannot be understood from the presentation, the paper should be rejected.

The real point of a review is that you are being asked to make an “assessment” of the paper. There are four possible assessments. An assessment is a somewhat subjective evaluation of the paper, but based on objective “assertions.” The review should contain a list of assertions, or factual statements about the paper, that support the assessment. The reviewing editor’s job is to look at the assertions from all reviewers and decide if they support the assessments, then form an aggregate assessment and make a recommendation to the editor-in-chief.

A good review will have a detailed discussion of all flaws in the paper, and also a clear statement of what must be changed for the paper to be considered acceptable. A good review also must be respectful to the authors. Reviews that are unclear or too short will not be returned to the authors. Reviews that are not written in a professional and respectful manner may be modified, returned to the reviewer, or not sent to the authors.

For more details on my philosophy behind reviewing papers, please see my editorial in the September 2007 issue of STVR.

Jeff Offutt
Editor in Chief, STVR
3 September, 2007
=======================================
Wiley’s Software Testing, Verification and Reliability
Useful research in making better software.
http://www.interscience.wiley.com/journal/softwaretesting
=======================================