This should be provided to all reviewers for Software Testing, Verification and Reliability.
I suggest the following criteria for reviews.
If the major results in the paper are correct, new, clearly presented, and worth publishing, the paper should be accepted as is.
If the major results are correct and new, but not clearly presented and the needed changes are clear enough to be made without further reviewing, a minor revision should be recommended.
If the major results are correct (or almost) and new, but either further research is needed to be convincing or the changes needed to clear up the presentation are so significant that they must be rechecked, a major revision should be recommended.
If the results are incorrect, not new, not significant or cannot be understood from the presentation, the paper should be rejected.
The real point of a review is that you are being asked to make an “assessment” of the paper. There are four possible assessments. An assessment is a somewhat subjective evaluation of the paper, but based on objective “assertions.” The review should contain a list of assertions, or factual statements about the paper, that support the assessment. The reviewing editor’s job is to look at the assertions from all reviewers and decide if they support the assessments, then form an aggregate assessment and make a recommendation to the editor-in-chief.
A good review will have a detailed discussion of all flaws in the paper, and also a clear statement of what must be changed for the paper to be considered acceptable. A good review also must be respectful to the authors. Reviews that are unclear or too short will not be returned to the authors. Reviews that are not written in a professional and respectful manner may be modified, returned to the reviewer, or not sent to the authors.
For more details on my philosophy behind reviewing papers, please see my editorial in the September 2007 issue of STVR.
Editor in Chief, STVR
3 September, 2007
Wiley’s Software Testing, Verification and Reliability
Useful research in making better software.