Excerpts from a conversation in 2003.
From a student:
Sometime when you have a few cycles to spare, could you give me some advice on the difference between a revise-and-resubmit and an outright reject decision? It seems to me that just about anything that is the latter could just as easily be a revise-and-resubmit. However, surely there is some distinguishing characteristics. I realize that there are obvious things that are crap and should go to the dumpster. I'm thinking about this in the context of a review I recently did. I thought that the paper was largely crap, but I did think that the authors had some interesting ideas and that a different paper might work. My recommendation was a revise-and-resubmit but the editor ultimately rejected it (which I'm okay with). The paper as a whole really stunk. I realize that I am asking you one of those meta-questions that has no easy answer. But my own biases are driving me crazy on this.
My wordy answer:
As an Associate Editor, I have to think about precisely this issue on a regular basis.
Parts of the decision are easy. If the ideas are worthless, reject.
If the results are not enough, but useful, reject for journal X and advise to resubmit to journal Y (where Y expects less).
If the ideas are good, but not they have not gone far enough, reject but in a very encouraging way.
Sometimes the ideas are good, the results are enough, but the paper is in such a shambles that a major revision will likely only get the paper to the point where it needs another major revision. In this case, the paper should probably be rejected, because two major revisions are not allowed. HOWEVER, an editor can say: "I reject this paper, but you should go through a major revision and submit a new/different paper." I got that letter on a paper about a year ago, and then used almost the same letter a few weeks ago.
The basic principle is fairly simple: is the paper salvageable? Secondarily, can and will the authors fix it?
The second question is more important for a conference than a journal, because if the answer is no, then the paper has to be rejected. For a journal, I'll reject if I'm confident the paper can't be fixed. For a conference, I'll accept if I'm confident the reviewers will fix it.