An analysis of Jester based on published papers, April 2005:

The Jester developers don't seem to really get mutation. The power of mutation depends on the mutation operators that create mutants of the program. If a tool implements good operators, engineers get good tests, if the tool implements poor operators, they get trash.

The published papers reports on two operators that Jester implements: (1) exchange 0s and 1s and (2) replace predicates with TRUE and FALSE. Experimental research has found that exchanging 0s and 1s turns out to be almost useless because any input will find them. This is known as an "unstable" operator. Replacing predicates gets branch testing, no more no less.

So Jester turns out to be a very expensive way to apply branch testing!

Mutation tools that implement stronger operators will help the test engineer find much stronger tests.

Soapbox time: This is indicative of a disturbing trend in software these days. On the positive side, I see industry paying more attention to issues of quality. On the negative side, I see a real lack of knowledge about underlying theory in software engineering. It's as if developers have recently decided to apply engineering principles, but they're trying to recreate all the ideas that researchers have built over the past 20 years, without learning what's out there. It's as if all they know about software engineering is what's covered in a one semester, broad-brush junior level CS course using an introductory book like Sommerville's or Schach's ...

wait a minute, duh ... that's all we teach most CS majors about Engineering Software!

I think of someone who may someday use Jester, think "wow, mutation doesn't work very well!", and I feel very disappointed ...

Jeff Offutt
April 2005