Editorial

This issue contains two excellent contributions to the literature on software testing. A formal approach to automatically analyse extra-functional properties in mobile applications, by Espada, del Mar Gallardo, Salmeròn, Panizo, and Merino, generates models of mobile apps based based on their UIs. These models are then used to test for extra-functional properties such as power consumption. (Recommended by Paul Strooper.) Choosing the fitness function for the job: Automated generation of test suites that detect real faults, by Salahirad, Almulla, and Gay, evaluated eight fitness functions on their ability to generate tests that found faults. (Recommended by Rob Hierons.)

Farewell and thanks for all the reviews

This is my last editorial as Editor-in-chief of STVR. I will take a brief look back to the beginning, discuss how the journal and the field has changed while I served, and talk about the current and future challenges.

When I began my service in 2007, it never occurred to me that I would do this for so long. My predecessors, Lee White and Martin Woodward, had created the best journal for research in software testing and I was proud when they asked me to take over. STVR was good for me as a younger scientist. Indeed, 3 of my top 10 papers on Google Scholar were published in STVR.

I started my tenure with very specific goals for the journal, many of which were introduced in my first editorial [1]. STVR was in great shape and poised to become even better, and Lee and I had several conversations about these goals before I became EiC.

1. Speed up response to authors: In the mid-2000s, the software engineering community was worried about how long it took to get first decisions from journals. I had not gotten a first-version review within less than 6 months in years, and it often took more than 12 months. I calculated that 3 months was reasonable and set that as a goal. Through numerous strategies, I am pleased to say that we currently average less than 4 months—I don’t know of any other serious research journal that can match that.
2. Introduce a two-tier system of associate editors: Assigning papers to AEs, who then find reviewers and make recommendations to the EiC, is now common. I learned how to manage that process effectively from John Knight when I was an AE for TSE, and implemented the process in STVR in my first year.
3. Acquire better quality reviews: Reviewing is hard work, but a good quality review can be incredibly valuable for the authors and the reviewers. A good review should encourage good papers, should find the flaws and point them out in a respectful professional way, and should recognize that even reviewers make mistakes. A disrespectful tone, content-free reviews, rude comments, and lack of clarity do not help the authors or the journal. In the early days, I sent reviews back for revision, marked reviewers as “non-useful,” and gave reviewers direct feedback on their reviews. As much as anything else, I tried to make it clear to young scientists that their reviews contribute to their professional reputation.
4. Special issues to increase diversity: Rob and I actively sought out special issues on topics such as model-based testing, mutation analysis, and testing extra-functional properties. We have also long partnered with the IEEE International Conference on Software Testing, Verification, and Validation (ICST) to produce special issues based on excellent papers from that conference.
5. Better communication to authors: I have often been confused by conflicting and unreasonable demands from reviewers. John Knight told me the solution is to make it clear which comments are required and which are optional ... just as in software requirements. I try to write my editor’s comments to clarify exactly which changes are required, and, when appropriate, suggest which changes may be unreasonable. I’ve often spent hours distilling a paper, three reviews, and the AE’s recommendation to one paragraph of guidance to the authors. I’ve had a lot of positive feedback over the years that the effort is worthwhile.

I can characterize those goals simply: I wanted to create the journal I needed when I started my career.

Commitments

An important part of any leadership position is to understand your commitments. Some are obvious—the day-to-day handling of papers. Some are less so. My first commitment was that I would not publish in STVR. That was not required by the publisher, and EiCs of some journals in the 2000s published their own papers. No matter how good the paper was, my view was that nobody would trust a process where an author is also a final decision-maker. The biggest cost for this decision, of course, was borne by my students who could not publish in the most important journal in their field. I maintained this ethical stance at their expense, and I thank them all for it.

I also committed to integrity—to treat every paper fairly, regardless of topic or person. And of course I was immediately tested. In my first six months, we received a paper that had great results, but significant problems and an author who could not navigate our new online submission system. The test was that the author had extended a feud with my PhD advisor by pro-actively going out of his way to review, and reject, several of my papers and proposals. Nonetheless, I shepherded this paper carefully, helped the author improve the presentation, took the revision by email and submitted it myself. It was a good paper. Another early paper was by a senior colleague who I liked and respected. But the paper was sub-par, and I had to reject it. I will always appreciate his gracious response: “It’s okay, Jeff, I should have written a better paper.”

I also committed to quick responses to AEs, reviewers, and authors. I set aside time every week to manage the email floods, often with 12-hour Fridays and more time on the weekends. I wanted to set examples of professional communication and behavior for less experienced colleagues. Ask colleagues who have been around for more than 15 years, and they will agree that our field has become more polite, more professional, and more reasonable. I like to think STVR helped a little.

Changes While I Was EiC

We have gone through significant changes in the last 12 years. When I came on, the journal was moving from a paper-based process to a completely online workflow. Unfortunately, this helped lead to a massive increase in the number of non-research, desk-rejectable, papers. These now account for more than half of our submissions [2]. We have also seen huge growth in software engineering research in general and software testing in particular. This growth has been paralleled by a proliferation of conferences and journals. Testing has also become significantly more important to industry in the past 15 years, and software quality is now viewed as essential to product success.

I also made several innovations. My first was the journal’s tagline: “Useful research in making better software.” Taglines are hard, and yes, I solicited professional help. One of the happiest changes (for me) was when Rob Hierons joined me as co-EiC in 2011. Rob has been an ideal partner and I probably would have quit long ago if he didn’t always have my back. I also started writing regular editorials that went beyond introducing the papers. This habit started with my first introductory editorial [1], followed by a couple of pleas for better quality reviews (one of my goals). These editorials are all archived on my website in a blog format. As mentioned, STVR has also partnered with ICST for special issues since ICST’s first edition in 2008.

Current Problems

We have made progress, but STVR, software testing, and software engineering still have many problems. Journals are getting more low-quality submissions every year. Plagiarism is rampant, as are papers with little or no research content, barely intelligible writing, false claims, and papers that are completely out of scope. It seems to be hard for many authors (and our students!) to tell the difference between a real research journal and a facade journal [3, 2, 4, 5]. Rob and I are more aggressive about rejecting papers, but these papers still clog the system and use too many scarce resources.

As a field, we have far too many conferences. As I discussed previously, this threatens to turn us into an elitist pay-to-publish field [6]. The desire for instant gratification and insane expectations for publishing is causing young scientists eschewing journals for more “popular” conferences. As a junior colleague recently put it: “To get tenure, the least of us has to publish more than the best in the 1990s, and get twice as much money. I can’t publish enough papers if I go to journals.” We are killing our babies!

Future of Software Testing

With all this, I am very positive about the future of software testing. As opposed to the 1990s, our work as researchers and educators is viewed as essential to the success of a fast-growing industry. We matter! We have to be careful though—to take advantage of this trend, we have to actively make sure our research is relevant. If we want to call our research software engineering, we must make sure that our ideas can eventually help real programmers build better software, cheaper.

We also continue to separate from mother-computer science. This trend is healthy for both CS and SE. The same thing happened with Physics and physical engineering fields (civil engineering, mechanical engineering, etc.) and despite much fear, was good for all.

I Couldn’t Have Done This Alone

STVR is a large and diverse team, and the EiC’s role is to serve the team. More than anyone else, I want to thank my long-term partner Rob Hierons. When I found the job getting too big, Rob stepped up without hesitation. He has been in charge of all special issues and handled many regular papers, especially when I was overwhelmed with other work. Most importantly, we have made every decision together. I don’t think I have ever disagreed with Rob, and when he disagreed with me, he was right.

Authors and readers of STVR owe a debt of gratitude to Derek Yates for founding the journal, to Lee White for serving as North American editor for 16 years, and Martin Woodward for serving as editor-in-chief for 15 years. They created and built an excellent journal that has greatly helped the field of software testing. I am grateful for the confidence they showed me in asking me to become EiC.

We also owe a big debt to the dozens of associate editors who have served the journal in the past 12 years. If you think that’s an easy job, I can assure you it’s not! In addition to analyzing and distilling three often conflicting reviews, getting those reviews is harder every year. As conferences eat more and more of the field’s reviewing energy, AEs have gone from inviting 3 reviewers to get 3, to 5, 10, sometimes more than 20! I can’t say how much I appreciate their efforts.

On a personal note, I want to thank my family. I’ve spent many evenings and weekends working on the journal when my wife and kids probably could have thought of something more fun to do. Thanks for bearing with me!

Of course, one of my loudest thanks goes to my successor: Tao Xie, Professor and Willett Faculty Scholar in the Department of Computer Science at University of Illinois at Urbana-Champaign. Tao is a very highly respected and successful member of the software testing research community. He is well known for his energy, sound judgment, good nature, and strong support for young scientists, women, and under-represented minorities. Rob and I are both confident that Tao is the co-Editor in Chief that STVR needs going forward.

@Tao: I’m proud of this journal. I only ask one simple thing: Make it even better.

I also suggest the following stance, which I adapt from Lao Tzu: Authors, reviewers, and associate editors don’t work for you, you work for them.

In closing, my biggest appreciation goes to the hundreds of reviewers who have unselfishly given their time, mental energy, and sound judgment to this journal and the field of software testing. I have articulated how much this service helps the reviewers [3, 7, 8, 9, 10, 11], but you and I both know that reviewing is hard work with meager gratitude.

I developed my trademark closing line in 2007: “Remember that good journals need good papers, but it’s the reviewers who make a journal great!” Those were never just words to me. I’ve reviewed thousands of papers and know that good reviews take time, thought, and multiple iterations. The value of a journal comes from the journey a paper takes from a rough draft to an excellent final paper. That journey is successful only if the reviews help the authors find the best way to present their results.

[1] Introduction and plans for the future (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 17(2), June 2007

[2] What is a facade journal? (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 28(6), August 2018

[3] What is the value of the peer-reviewing system? (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 28(5), July 2018

[4] How can we recognize facade journals? (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 28(7), October 2018

[5] Why do people publish in facade journals? (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 28(8), December 2018

[6] I love journal papers and you should too (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 29(3), April 2019

[7] Standards for reviewing papers (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 17(3), September 2007

[8] Why should I review papers? (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 17(4), December 2007

[9] Non-Expert Reviews Considered Helpful (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 22(3), May 2012

[10] How to Become an STVR Reviewer (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 23(1), February 2013

[11] Is Paper Reviewing a Transaction, a Service, or an Opportunity? (Editorial—Jeff Offutt), Wiley’s journal of Software Testing, Verification, and Reliability, 27(4-5), June 2017

Jeff Offutt
George Mason University
offutt@gmu.edu
20 June 2019