Agility must be good for testing

Published in volume 20, issue 3, August 2010

This issue has three papers that advance our knowledge in criteria-based test design. The first, Improving the coverage criteria of UML state machines using data flow analysis, by Lionel Briand, Yvan Labiche, and Q. Lin, proposes coverage criteria to design tests from UML state machines. The second, Automatic string test data generation for detecting domain errors, by Ruilian Zhao, Michael R. Lyu, and Yinghua Min, presents a new method for automatically generating strings as test values. The third, Full predicate coverage for testing SQL database queries, by Javier Tuya, Maria José Suárez-Cabal, and Claudio de la Riva, adapts test criteria in a novel way, to testing database queries.

Unless you've been hiding or never look outside of academia, you must have seen the wave of agile processes that have been sweeping the software industry. Is the term agile process just one more of the hundreds of buzzwords that have made the rounds, only to disappear in obscurity, or does it describe something that will have a lasting positive impact on software development?

If you hope for an answer to that question here, I am no fool to make such a rash prediction. I am neither an advocate nor a critic of agile processes, and not knowledgeable enough to be either. My mind is truly open on this subject. Through my students and other industrial contacts, I have talked with numerous people who applied an agile process correctly and thought it helped very much. I have also talked with people who failed miserably with an agile process. Many of them seem to have applied the process incorrectly.

Without a long tutorial, agile processes involve some speedups to older processes and also some changes that are intended to make major quality improvements. Most companies I have seen fail left out two things: they did not refactor and they did not document. Both are crucial for quality and the companies leave them out to "save time." But leaving those out means after three or four maintenance cycles, the software is in danger of turning into a huge unmaintainable mess.

But these issues are well known and discussed by people who are far more knowledgeable about process than I am. My interest is more personal. Whether agile processes are good for software development or, I like them! I like them for the simple reason that they put more emphasis on testing.

Perhaps my favorite new term is Developer Testing. Teachers and researchers have been advocating more "unit testing" for decades, but agile processes emphasize "developer testing," where developers are responsible for testing their own classes before integrating them into the rest of the system. Get it? Developer testing, not unit testing. Okay, I don't see the difference either ... but the new term works! In the last five years hundreds of software companies have been pushing their programmers to do a better job of developer testing. And the most widely used testing tool is probably JUnit, a simple unit test automation framework that looks a lot like semester projects I assigned in the early 1990s.When practitioners heard about my 1988 PhD dissertation on automatic (unit) test data generation, they often asked how it worked on "real" programs, but now industry needs all our unit testing knowledge.

Another major emphasis of many agile processes is Test Driven Development (TDD). TDD uses tests as requirements, and programmers go through a cyclical development process where each cycle adds enough software to satisfy one more test.

Between developer testing and TDD, software testing assumes a much more prominent place in software development. In particular, testing is moved from primarily being a downstream activity (system testing after implementation) to an upstream activity, integrated with requirements and design. Whether this is good for software development is still an open question, but it must be good for software testing!

A common problem among companies in my area is that they switch to agile processes, then realize they do not have enough testing expertise. If the programmers know little about testing, how can they do well at unit testing? Very few universities teach testing to undergraduates, so most professionals have little testing knowledge. If the testers know the domain but not much about testing or software, the TDD tests will be incomplete. This is an opportunity for educators and researchers.

Another issue is that some companies will develop tests for TDD, then use those same tests as system tests. All the TDD books say not to do that, but the apparent potential for savings is simply too attractive for some companies to resist. Unfortunately, TDD tests will not be complete or to do things like test for subtle interactions among features.

So what does this mean for researchers and educators? For many years, I felt like I was trying to teach the proverbial farmer a better way to plow only to hear him say "I already don't plow as good as I know how." Now, industry doesn't know how to test as well as they need, which means they need our ideas and inventions more than ever!

Jeff Offutt
10 August 2010