Introduction to Software Testing

Paul Ammann and Jeff Offutt


This table of contents is for an advanced graduate research-oriented course on software testing. In fact, one of the problems that we hoped to solve with this book is that research students have had to read hundreds of papers in dozens of journals and conferences spanning several decades to reach minimum competence in the field. Thus many young researchers in testing had huge gaps in their knowledge. Sections that are not recommended for this course are grayed out. This includes every section of the book, but the most important are the "Bibliographic Notes" sections at the end of each chapter. We have tried to capture the development of the ideas through the literature. Of course, getting all the important references is an all but impossible task, but with 366 citations to the literature, we have tried hard to come close.


Chapter List
1. Introduction
2. Graph Testing
3. Logic Testing
4. Input Space Partitioning
5. Syntax Testing
6. Practical Considerations
7. Engineering Criteria for Technologies
8. Building Testing Tools
9. Challenges in Testing Software

Part I: Overview
    1. Introduction
       1.1 Activities of a Test Engineer
          1.1.i Testing Levels Based on Software Activity
          1.1.ii Beizer's Testing Levels Based on Test Process Maturity
          1.1.iii Automation of Test Activities
       1.2 Software Testing Limitations and Terminology
       1.3 Coverage Criteria for Testing
          1.3.i Infeasibility and Subsumption
          1.3.ii Characteristics of a Good Coverage Criterion
       1.4 Older Software Testing Terminology
       1.5 Bibliographic Notes

Part II: Coverage Criteria
    2. Graph Testing
       2.1 Overview
       2.2 Graph Coverage Criteria
          2.2.i Structural Coverage Criteria
          2.2.ii Data Flow Criteria
          2.2.iii Subsumption Relationships among Graph Coverage Criteria
       2.3 Graph Coverage for Source Code
          2.3.i Structural Graph Coverage for Source Code
          2.3.ii Data Flow Graph Coverage for Source Code
       2.4 Graph Coverage for Design Elements
          2.4.i Structural Graph Coverage for Design Elements
          2.4.ii Data Flow Graph Coverage for Design Elements
       2.5 Graph Coverage for Specifications
          2.5.i Testing Sequencing Constraints
          2.5.ii Testing State Behavior of Software
       2.6 Graph Coverage for Use Cases
          2.6.i Use Case Scenarios
       2.7 Representing Graphs Algebraically
          2.7.i Reducing Graphs to Path Expressions
          2.7.ii Applications of Path Expressions
          2.7.iii Deriving Test Inputs
          2.7.iv Counting Paths in a Flow Graph & Max Path Length
          2.7.v Minimum Number of Paths to Reach All Edges
          2.7.vi Complementary Operations Analysis
       2.8 Bibliographic Notes
    3. Logic Testing
       3.1 Overview-Logic Predicates and Clauses
       3.2 Logic Expression Coverage Criteria
          3.2.i Active Clause Coverage
          3.2.ii Inactive Clause Coverage
          3.2.iii Infeasibility and Subsumption
          3.2.iv Making a Clause Determine a Predicate
          3.2.v Finding Satisfying Values
       3.3 Structural Logic Coverage of Programs
          3.3.i Predicate Transformation Issues
       3.4 Specification-Based Logic Coverage
       3.5 Logic Coverage of Finite State Machines
       3.6 Disjunctive Normal Form Criteria
       3.7 Bibliographic Notes
    4. Input Space Partitioning
       4.1 Input Domain Modeling
          4.1.i Interface-Based Input Domain Modeling
          4.1.ii Functionality-Based Input Domain Modeling
          4.1.iii Identifying Characteristics
          4.1.iv Choosing Blocks and Values
          4.1.v Using More than One Input Domain Model
          4.1.vi Checking the Input Domain Model
       4.2 Combination Strategies Criteria
       4.3 Constraints Among Partitions
       4.4 Bibliographic Notes
    5. Syntax Testing
       5.1 Syntax-Based Coverage Criteria
          5.1.i BNF Coverage Criteria
          5.1.ii Mutation Testing
       5.2 Program-Based Grammars
          5.2.i BNF Grammars for Languages
          5.2.ii Program-Based Mutation
       5.3 Integration and Object-Oriented Testing
          5.3.i BNF Integration Testing
          5.3.ii Integration Mutation
       5.4 Specification-Based Grammars
          5.4.i BNF Grammars
          5.4.ii Specification-Based Mutation
       5.5 Input Space Grammars
          5.5.i BNF Grammars
          5.5.ii Mutation for Input Grammars
       5.6 Bibliographic Notes

Part III: Applying Criteria in Practice
    6. Practical Considerations
       6.1 Regression Testing
       6.2 Integration and Testing
          6.2.i Stubs and Drivers
          6.2.ii Class Integration Test Order
       6.3 Test Process
          6.3.i Requirements Analysis and Specification
          6.3.ii System and Software Design
          6.3.iii Intermediate Design
          6.3.iv Detailed Design
          6.3.v Implementation
          6.3.vi Integration
          6.3.vii System Deployment
          6.3.viii Operation and Maintenance
          6.3.ix Summary
       6.4 Test Plans
       6.5 Identifying Correct Outputs
          6.5.i Direct Verification of Outputs
          6.5.ii Redundant Computations
          6.5.iii Consistency Checks
          6.5.iv Data Redundancy
       6.6 Bibliographic Notes
    7. Engineering Criteria for Technologies
       7.1 Testing Object-Oriented Software
          7.1.i Unique Issues with Testing OO Software
          7.1.ii Types of Object-Oriented Faults
       7.2 Testing Web Applications and Web Services
          7.2.i Testing Static Hyper Text Web Sites
          7.2.ii Testing Dynamic Web Applications
          7.2.iii Testing Web Services
       7.3 Testing Graphical User Interfaces
          7.3.i Testing GUIs
       7.4 Real-Time Software and Embedded Software
       7.5 Bibliographic Notes
    8. Building Testing Tools
       8.1 Instrumentation for Graph and Logical Expression Criteria
          8.1.i Node and Edge Coverage
          8.1.ii Data Flow Coverage
          8.1.iii Logic Coverage
       8.2 Building Mutation Testing Tools
          8.2.i The Interpretation Approach
          8.2.ii The Separate Compilation Approach
          8.2.iii The Schema-Based Approach
          8.2.iv Using Java Reflection
          8.2.v Implementing a Modern Mutation System
       8.3 Bibliographic Notes
    9. Challenges in Testing Software
       9.1 Testing for Emergent Properties: Safety and Security
          9.1.i Classes of Test Cases for Emergent Properties
       9.2 Software Testability
          9.2.i Testability for Common Technologies
       9.3 Test Criteria and the Future of Software Testing
          9.3.i Going Forward with Testing Research
       9.4 Bibliographic Notes