A hands-on guide to testing techniques that deliver reliable software and systems
Testing even a simple system can quickly turn into a potentially infinite task. Faced with tight costs and schedules, testers need to have a toolkit of practical techniques combined with hands-on experience and the right strategies in order to complete a successful project. World-renowned testing expert Rex Black provides you with the proven methods and concepts that test professionals must know. He presents you with the fundamental techniques for testing and clearly shows you how to select and apply successful strategies to test a system with budget and time constraints.
Black begins by discussing the goals and tactics of effective and efficient testing. Next, he lays the foundation of his technique for risk-based testing, explaining how to analyze, prioritize, and document risks to the quality of the system using both informal and formal techniques. He then clearly describes how to design, develop, and, ultimately, document various kinds of tests. Because this is a hands-on activity, Black includes realistic, life-sized exercises that illustrate all of the major test techniques with detailed solutions.
Table of Contents:
Acknowledgments xiii
Introduction xv
Part I Goals, Strategies, and Tactics 1
Chapter 1 What Does It Mean to Be Pragmatic? 3
What Do Effective and Efficient Mean? 3
What Effects Do You Want? 4
What Is the Right Level of Efficiency? 4
Avoiding Redundancy 4
Reducing Cost 5
What Software Testing Isn’t…But Is Often Thought to Be 6
Five Phases of a Tester’s Mental Life 7
Other Founding Views on Testing 8
Testing as a Form of Quality Risk Management 9
So What Is the Test Team All About? 9
What Does “Quality” Mean to You? 10
Chapter 2 Triangle Test Exercise 13
Exercise: The Triangle Test 13
Your Solution for the Triangle Test 14
Your Solution for the Triangle Test (Continued) 15
Author’s Triangle Test Solution 15
Chapter 3 Aligning Testing with the Project 19
Why Do Organizations Test? 19
Perspectives on Testing 20
Testing in Context 20
Common Test Phases and Objectives 21
Testing Throughout the Organization 21
The V Model 23
Evolutionary and Incremental Models 25
The Spiral Model 28
Regrettably Common Model: Code and Fix 28
Testing Maintenance Releases 29
System Integration 30
Hardware/Software Development 32
The Test Process 32
Chapter 4 Understanding Test Strategies, Tactics, and Design 35
Aligning Mission, Strategies, and Tactics 35
Analytical Test Strategies 36
Model-Based Test Strategies 37
Methodical Test Strategies 38
Process-Oriented Test Strategies 38
Dynamic Test Strategies 39
Philosophical Test Strategies 40
Regression 41
Regression Strategy 1: Repeat All Tests 42
Regression Strategy 2: Repeat Some Tests 42
Three Other Regression Strategies 43
Tactics: Categories of Testing Techniques 44
Strategic and Tactical Considerations 46
The Test System 47
Classic Principles for Test Design 50
Phases of Test Development 51
Synopsis of Test Strategies, Tactics, and Design 52
Part II Risk-Based Testing 53
Chapter 5 Understanding Risks to System Quality 55
Categories of Quality Risks 56
Functionality 56
Performance and Reliability 57
Stress, Capacity, and Volume 59
States 59
Transactions 60
Installation and Deinstallation 61
Operations 62
Maintenance and Maintainability 63
Regression 63
Usability and User Interface 64
Data Quality 65
Error and Disaster Handling and Recovery 66
Date and Time Handling 67
Localization 68
Configuration and Compatibility 69
Networked, Internetworked, and Distributed 69
Standards and Regulatory Compliance 70
Security 70
Timing and Coordination 71
Documentation 71
Can You Think of Other Quality Risks? 72
Chapter 6 Aligning Testing with Quality Risks 73
Prioritize Risks to System Quality 73
Testing, Customer Usage, and System Configurations 74
Approaches for Quality Risks Analysis 76
Informal Quality Risk Analysis 78
Tips for Risk Analysis 81
Challenges of Risk Analysis 82
Chapter 7 Quality Risk Analysis Exercise 85
My Solution 86
Comments on My Solution 91
A Funny Thing Happened to Me on the Way to the Quality Risks… 92
Bonus Exercise 95
Template 96
My Solution 100
Comments on My Solution 101
Bonus Example: Grays and Blues Quality Risk Analysis 102
Part III Static Testing 105
Chapter 8 Reviews and Other Static Tests 107
Testing Requirements and Designs 107
Reviews Costs and Benefits 108
Types of Reviews 109
Reviews for Verification and Validation 109
Reviews as a Way to Improve the Process, Improve the System, and Reduce Costs 110
Answers to the Three Questions about Static Testing 114
Reviews as a Way to Achieve Consensus and Understanding 115
The Review Process, Roles, and Responsibilities 116
Deliverables and Ground Rules from Reviews 117
Common Requirements and Design Bugs 118
Reviewing (and Testing) Documentation 119
Other Static Tests 120
Chapter 9 Review Exercise 123
Reviews 123
Your Solution 124
My Solution 128
Comments on My Solution 132
Bonus Exercise: Reviewing Triangle Requirements Title Page 132
Triangle Requirements Review Using Wiegers’s List 132
The Requirements Bugs You Found 134
My Solution: Requirements Bugs 137
Comments on My Solution 139
Part IV Behavioral Testing 141
Chapter 10 Equivalence Classes and Boundary Values 143
Equivalence Partitioning 144
Boundary Value Analysis 144
Integer 145
Real Numbers 147
Character and String 149
Date 152
Time 154
Currency 155
Beyond Functional Classes and Boundaries 157
Chapter 11 Equivalence Classes and Boundary Values Exercise 159
Functional Boundaries and Classes 159
My Solution and Comments 161
Nonfunctional Boundaries and Classes 166
Omninet Marketing Requirements Document (v0.3) 166
Omninet System Requirements Document (v0.2) 168
Chapter 12 Use Cases, Live Data, and Decision Tables 169
Use Cases and Scenario Tests 169
Nouns and Verbs, Adjectives and Adverbs 172
Live Data and Customer Tests 173
Decision Tables 174
Chapter 13 Decision Table Exercise 177
Decision Table Tests 177
My Solution and Comments 178
Decision Tables and Boundary Values 181
My Solution and Comments 183
Building a Decision Table for Testing 188
My Solution and Comments 190
Chapter 14 State Transition Diagrams 197
Describing Systems with States Using State Transition Diagrams 197
State Tables 199
Printer Server State Transition Diagram 200
Chapter 15 State Transition Diagram Exercise 203
Kiosk States 203
My Solution and Comments 204
ATM State Models 211
My Solution and Comments 212
Grays and Blues and State Transition Diagrams 213
My Solution and Comments 214
Chapter 16 Domain Testing 217
Combinatorial Explosions 217
A Domain Example Using Frequent-Flyer Programs 218
Possible Domain Test Values 220
An Aerospace Example 221
When Domain Rules Change 223
Domain Analysis Summary 224
Complex Domain Testing Example 225
A General Rule for Complex Domains 228
Chapter 17 Domain Testing Exercise 233
My Solution and Comments 234
Chapter 18 Orthogonal Arrays and All Pairs 239
Combinatorial Explosions 239
Orthogonal Arrays and All-Pairs Tables 241
Two Orthogonal Arrays 241
Selecting an Orthogonal Array 243
Applying Orthogonal Arrays to a Complicated Real-World Example 244
All-Pairs Tables 258
Other Thoughts on Configuration Testing 259
Chapter 19 Orthogonal Arrays Exercise 261
My Solution and Comments 263
Chapter 20 Reactive Testing 267
General Facts about Reactive Tests 267
Error Guessing, Attacks, and Bug Taxonomies 268
Bug Hunting 269
Exploratory Tests 270
Checklists 271
Other Sources of Inspiration for Reactive Testing 273
Advantages and Disadvantages 274
A Case Study of Exploratory Testing 276
Part V Structural Testing 279
Chapter 21 Control-Flow Testing 281
Code Coverage 282
McCabe Cyclomatic Complexity 284
Chapter 22 Control-Flow Testing Exercise 287
My Solution and Comments 288
Chapter 23 Data-Flow Testing 293
Chapter 24 Data-Flow Testing Exercise 297
My Solutions and Comments 298
Chapter 25 Integration Testing 307
Drivers and Stubs 308
Integration Techniques 309
Backbone Integration 311
McCabe Basis Paths for Integration 313
Enhanced Hex Converter Program 316
Call Flow 318
Chapter 26 Integration Basis Test Exercise 321
My Solution and Comments 323
Part VI Appendices 325
Appendix A Omninet: The Internet Everywhere Marketing Requirements Document 327
1 Scope 329
1.1 Terms, Acronyms, and Abbreviations 329
1.2 Applicable Documents 330
2 Required release date 330
3 Description of requirements 330
3.1 General technical requirements 330
3.1.1 Welcome 330
3.1.2 Payment 330
3.1.3 Internet Browser 331
3.1.4 Performance 331
3.1.5 Localization 331
3.1.6 Content Control 331
3.1.7 Session Termination 331
3.1.8 Confidentiality 332
3.2 Administration 332
3.2.1 Software Updates 332
3.2.2 View Kiosks 332
3.2.3 View Users 333
3.2.4 Modify User 333
3.2.5 Terminate User 333
Appendix B Omninet: The Internet Everywhere System Requirements Document 335
Functionality System Requirements 336
Reliability System Requirements 340
Usability System Requirements 341
Efficiency System Requirements 342
Maintainability System Requirements 343
Portability System Requirements 344
Design Models 345
Omninet System Architecture 345
Payment Processing Decision Table 346
Kiosk Module Flow 347
Kiosk State-Transition Diagram 348
Kiosk State-Transition Table 348
Kiosk OS/Browser/Connection Speed Configuration Orthogonal Array 348
Appendix C Bibliography and Other Recommended Readings 351
RBCS Company Profile 353
Index 355
About the Author :
Rex Black is a 20-plus-year software and systems engineering veteran, and President and Principal Consultant of RBCS, Inc., a leader in software, hardware, and systems testing. He is the author of Managing the Testing Process as well as numerous articles. Black is also President of both the International Software Testing Qualifications Board and the American Software Testing Qualifications Board.
