Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality

“This book fills a huge gap in our knowledge of software testing. It does an excellent job describing how test automation differs from other test activities, and clearly lays out what kind of skills and knowledge are needed to automate tests. The book is essential reading for students of testing and a bible for practitioners.” –Jeff Offutt, Professor of Software Engineering, George Mason University “This new book naturally expands upon its predecessor, Automated Software Testing, and is the perfect reference for software practitioners applying automated software testing to their development efforts. Mandatory reading for software testing professionals!” –Jeff Rashka, PMP, Coauthor of Automated Software Testing and Quality Web Systems Testing accounts for an increasingly large percentage of the time and cost of new software development. Using automated software testing (AST), developers and software testers can optimize the software testing lifecycle and thus reduce cost. As technologies and development grow increasingly complex, AST becomes even more indispensable.  This book builds on some of the proven practices and the automated testing lifecycle methodology (ATLM) described in Automated Software Testing and provides a renewed practical, start-to-finish guide to implementing AST successfully. In Implementing Automated Software Testing, three leading experts explain AST in detail, systematically reviewing its components, capabilities, and limitations. Drawing on their experience deploying AST in both defense and commercial industry, they walk you through the entire implementation process–identifying best practices, crucial success factors, and key pitfalls along with solutions for avoiding them. You will learn how to:     Make a realistic business case for AST, and use it to drive your initiative     Clarify your testing requirements and develop an automation strategy that reflects them     Build efficient test environments and choose the right automation tools and techniques for your environment     Use proven metrics to continuously track your progress and adjust accordingly Whether you’re a test professional, QA specialist, project manager, or developer, this book can help you bring unprecedented efficiency to testing–and then use AST to improve your entire development lifecycle.

Table of Contents:
Foreword by Admiral Edmund P. Giambastiani, Jr.     xv Foreword by Dr. William Nylin, Jr.       xvii Preface      xix Acknowledgments      xxiii About the Authors      xxv Part I: What Is Automated Software Testing and Why Should We Automate?      1 Chapter 1: What Is Effective Automated Software Testing (AST)?      3 1.1 Automated Software Testing Definition 4 1.2 Automated Software Testing Recipes 5 1.3 Advances in AST Technologies 8 1.4 Automating Various Software Testing Types 11 1.5 Providing AST-Based Production Support 16 1.6 Automating Standards Assessments 18 Summary 20 Notes 21 Chapter 2: Why Automate?      23 2.1 The Challenges of Testing Software Today 24 2.2 Reducing the Time and Cost of Software Testing 26 2.3 Impacting Software Quality 38 2.4 Improvements to Your Software Test Program 42 Summary 49 Notes 50 Chapter 3: The Business Case      51 3.1 Definition of the Business Case 51 3.2 Identifying the Business Needs 53 3.3 Justifying Automation in Terms of Cost and Benefits 55 3.4 Risks 65 3.5 Other Considerations 67 Summary 68 Notes 68 Chapter 4: Why Automated Software Testing Fails and Pitfalls to Avoid      69 4.1 R&D Does Not Generally Focus on Automated or Manual Testing Efforts 71 4.2 AST Myths and Realities 74 4.3 Lack of Software Development Considerations for AST 83 4.4 The Forest for the Trees–Not Knowing Which Tool to Pick 91 4.5 Lack of Automation Standards across Tool Vendors 94 4.6 Lack of Business Case 97 Summary 97 Notes 97 Part II: How to Automate: Top Six Keys for Automation Payoff      99 Chapter 5: Key 1: Know Your Requirements      101 5.1 Understand the AST Support Requirements 102 5.2 Additional Information in Support of AST Requirements 114 5.3 When Information Is Not Available 116 5.4 Start Implementing Your Requirements Traceability Matrix (RTM) 124 Summary 128 Notes 128 Chapter 6: Key 2: Develop the Automated Test Strategy      129 6.1 The AST Strategy Document 131 6.2 Scope and Automated Test Objectives 132 6.3 Identify the Approach 139 6.4 Automated Software Test Framework (ASTF) 146 6.5 AST Environment/Configuration 150 6.6 Automating the RTM 159 6.7 Automated Defect Tracking 164 Summary 164 Notes 165 Chapter 7: Key 3: Test the Automated Software Test Framework (ASTF)      167 7.1 Verify That the ASTF Meets Specified Requirements and That Features Behave As Expected 169 7.2 Peer-Review All ASTF-Related Artifacts, Including Design, Development, and Test Cases 170 7.3 Verify Requirements and Coverage 182 7.4 Hold a Customer Review 183 Summary 184 Notes 184 Chapter 8: Key 4: Continuously Track Progress–and Adjust Accordingly      187 8.1 AST Program Tracking and Defect Prevention 188 8.2 AST Metrics 192 8.3 Root Cause Analysis 205 Summary 206 Notes 207 Chapter 9: Key 5: Implement AST Processes      209 9.1 AST Phases and Milestones 211 9.2 AST Phase 1: Requirements Gathering–Analyze Automated Testing Needs 212 9.3 AST Phase 2: Test Case Design and Development 215 9.4 AST Phase 3: Automated Software Testing Framework (ASTF) and Test Script Development 216 9.5 AST Phase 4: Automated Test Execution and Results Reporting 217 9.6 AST Phase 5: Program Review and Assessment 218 9.7 Virtual Quality Gates 219 9.8 Process Measurement 220 Summary 221 Notes 222 Chapter 10: Key 6: Put the Right People on the Project–Know the Skill Sets Required      223 10.1 Program Management 232 10.2 Systems Engineering 236 10.3 Software Development 239 10.4 Configuration Management 242 10.5 Quality Assurance 244 10.6 Subject Matter Experts (SMEs) 246 Summary 247 Notes 248 Appendices      249 Appendix A: Process Checklist      251 A.1 AST Phase 1: Requirements Gathering–Analyze Automated Testing Needs 252 A.2 AST Phase 2: Test Case Design and Development 253 A.3 AST Phase 3: Automated Software Testing Framework (ASTF) and Test Script Development 254 A.4 AST Phase 4: Automated Test Execution and Results Reporting 255 A.5 AST Phase 5: Program Review and Assessment 256 Appendix B: AST Applied to Various Testing Types      257 B.1 Security Testing 257 B.2 Soak Testing 261 B.3 Concurrency Testing 263 B.4 Performance Testing 265 B.5 Code Coverage Testing 269 B.6 Unit Testing 271 Notes 274 Appendix C: The Right Tool for the Job      275 C.1 Requirements Management (RM) 276 C.2 Unit Testing Frameworks–Example Evaluation 281 C.3 Configuration Management–Example Evaluation 284 C.4 Defect Tracking–Example Evaluation 292 C.5 Security Testing 299 C.6 Automated Software Testing Framework (ASTF)–Example Evaluation 306 C.7 Other STL Support Tools 316 Notes 319 Appendix D: Case Study: An Automated Software Testing Framework (ASTF) Example      321 D.1 Key Design Features 323 D.2 Test Manager 325 D.3 More on Automated Test Case and Test Code Generation 326 D.4 Results Reporting 328 D.5 Automated Defect Reporting 328 Notes 329 Contributing Authors      331 Index      333