Advanced Testing of Systems-of-Systems, Volume 2: Practical Aspects

As a society today, we are so dependent on systems-of-systems that any malfunction has devastating consequences, both human and financial. Their technical design, functional complexity and numerous interfaces justify a significant investment in testing in order to limit anomalies and malfunctions. Based on more than 40 years of practice, this book goes beyond the simple testing of an application – already extensively covered by other authors – to focus on methodologies, techniques, continuous improvement processes, load estimates, metrics and reporting, which are illustrated by a case study. It also discusses several challenges for the near future. Pragmatic and clear, this book displays many examples and references that will help you improve the quality of your systemsof-systems efficiently and effectively and lead you to identify the impact of upstream decisions and their consequences. Advanced Testing of Systems-of-Systems 2 deals with the practical implementation and use of the techniques and methodologies proposed in the first volume.

Table of Contents:
Dedication and Acknowledgments xiii Preface xv Chapter 1 Test Project Management 1 1.1 General principles 1 1.1.1 Quality of requirements 2 1.1.2 Completeness of deliveries 3 1.1.3 Availability of test environments 3 1.1.4 Availability of test data 4 1.1.5 Compliance of deliveries and schedules 5 1.1.6 Coordinating and setting up environments 6 1.1.7 Validation of prerequisites – Test Readiness Review (TRR) 6 1.1.8 Delivery of datasets (TDS) 7 1.1.9 Go-NoGo decision – Test Review Board (TRB) 7 1.1.10 Continuous delivery and deployment 8 1.2 Tracking test projects 9 1.3 Risks and systems-of-systems 10 1.4 Particularities related to SoS 11 1.5 Particularities related to SoS methodologies 11 1.5.1 Components definition 12 1.5.2 Testing and quality assurance activities 12 1.6 Particularities related to teams 12 Chapter 2 Testing Process 15 2.1 Organization 17 2.2 Planning 18 2.2.1 Project WBS and planning 19 2.3 Control of test activities 21 2.4 Analyze 22 2.5 Design 23 2.6 Implementation 24 2.7 Test execution 25 2.8 Evaluation 26 2.9 Reporting 28 2.10 Closure 29 2.11 Infrastructure management 29 2.12 Reviews 30 2.13 Adapting processes 31 2.14 RACI matrix 32 2.15 Automation of processes or tests 33 2.15.1 Automate or industrialize? 33 2.15.2 What to automate? 33 2.15.3 Selecting what to automate 34 Chapter 3 Continuous Process Improvement 37 3.1 Modeling improvements 37 3.1.1 PDCA and IDEAL 38 3.1.2 CTP 39 3.1.3 SMART 41 3.2 Why and how to improve? 41 3.3 Improvement methods 42 3.3.1 External/internal referential 42 3.4 Process quality 46 3.4.1 Fault seeding 46 3.4.2 Statistics 46 3.4.3 A posteriori 47 3.4.4 Avoiding introduction of defects 47 3.5 Effectiveness of improvement activities 48 3.6 Recommendations 50 Chapter 4 Test, QA or IV&V Teams 51 4.1 Need for a test team 52 4.2 Characteristics of a good test team 53 4.3 Ideal test team profile 54 4.4 Team evaluation 55 4.4.1 Skills assessment table 56 4.4.2 Composition 58 4.4.3 Select, hire and retain 59 4.5 Test manager 59 4.5.1 Lead or direct? 60 4.5.2 Evaluate and measure 61 4.5.3 Recurring questions for test managers 62 4.6 Test analyst 63 4.7 Technical test analyst 64 4.8 Test automator 65 4.9 Test technician 66 4.10 Choose our testers 66 4.11 Training, certification or experience? 67 4.12 Hire or subcontract? 67 4.12.1 Effective subcontracting 68 4.13 Organization of multi-level test teams 68 4.13.1 Compliance, strategy and organization 69 4.13.2 Unit test teams (UT/CT) 70 4.13.3 Integration testing team (IT) 70 4.13.4 System test team (SYST) 70 4.13.5 Acceptance testing team (UAT) 71 4.13.6 Technical test teams (TT) 71 4.14 Insourcing and outsourcing challenges 72 4.14.1 Internalization and collocation 72 4.14.2 Near outsourcing 73 4.14.3 Geographically distant outsourcing 74 Chapter 5 Test Workload Estimation 75 5.1 Difficulty to estimate workload 75 5.2 Evaluation techniques 76 5.2.1 Experience-based estimation 76 5.2.2 Based on function points or TPA 77 5.2.3 Requirements scope creep 79 5.2.4 Estimations based on historical data 80 5.2.5 WBS or TBS 80 5.2.6 Agility, estimation and velocity 81 5.2.7 Retroplanning 82 5.2.8 Ratio between developers – testers 82 5.2.9 Elements influencing the estimate 83 5.3 Test workload overview 85 5.3.1 Workload assessment verification and validation 86 5.3.2 Some values 86 5.4 Understanding the test workload 87 5.4.1 Component coverage 87 5.4.2 Feature coverage 88 5.4.3 Technical coverage 88 5.4.4 Test campaign preparation 89 5.4.5 Running test campaigns 89 5.4.6 Defects management 90 5.5 Defending our test workload estimate 91 5.6 Multi-tasking and crunch 92 5.7 Adapting and tracking the test workload 92 Chapter 6 Metrics, KPI and Measurements 95 6.1 Selecting metrics 96 6.2 Metrics precision 97 6.2.1 Special case of the cost of defaults 97 6.2.2 Special case of defects 98 6.2.3 Accuracy or order of magnitude? 98 6.2.4 Measurement frequency 99 6.2.5 Using metrics 99 6.2.6 Continuous improvement of metrics 100 6.3 Product metrics 101 6.3.1 FTR: first time right 101 6.3.2 Coverage rate 102 6.3.3 Code churn 103 6.4 Process metrics 104 6.4.1 Effectiveness metrics 104 6.4.2 Efficiency metrics 107 6.5 Definition of metrics 108 6.5.1 Quality model metrics 109 6.6 Validation of metrics and measures 110 6.6.1 Baseline 110 6.6.2 Historical data 111 6.6.3 Periodic improvements 112 6.7 Measurement reporting 112 6.7.1 Internal test reporting 113 6.7.2 Reporting to the development team 114 6.7.3 Reporting to the management 114 6.7.4 Reporting to the clients or product owners 115 6.7.5 Reporting to the direction and upper management 116 Chapter 7 Requirements Management 119 7.1 Requirements documents 119 7.2 Qualities of requirements 120 7.3 Good practices in requirements management 122 7.3.1 Elicitation 122 7.3.2 Analysis 123 7.3.3 Specifications 123 7.3.4 Approval and validation 124 7.3.5 Requirements management 124 7.3.6 Requirements and business knowledge management 125 7.3.7 Requirements and project management 125 7.4 Levels of requirements 126 7.5 Completeness of requirements 126 7.5.1 Management of TBDs and TBCs 126 7.5.2 Avoiding incompleteness 127 7.6 Requirements and agility 127 7.7 Requirements issues 128 Chapter 8 Defects Management 129 8.1 Defect management, MOA and MOE 129 8.1.1 What is a defect? 129 8.1.2 Defects and MOA 130 8.1.3 Defects and MOE 130 8.2 Defect management workflow 131 8.2.1 Example 131 8.2.2 Simplify 132 8.3 Triage meetings 133 8.3.1 Priority and severity of defects 133 8.3.2 Defect detection 134 8.3.3 Correction and urgency 135 8.3.4 Compliance with processes 136 8.4 Specificities of TDDs, ATDDs and BDDs 136 8.4.1 TDD: test-driven development 136 8.4.2 ATDD and BDD 137 8.5 Defects reporting 138 8.5.1 Defects backlog management 139 8.6 Other useful reporting 141 8.7 Don’t forget minor defects 141 Chapter 9 Configuration Management 143 9.1 Why manage configuration? 143 9.2 Impact of configuration management 144 9.3 Components 145 9.4 Processes 145 9.5 Organization and standards 146 9.6 Baseline or stages, branches and merges 147 9.6.1 Stages 148 9.6.2 Branches 148 9.6.3 Merge 148 9.7 Change control board (CCB) 149 9.8 Delivery frequencies 149 9.9 Modularity 150 9.10 Version management 150 9.11 Delivery management 151 9.11.1 Preparing for delivery 153 9.11.2 Delivery validation 154 9.12 Configuration management and deployments 155 Chapter 10 Test Tools and Test Automation 157 10.1 Objectives of test automation 157 10.1.1 Find more defects 158 10.1.2 Automating dynamic tests 159 10.1.3 Find all regressions 160 10.1.4 Run test campaigns faster 161 10.2 Test tool challenges 161 10.2.1 Positioning test automation 162 10.2.2 Test process analysis 162 10.2.3 Test tool integration 162 10.2.4 Qualification of tools 163 10.2.5 Synchronizing test cases 164 10.2.6 Managing test data 164 10.2.7 Managing reporting (level of trust in test tools) 165 10.3 What to automate? 165 10.4 Test tooling 166 10.4.1 Selecting tools 167 10.4.2 Computing the return on investment (ROI) 169 10.4.3 Avoiding abandonment of tools and automation 169 10.5 Automated testing strategies 170 10.6 Test automation challenge for SoS 171 10.6.1 Mastering test automation 171 10.6.2 Preparing test automation 173 10.6.3 Defect injection/fault seeding 173 10.7 Typology of test tools and their specific challenges 174 10.7.1 Static test tools versus dynamic test tools 175 10.7.2 Data-driven testing (DDT) 176 10.7.3 Keyword-driven testing (KDT) 176 10.7.4 Model-based testing (MBT) 177 10.8 Automated regression testing 178 10.8.1 Regression tests in builds 178 10.8.2 Regression tests when environments change 179 10.8.3 Prevalidation regression tests, sanity checks and smoke tests 179 10.8.4 What to automate? 180 10.8.5 Test frameworks 182 10.8.6 E2E test cases 183 10.8.7 Automated test case maintenance or not? 184 10.9 Reporting 185 10.9.1 Automated reporting for the test manager 186 Chapter 11 Standards and Regulations 187 11.1 Definition of standards 189 11.2 Usefulness and interest 189 11.3 Implementation 190 11.4 Demonstration of compliance – IADT 190 11.5 Pseudo-standards and good practices 191 11.6 Adapting standards to needs 191 11.7 Standards and procedures 192 11.8 Internal and external coherence of standards 192 Chapter 12 Case Study 195 12.1 Case study: improvement of an existing complex system 195 12.1.1 Context and organization 196 12.1.2 Risks, characteristics and business domains 198 12.1.3 Approach and environment 200 12.1.4 Resources, tools and personnel 210 12.1.5 Deliverables, reporting and documentation 212 12.1.6 Planning and progress 213 12.1.7 Logistics and campaigns 216 12.1.8 Test techniques 217 12.1.9 Conclusions and return on experience 218 Chapter 13 Future Testing Challenges 223 13.1 Technical debt 223 13.1.1 Origin of the technical debt 224 13.1.2 Technical debt elements 225 13.1.3 Measuring technical debt 226 13.1.4 Reducing technical debt 227 13.2 Systems-of-systems specific challenges 228 13.3 Correct project management 229 13.4 DevOps 230 13.4.1 DevOps ideals 231 13.4.2 DevOps-specific challenges 231 13.5 IoT (Internet of Things) 232 13.6 Big Data 233 13.7 Services and microservices 234 13.8 Containers, Docker, Kubernetes, etc 235 13.9 Artificial intelligence and machine learning (AI/ML) 235 13.10 Multi-platforms, mobility and availability 237 13.11 Complexity 238 13.12 Unknown dependencies 238 13.13 Automation of tests 239 13.13.1 Unrealistic expectations 240 13.13.2 Difficult to reach ROI 241 13.13.3 Implementation difficulties 242 13.13.4 Think about maintenance 243 13.13.5 Can you trust your tools and your results? 244 13.14 Security 245 13.15 Blindness or cognitive dissonance 245 13.16 Four truths 246 13.16.1 Importance of Individuals 247 13.16.2 Quality versus quantity 247 13.16.3 Training, experience and expertise 248 13.16.4 Usefulness of certifications 248 13.17 Need to anticipate 249 13.18 Always reinvent yourself 250 13.19 Last but not least 250 Terminology 253 References 261 Index 267 Summary of Volume 1 269

About the Author :
Bernard Homès is the founder of ISTQB, AST and the CFTL, senior member of the IEEE Standards Association and president of TESSCO sas. He has published several recognized books on software testing, standards and ISTQB Advanced v2007 and v2012 syllabi.