How to Measure Anything in Project Management

Uncover common project management myths to improve project success How to Measure Anything in Project Management explains why popular methods for measurement in project management are flawed and describes how to conduct measurements that better inform decisions, reduce project risks, and improve the chance of project success. The authors argue that anything that matters to project management at all is measurable and that these measurements address many of the problems in project management. The authors leverage an exclusive survey on the state-of-the-art of measuring projects, new case studies of things that are seemingly hard to measure and a database, collected by Oxford Global Projects, of thousands of projects in software development, construction, energy, and many other fields, including some of the biggest projects in history. The book is accompanied by a set of useful spreadsheet-based "power tools" that support the more technical aspects of quantifying project risk, forecasting outcomes, and conducting seemingly difficult measurements. In this book, readers will learn: Why many of the methods they have been taught to use are little more than a type of “analysis placebo” Why many popular methods lead to extreme overconfidence in estimates How some of the most important measurements a project could conduct are currently rarely used How to Measure Anything in Project Management earns a well-deserved spot on the bookshelves of managers, executives, auditors, controllers, and consultants seeking to improve project performance through superior measurement methodology.

Table of Contents:
Foreword xv Preface xix Acknowledgments xxi About the Authors xxiii Chapter 1 A World-scale Risk and a World-scale Opportunity 1 The Size of Projects 2 The Size of Project Problems 4 Efforts to Fix Projects: The Emergence of Project Management 5 A Path Forward: The Meta Project 8 Notes 10 Chapter 2 A Measurement Primer for Project Management 13 The Concept of Measurement 14 A Definition of Measurement 15 Measurement and Probabilities for Practical Decision-making 16 Are Scales Really Measurements? 18 The Object of Measurement 21 What Do You See When You See More of It? 21 Why Do You Care? 23 The Methods of Measurement 25 Statistical Significance: What’s the Significance? 26 Small Samples Tell You More Than You Think 28 Other Sources of Measurement Aversion 30 The Cost Objection 30 Measurements Change What Is Being Measured 31 Statistics Can Prove Anything 32 Ethical Objections to Measurement 33 Notes 34 Chapter 3 How We Know What Works 35 Skepticism for Project Managers 36 The Analysis Placebo 36 The Problem of Feedback and Learning 38 How to Test Methods 40 Controlled Experiments and Component Testing 40 Evaluating Sources 41 The Performance of Quantitative Methods 43 Experts Versus Algorithms 43 The Exsupero Ursus Fallacy: Algorithm Aversion 44 Potential Reasons for Exsupero Ursus 45 Improving the Human Expert 47 Calibrating the Expert 48 The Expert Consistency Component 49 Collaboration on Estimates 50 The Decomposition Component 52 A Summary of Research on Other Project Planning and Management Methods 54 Reference Class Forecasting 54 Various Project Management Methods 55 The Performance of Monte Carlo Simulations 58 Notes 60 Chapter 4 The Project Decision Model: The Reason for Measurements 63 Two Types of Project Measurements 64 Proto-purpose Discovery Measurements 64 Decision-driven Measurements 66 Unproductive Incentives vs. Measurements 69 Decisions Before: Thinking Slow 70 Exploration vs. Exploitation 71 Tracking the Outside World 73 Choosing How to Run the Project 74 How Models Indicate What to Measure 77 The Expected Value of Information: A Simple Introduction 77 The Measurement Inversion: Measuring the Wrong Things 79 The Value of Imperfect Measurements 80 An Aspirational Model 82 The Rise of Digital Twins 83 Digital Twins in Project Management 84 A Practical Path Forward 87 Notes 88 Chapter 5 Project Uncertainty and Risk: A Primer 91 Basic Concepts and Definitions 92 Uncertainty as a Probability Distribution 93 Risk: A Special Case of Uncertainty 96 The Problem with Current Methods 98 Why Risk “Scores” Don’t Work 99 How the Risk Matrix Makes Scores Worse 101 A Quantitative Risk Model: Starting Very Simple 105 The One-for-One Substitution 106 Monte Carlo Mechanics: A Brief Introduction 108 Supporting Decisions 111 A Return on Mitigation 112 How Much Risk Do You Tolerate? 113 Risk Versus Return: The Powerful Theory of Utility 115 Simple Tools for Measuring Uncertainty and Risk 117 A First Estimate of a Discrete Probability 118 A First Estimate of a Continuous Probability 119 Final Clarifications 120 Case Examples for What Probability Means 121 Uncertainty Versus Risk Versus Opportunity 123 Epistemic Versus Aleatory Uncertainty 124 Even More Ordinal Scales 125 Risk as Governance or Compliance 125 The Problem of “Black Swans” 126 Some Items That Aren’t Really Risks 127 More Improvements to Come 128 Notes 129 Chapter 6 Calibrated Subjective Probability Estimates 131 Introduction to Subjective Probability 132 Calibration Exercise 135 The Calibration Exercises 136 Evaluating Performance and Typical Results 137 Improving Calibration 140 The Equivalent Bet 141 More Techniques 142 More Advanced Calibration Topics to Come 144 The Effects of Calibration 146 Conceptual Obstacles to Calibration 149 Conflating Uncertainty with Knowing Nothing 149 Hypotheses That Contradict the Data 152 Objections Based on the Philosophical Debate in Statistics 153 Notes 155 Chapter 7 Cost and Schedule Measurements 157 The Big Plan Versus Iteration: Meta-measurements of Common Estimation Methods 158 Top-down Estimations: Reference Class Forecasting 162 Bottom-up Forecasting with Monte Carlo 165 A Deterministic View of Tasks 165 Probability Distributions for Project Tasks 167 Correlations 168 Multiple Prerequisites and Stochastic Critical Paths 170 Parade of Trades 171 Comparing Top Down and Bottom Up: Case Examples 174 The Swedish Nuclear Waste Program 175 High-speed Rail 176 How to Improve the Models 181 The Granularity of the Monte Carlo Model 182 Distributions and Biases 182 Correlations 183 Improving the RCF with Monte Carlo 184 Notes 185 Chapter 8 Betting on Benefits 187 Meta-measurements of Benefits 189 How Much Should Benefits Be to Justify a Project? 190 Why This May Be Optimistic 192 Why Measuring Benefits Is Rare 195 Fermi Decompositions for Benefits 196 Introduction to Fermi 197 Some Example Decompositions 199 Monetizing Benefits 201 Forecasts of Monetary Impacts 201 Preferences 202 Quantifying Preferences 203 The Use of Scores and Multiple Objectives 205 An Example of Challenging Benefit Measurement: Biodiversity 206 Measuring What Matters in Projects 206 A (Slightly) More Realistic Information Value Calculation 207 The High Information Values for Projects 209 Getting Started on Measuring What Matters 211 Considering Risk and Return 213 A Risk Neutral Decision-maker for Projects 214 Adding Utility Theory to Projects 215 Some Alternatives within Utility Math 217 Are Executives Too Risk Averse for Projects? 219 A Framework and Its Consequences 221 Findings from Quantitative Analysis of Past Projects 223 How and When, Not Just Whether 223 Benefits Are Not Just for Project Approval Decisions 224 Notes 225 Chapter 9 Measuring Progress 227 The Progress Problem 227 Simple Progress, Simple Interventions 228 Earned Value Management 229 EVM Basics 230 The XRL Example 231 Recovery vs. Performance 233 Forecasting with EVM 235 Progress in Information Projects 237 Waterfall 237 Agile and Measurement in Other Software Development Methods 237 Summarizing Software Metric Difficulties 239 Four Stories and Lessons 240 Interfaces in a Global Bank Transformation 240 An Energy Project Front End 241 Construction Constraints 243 Testing as Software Checkpoints 245 Lessons 246 The Remaining Project Simulation 247 Conditional Reference Class Forecasting (CRCF) 247 The Bottom-up Simulation for the Remaining Project 251 Further Considerations for the RPA 252 Notes 254 Chapter 10 More Measurement Methods Made Easy 257 Intuition for the Habitually Scientific 258 A Jelly Bean Example 258 A Little Probability Theory 260 Consequences of Probability Theory 262 Myths Exposed by Probability Theory 262 Significant Points About Statistical Significance 265 Basic Sampling Methods 266 The “Mathless” Table for Medians 269 Estimating a Population Proportion 270 Project Cancellation Rates as a Function of Duration 274 Measuring Population Size 274 Measuring Some Things by Knowing Other Things 276 Controlled Experiments 277 Regression 277 More Advanced Methods of Regression and Classification 283 Estimating the Whole Distribution 285 Summarizing Methods 289 Brainstorming a Measurement Approach 289 Data Gathering Methods 291 A Review of Methods in This Chapter 292 Notes on Surveys 293 Notes 296 Chapter 11 The Meta-project: Implementing Better Project Measurements 297 Start with the End in Mind: The Continuous Improvement Process 299 Measure What Matters 299 (Real) Skepticism and Meta-measurements 301 Measuring and Forecasting the Outside World 302 AI: The Most Important Project Ecosystem Measurement? 304 More Thinking, Fewer Projects, Bigger Wins 307 Start Your Meta-project 307 Examples of Meta-projects Deliverables: Continuous Improvement 308 Develop an Initial Team 309 Assess the Current State of the Project Portfolio 310 Considerations for the Meta-project Plan 312 The Pilot Project 312 Scaling to the Final Deliverable 314 Organizational Challenges 315 Resistance to Change 315 Addressing Organizational Objections to Measurement 316 The Politics of Measurement 318 Notes 319 Chapter 12 A Call to Action for the Industry 321 Call for Action for Project Software Vendors 321 Put Decisions at the Center 322 Deal in Uncertainties 324 Build the User-buyer-builder Federation 325 Be the Vendor That Measures Its Performance 325 Be Forward-looking 326 Call for Action for the Standard-setting Bodies 327 Call to Action for Consultants, Researchers, and Advisory Firms 329 Big Future Projects 331 A Mars Mission 331 Stopping Hurricanes 332 The Meta-Project 333 Notes 333 Appendix 1 Analysis of Survey Responses on Project Management Practices 335 Introduction and data overview 335 Success Metrics: Cost and Schedule Overrun Ratios 337 Overview of Project Management Practices Reported in the Survey 339 Project Management Methodologies 339 Cost and Schedule Estimation Methods 339 Uncertainty and Risk Assessment Tools 340 Certifications 341 Results 341 Project Management Methodologies 341 Cost and Schedule Estimation Methods 343 Uncertainty and Risk Assessment Tools 343 Certifications 343 Interpreting the (Mostly) Statistically Insignificant Results 344 Appendix 2 Reference Class Data on Project Cost, Schedule, and Benefit Overruns 345 Relevance of the Data and Reference Class Forecasting 346 Using Historical Data to Improve Estimates – An Example 347 Notes 351 Appendix 3 Selected Distributions 353 Uniform 354 Beta 355 Beta PERT 356 Triangular 357 Binary 358 Normal 359 Lognormal 360 Power Law 361 Truncated Power Law 362 Quantile-parameterized 363 Gamma Poisson 365 Stochastic Information Packet 366 Appendix 4 Chapter 6 Calibration Question Answers 369 Answers to Confidence Interval Questions 369 Answers to True/False Questions 371 Appendix 5 Measuring Biodiversity 373 The Benefits of Biodiversity 373 Measuring Biodiversity 375 Notes 376 Index 377

About the Author :
DOUGLAS W. HUBBARD has 35 years’ experience as a management consultant with a focus on the application of quantitative methods in decision making. He is the founder and president of Hubbard Decision Research and the creator of the “Applied Information Economics” method. He is also the author of the original How to Measure Anything: Finding the Value of Intangibles in Business as well as other books in measurement, risk analysis and decision making. ALEXANDER BUDZIER, PHD, is a Fellow at the University of Oxford’s Saïd Business School. He specializes in IT, infrastructure, energy, mega-events and change. ANDREAS BANG LEED is the Head of Data Science at Oxford Global Projects. He specializes in data-driven project planning and risk analysis for some of the world’s most ambitious mega-projects.