Evaluation in the Face of Uncertainty: Anticipating Surprise and Responding to the Inevitable

About the Book

Unexpected events during an evaluation all too often send evaluators into crisis mode. This insightful book provides a systematic framework for diagnosing, anticipating, accommodating, and reining in costs of evaluation surprises.

Table of Contents:

1. From Firefighting to Systematic Action

Adding “Surprise” to the Mix

Historical Roots: Evaluation, Planning, and System Behavior

From Explaining Surprise to Dealing with It

Development Path of This Book

Guiding Principles

How to Read This Book

In Sum

2. Structure of the Unexpected

Where Does Surprise Come From?

Beyond Simple Distinctions

In Sum

3. Placing Surprise in the Evaluation Landscape

When Is the Probability of Surprise High?

When Is Surprise Disruptive to Evaluation?

In Sum

4. Minimizing Foreseeable Surprise

Theory: Using Explanatory Power and Simplified Relationships

Exploiting Past Experience: Capitalizing on What We Already Know

Limiting Time Frames to Minimize the Opportunity for Surprise

In Sum

5. Shifting from Advance Planning to Early Detection

Leading Indicators

System-Based Logic Modeling

In Sum

6. Agile Evaluation

Data

Agile Methodology

Retooling Program Theory

Agility and Stakeholder Needs

In Sum

7. How Much Is Too Much?: Appreciating Trade-Offs and Managing the Balance

A Framework for Appreciating Design Trade-Offs

Maximizing Choice, Minimizing Risk

Evaluation Design

In Sum

8. Applying the Examples to Categories of Cases: The Life Cycle View

“Unintended Consequences”: Unity across Programs and Their Evaluations

Interpreting Cases through a Life Cycle Perspective

In Sum

9. Applying the Examples to Categories of Cases: The Social/Organizational View

Navigating through the Cases

Placement of Cases on the Social/Organizational Map

Categorizations Derived from the Data

In Sum

10. Lessons fr

About the Author :
Jonathan A. Morell, Senior Policy Analyst, Vector Research Center, TechTeam Government Solutions, Ann Arbor, Michigan, USA

Review :
"If your world, including your evaluation work, is often complex, uncertain, and unpredictable, you have a fellow traveler and real-world guide in Morell. He applies more than three decades of experience to the challenges of distinguishing what can and cannot be foreseen, anticipating the unexpected, and dealing with the unforeseeable. This book draws on concrete cases, expert wisdom, practitioner experiences, scholarly knowledge, and organizational theory to explore evaluation approaches and methods that are agile, flexible, emergent, and responsive. Morell's voice is personable, his guidance realistic, and his insights important. You'll be surprised how much better you can get at anticipating and learning from surprises." - Michael Quinn Patton, Director, Utilization-Focused Evaluation, St. Paul, Minnesota, USA "Insightful and provocative. Though Morell writes from the stance of an evaluator, his descriptions of 'things that go awry' apply to a wide swath of research methodologies. The idea that all research projects encounter unanticipated or unintended outcomes is aptly illustrated through a variety of case studies; for example, No Child Left Behind evaluation studies, health impacts of central heating, and outcomes of abolishing user fees in health clinics in Niger. The cases provide ample evidence of why things went awry and how unanticipated or unintended outcomes may be predicted and controlled. This book would be ideal for graduate-level courses on research design or program evaluation, either as a textbook or a supplement." - James E. Gruber, Department of Behavioral Sciences, University of Michigan--Dearborn, USA "This clearly written, well-organized book presents a lexicon of the surprises that occur in evaluation practice, both in the program - how it unfolds between planning and completion of the evaluation - and in the process of data collection and analysis. Morell outlines a structure for understanding what these surprises are, where they occur in the programming and evaluation process, why they are inevitable, and how they can (or sometimes cannot) be foreseen. The book provides practitioners with a systematic way of diagnosing and possibly even anticipating surprises, and explains how to accommodate them." - Deborah Wasserman, Principal Consultant, PERSolutions: Program Evaluation and Research, Columbus, Ohio, USA