The Science of Evaluation: A Realist Manifesto

Evaluation researchers are tasked with providing the evidence to guide programme building and to assess its outcomes. As such, they labour under the highest expectations - bringing independence and objectivity to policy making. They face huge challenges, given the complexity of modern interventions and the politicised backdrop to all of their investigations. They have responded with a huge portfolio of research techniques and, through their professional associations, have set up schemes to establish standards for evaluative inquiry and to accredit evaluation practitioners. A big question remains. Has this monumental effort produced a progressive, cumulative and authoritative body of knowledge that we might think of as evaluation science? This is the question addressed by Ray Pawson in this sequel to Realistic Evaluation and Evidence-based Policy. In answer, he provides a detailed blueprint for an evaluation science based on realist principles.

Table of Contents:
Preface: The Armchair Methodologist and the Jobbing Researcher PART ONE: PRECURSORS AND PRINCIPLES Precursors: From the Library of Ray Pawson First Principles: A Realist Diagnostic Workshop PART TWO: THE CHALLENGE OF COMPLEXITY - DROWNING OR WAVING? A Complexity Checklist Contested Complexity Informed Guesswork: The Realist Response to Complexity PART THREE: TOWARDS EVALUATION SCIENCE Invisible Mechanisms I: The Long Road to Behavioural Change Invisible Mechanisms II: Clinical Interventions as Social Interventions Synthesis as Science: The Bumpy Road to Legislative Change Conclusion: A Mutually Monitoring, Disputatious Community of Truth Seekers

About the Author :
Given my job title, it will come as no surprise that my main interest lies in research methodology. This does not quite bracket me with the technical nerds, however, for I have written widely on the philosophy and practice of research, covering methods qualitative and quantitative, pure and applied, contemporaneous and historical. There is a common ′realist′ thread underlying every word, albeit a modest, middle-range, empirically-rich kind of realism.

Review :
This work provides a real breakthrough. It teaches us to appreciate the hugely varied interactions that can occur when our policy interventions are inserted into the complex environments in which we live. More importantly, it teaches us how to extract general lessons from the diverse occurrences we observe in various policy settings, so that we can have a better understanding and better results in the future. Nancy Cartwright Professor of Philosophy, University of California San Diego Pawson has lost none of his touch for the pithy phrase and no-nonsense argument, but this time the style is put to service in a work of architectural proportions. I hope and expect this book to become required reading within and beyond the evaluation community. Christopher Pollitt Emeritus Professor, Katholieke Universiteit Leuven, Belgium Ray Pawson′s new book is a treat for evaluators, particularly in a world full of checklists, protocols, guidelines, and best practices. Evaluation is back to where it belongs: a full consideration of the theory and method of applied research with its scientific roots taken seriously. Frans Leeuw Professor of Law, Public Administration and Social Research, Maastricht University, The Netherlands Like his earlier books, Realistic Evaluation, with Nick Tilley, and Evidence-Based Policy: A Realist Approach, this book has the potential to transform our understanding of what evaluation can be and how it can be used. Pawson addresses common questions about realist evaluation and shows clearly that it is more than simply identifying patterns in results but essentially involves iteratively building and testing theories to explain these patterns. His revisiting of classic evaluation texts demonstrates their realist underpinnings and he convincingly reclaims the Campbell mantle from those who would invoke his name only for a narrow range of experimental research designs. The book deserves an audience among evaluators and users of evaluation, especially those who are sceptical of realist evaluation who can create the "mutually monitoring, disputatious community of scholars" to properly review and revise these ideas. Patricia Rogers Professor of Public Sector Evaluation, Royal Melbourne Institute of Technology, Australia I really enjoyed the evident relish which Ray Pawson takes into methodological debates, and the disputatious but constructive style of the whole book - a good example being the section on the MRC complex interventions evaluation framework. One particular highlight in the book for me was the chapter on ′clinical interventions as social interventions′ which should be required reading for all biomedicalinical researchers, if only because they′ll disagree with it. Kieran Walshe Professor of Health Policy and Management, Manchester Business School Ray Pawson′s The Science of Evaluation completes the realist evaluation trilogy composed of Realist Evaluation (co-author NickTilley) and Evidence-based Policy. His approach has beguiled many in the field but has left us with many puzzles: How to deal with complex programs? How to choose and develop program theories? How to deal with uncertainty? This new work provides answers, with characteristic brio, by moving backwards into the theoretical roots of realism, with its generative view of causation, and the legacy of the evolutionary epistemology - and also by moving forward presenting illustrations of the logic of inquiry of evaluations that provide a "realist response to complexity". Pawson′s argument for "middle range theories" and "partial knowledge" is an invitation to engage in a "never-ending journey" that will take evaluation to the core of policy-making and social change. Prof Nicoletta Stame Sapienza: Università di Roma Ray Pawson is quite a rare creature - a professor of social research methodology, and this is a book about methodology, the ′how to′ of the evaluation of public policies and programmes. It both argues for a ′realist′ approach to these activities and explores in some detail what such an approach entails. It is written in a lively, vivid, personal and often humorous way that will be familiar to those who read the author′s previous works...I would say that, whether or not you intend to embark on this particular voyage, whether or not you like academic texts which are sprinkled with humour, you should consider reading this book. It strikes me as genuinely important, and very challenging. And certainly all those who think they believe in (or at least believe it is judicious to pretend to believe in) ′evidence-based policymaking′ should explore Pawson′s book - in order to get a taste for just what that would actually entail. Christopher Pollitt The International Review of Administrative Sciences   ′The Science of Evaluation deserves to be widely read and discussed. In a literature that often struggles to be lucid, accessible and engaging, writing with clarity, honesty and passion, Pawson thoughtfully outlines how evaluation research can embrace a realist perspective without fearing it will lose its methodological ardour′ Pawson calls for an endless journey into evaluation, organised with interventiontheories as the unit of analysis, so as to generate a phalanx of middle-range researchprogrammes. His second suggestion is for maintaining an ever-shifting and improvingmiddle ground that can only be achieved collectively and through constant criticalscrutiny of each other’s work. Thus his hope is that ideas and problems raised inthis book will stimulate and encourage more research about the scientific status ofevaluation. The Science of Evaluation is a key text for anyone interested in research in the social sciences. It is a brilliant book which will justifiably become a fixture on students’ reading lists, as it brings together in one place Pawson’s thinking and ideas in relation to Realist Evaluation.