Enhancing Evaluation Use: Insights from Internal Evaluation Units

Enhancing Evaluation Use: Insights from Internal Evaluation Units offers invaluable insights from real evaluators who share strategies they have adopted through their own experiences in evaluation. Readers will learn about the challenges, solutions, and lessons drawn from the experience of evaluators working in a wide range of organizations. Referencing the latest literature, contributors discuss factors that help or undermine attempts to foster an evaluative thinking and learning culture within an organization. Applicable in a wide range of situations, their accounts demonstrate the initiative and innovative thinking they use to address challenges in various, sometimes complex, evaluation settings. Questions at the end of each chapter stimulate thought and discussions about the issues raised and allow readers to apply their findings to their own situations. "This book speaks to a cutting-edge topic, that is, the potential to generalize program evaluation expertise to larger organizational questions, and the cases from multiple international contexts represent a unique feature." —John Clayton Thomas, Georgia State University   "The use of actual cases to highlight major concepts in evaluation in the public sector is a great feature." —Danica G. Hays, Old Dominion University   "The text provides practical information from a variety of organizational contexts and the integration of international experiences provides for expanded discussion of evaluation theory and practice." —Kathleen Norris, Plymouth State University   "The key strengths of this book lie in its national, supra-national and international organizational contexts, its consistency in insider perspectives, and the detailed examples provided." —Donna Haig Friedman, University of Massachusetts, Boston "The book of essays reviewed here was edited by two eminent evaluators. It fills an important gap in the literature: in pursuit of improved quality of evaluation products, evaluation thinkers have lavished attention on evaluation methods, ethics and use but they have sorely neglected evaluation governance issues and have largely failed to probe the workings of evaluation within organizations. Yet, most evaluations are commissioned by (or undertaken within) organizations. The choices organizations make in structuring evaluation functions and designing evaluation processes have a major bearing on the relevance, validity and usefulness of evaluations. Refreshingly, the book offers fresh mental models and practical lessons about evaluation systems and practices adopted in diverse organizational settings. All contributors to the book are seasoned practitioners. They hail from national, supranational and international organizations and many of them have trespassed across these thematic and organizational boundaries. They all are equipped to draw on a vast reservoir of hands---on experience as evaluation commissioners, managers, internal evaluators or external practitioners. Remarkably, they also display considerable familiarity with relevant themes treated by the organization management and evaluation literature. Given its pragmatic focus the book is bound to elicit broad based interest among evaluation practitioners. While it addresses familiar dilemmas and challenges (evaluation independence, evaluation utilization, organizational learning, nurturing of an evaluation culture, etc.) it does so from the distinctive perspective of "insiders" who have had to contend with a variety of organizational constraints and management pressures. Revealingly most contributors find good reasons to be optimistic about the possibility of positive change. The stage is set by John Mayne whose introductory and authoritative chapter identifies the critical importance of organizational factors in the utilization of evaluation results. This is followed by Bastiaan de Laat’s chapter which puts forward an ingenious analytical construct: the "tricky triangle" of relationships that links commissioners, evaluators and evaluands. Specifically De Laat unmasks and assesses the different and complex configurations that result from this triangular interplay. Penny Hawkins’ elegant contribution underlines the importance of evaluation independence. It highlights the considerable impact that the electoral cycle may have on evaluation approaches. It also reveals that indigenous cultures may present formidable obstacles to the very notion of external evaluative oversight. The next chapter penned by Marlene Laubli’s is equally perceptive in tracing the creative adaptation that the evaluation function underwent in response to changes in the organizational force field of the Swiss Federal Office of Public Health’s Evaluation Unit. In a similar vein Erica Wimbush provides a lucid exposition of the strategies used to bridge evaluative knowledge and management action in Scotland’s public health agency. Still within the health field, Nancy Porteous and Steve Montague show convincingly that transformative organizational change in Canada’s Public Health Agency was facilitated through judicious evaluation programming and what has come to be called "double loop learning". Finally, Bastiaan de Laat and Kevin Williams summarize important findings from of two empirical studies of evaluation use in the European Commission that go a long way in identifying good practices for evaluation commissioning in a supra--- national organizational context. The next two chapters of the book focus on two specialized agencies of the United Nations. They tread gently in what is admittedly a highly sensitive and contested terrain. Maria J. Santamaria Hergueta, Alan Schnur and Deepak Thapa discuss how the World Health Organization’s evaluation function evolved towards greater independence over several decades while Janet Neubecker, Matthew Ripley and Craig Russon focus on the innovative techniques used within the International Labour Organization to enhance utilization of self evaluation findings. The final chapter by Marlene Laubli Loud misses the opportunity to compensate for the "good news" bias that understandably characterizes the previous contributions. Surprisingly, it displays ambivalence towards evaluation independence which it equates with externality, a concept soundly rejected by de Laat. But setting this dimension aside, the chapter is interesting and valuable since it reflects the co---editor’ vast experience as an evaluator, evaluation trainer and evaluation manager in diverse national and international contexts. Thus the chapter provides a useful compilation of "take away" messages focused on good evaluation management practices. It is on solid grounds when it describes how to make effective use of advisory committees, motivate potential evaluation users, design good terms of reference, select the right evaluators, craft judicious evaluation policies and keep an eye out for political "windows of opportunity". Each chapter of the book concludes with a crisp exposition of overarching findings, discussion topics and references. Thus the volume should be of practical value to teachers, students, professional evaluators as well as evaluation commissioners and programme managers. All in all, this is a book that belongs on your shelf if you are intent on enhancing the role that evaluation plays in your organization." —Robert Picciotto, UKES Council Member

Table of Contents:
Preface - Marlène Läubli Loud 1. Issues in Enhancing Evaluation Use - John Mayne 2. Evaluator, Evaluand, Evaluation Commissioner: A Tricky Triangle - Bastiaan de Laat 3. Evaluation Management: Lessons From New Zealand and International Development Evaluation - Penny Hawkins 4. Institutionalization and Evaluation Culture—Interplay Between the One and the Other: Lessons From the Swiss Federal Office of Public Health (FOPH) - Marlène Läubli Loud 5. Reshaping Evaluation to Enhance Utilization in Scotland: The Role of Intermediary Bodies in Knowledge-to-Action Strategies - Erica Wimbush 6. From Discrete Evaluations to a More Holistic Organizational Approach: The Case of the Public Health Agency of Canada - Nancy Porteous and Steve Montague 7. Evaluation Use Within the European Commission (EC): Lessons for the Evaluation Commissioner - Bastiaan de Laat and Kevin Williams 8. Evaluation Policy and Practice in a Changing Environment: Evolution of the Evaluation Function in the World Health Organization (WHO) - Maria Santamaria, Alan Schnur, and Deepak Thapa 9. Building for Utilization: The Case of the International Labour Organization (ILO) - Janet Neubecker, Matthew Ripley, and Craig Russon 10. What Can We Learn From Practitioners? Some Thoughts and Take-Home Messages for Internal Evaluation Units - Marlène Läubli Loud Index About the Editors About the Contributors

About the Author :
Marlène Läubli Loud (DPhil) is currently an independent consultant and trainer in public sector evaluation. She has worked with a range of organizations, small and large, including the European Commission, the World Health Organization, the United Nations Evaluation Group, the UK Department of Employment, UK Health Promotion Agency (now merged and become NICE), and the English National Board for Nursing, Midwifery and Health Visiting. She was head of the Research and Evaluation Unit at the Swiss Federal Office of Public Health (SFOPH) for nearly twenty years where she gained much experience in evaluation management, and especially in the ways and means for improving the use and utility of evaluation in organizations. She continues to have a keen theoretical and practical interest in this area and is now leading a working group for the Swiss Evaluation Society (SEVAL) on competencies for evaluation managers. Prior to her work with the SFOPH, she was an independent evaluator in the UK, specializing in the evaluation of developmental programs in health and general education. She was also a research fellow at the Department of Education, University of Surrey, and in the Social Science Faculty, University of Oxford, UK. Marlène has facilitated several workshops on public sector evaluation for a range of health and other practitioners. She was a session lecturer in the University of Fribourg’s Sociology and Social Policy master’s program for more than 10 years, and a guest lecturer at several other Swiss universities. She is a member of the European Evaluation Society and the SEVAL. She served on the SEVAL Executive Committee for more than 10 years with special responsibility for professional development. Marlène has many years international experience too. She has worked in several countries including Algeria, Switzerland, the UK, and Italy. She was also part-time Director of the European Office of International Physicians for the Prevention of Nuclear War—winner of the 1985 Nobel Peace Prize. John Mayne (PhD) is an independent advisor on public sector performance. He has been working with a number of organizations and jurisdictions around the world, including several agencies of the UN and development banks, a number of governments and international NGOs, the European Union, the OECD and several Canadian federal departments on results management, evaluation, and accountability issues. Until 2004, he was at the Office of the Auditor General where he led efforts at developing practices for effective managing for results and performance reporting in the government of Canada, as well as leading the office’s audit efforts in accountability and governance. Prior to 1995, John was with the Canadian Treasury Board Secretariat and Office of the Comptroller General. He has authored numerous articles and reports on results management, evaluation, and evaluation methodologies and edited five books in the areas of evaluation, public administration, and performance monitoring. In 1989 and in 1995, he was awarded the Canadian Evaluation Society Award for Contribution to Evaluation in Canada. In 2006, he became a Canadian Evaluation Society Fellow.

Review :
This book is valuable to those involved in the evaluation of public organisations. Particularly insightful is the realization that similar challenges faced in various organizations are addressed with very different solutions. "Overall, this is a good book that is neatly organized. The book can be used in teaching because the examples provide realistic flesh and blood to supplement new students’ learning of dry evaluation theory. Pedagogical discussion questions follow each chapter...Let me add that, as a reader, I feel reassured that the editors and authors collectively are experienced and competent."