With the rapid proliferation of distance education and e-learning courses, the need is growing for a comprehensive, professional approach to evaluating their effectiveness. This indispensable book offers a road map to guide evaluation practice in these innovative learning environments. Providing practical, step-by-step guidelines and tools for conducting evaluation studies—including how to deal with stakeholders, develop surveys and interview protocols, collect other scientific evidence, and analyze and blend mixed-methods data—the work also features a template for writing high-quality reports. The unfolding model developed by the authors draws on Messick's influential assessment framework and applies it to program evaluation. Two case studies of actual programs (a distance learning course and an e-learning course) demonstrate the unfolding model in action.
Table of Contents:
_x000D_ _x000D_ _x000D_
l. Why Do We Need a New Approach to Evaluation in Distance Education and E-Learning?
_x000D_
Distance Education versus E-Learning
_x000D_
The Rapid Expansion of Distance Education and E-Learning
_x000D_
What Is Evaluation?
_x000D_
Why Do We Need a Professional Approach to Evaluation?
_x000D_
What Does a Professional Approach to Evaluation Look Like?
_x000D_
Responding to the Call for a Professional Evaluation Approach: The Unfolding Model
_x000D_
Conclusions: Our Approach to Evaluation
_x000D_
2. The Theory and Practice of Program Evaluation
_x000D_
Why Are Program Evaluation Models Important?
_x000D_
Classification Frameworks for Program Evaluation Models
_x000D_
Alkin and Christie's Evaluation Tree: The Roots and Branches
_x000D_
Where Do We Diverge from Alkin and Christie?
_x000D_
Conclusions
_x000D_
3. Evaluation Theory and Practice in Distance Education and E-Learning
_x000D_
Evaluation Theory
_x000D_
Models with Scientific Evidence in the Foreground
_x000D_
Models Based on Evidence, Values, and Consequences
_x000D_
Models Based on Messick's (1989) Framework
_x000D_
A Summary of Evaluation Models in Distance Education
_x000D_
Evaluation Studies
_x000D_
Evaluation Practice
_x000D_
Do Unintended Consequences Emerge in Authentic Evaluation Studies?
_x000D_
Conclusions
_x000D_
4. Messick's Framework: What Do Evaluators Need to Know?
_x000D_
The Overlap between Test Validity and Program Evaluation
_x000D_
Messick's Contributions
_x000D_
Messick's Framework
_x000D_
The Overlap among the Four Facets
_x000D_
The Controversy over Unintended Consequences
_x000D_
Implications for Evaluation
_x000D_
Conclusions
_x000D_
5. Getting Started
_x000D_
Planning the Evaluation Study
_x000D_
The Ethics Review Process
_x000D_
The Political Context of Evaluation
_x000D_
Using the Unfolding Model as a “Road Map”
_x000D_
Mixed Methods: Blending Quantitative and Qualitative Data
_x000D_
What Is Essential to Our Approach?
_x000D_
Tailoring the Unfolding Model to Your Needs
_x000D_
Conclusions
_x000D_
6. The Unfolding Model: Scientific Evidence
_x000D_
Definition of “Scientific Evidence”
_x000D_
Scientific Evidence
_x000D_
How to Write Good Survey Questions
_x000D_
Using the Unfolding Model to Write Survey and Interview Questions
_x000D_
Administering Surveys
_x000D_
Analyzing Survey Data
_x000D_
Qualitative Data: Interviews, Focus Groups, and Online Ethnographies
_x000D_
Qualitative Data Analysis
_x000D_
Outcomes
_x000D_
The Evaluation of Environmental Quality
_x000D_
Relevance
_x000D_
Cost–Benefit Analysis
_x000D_
Bringing It All Together: Mixed Methods
_x000D_
Conclusions
_x000D_
7. The Unfolding Model: Values and Consequences
_x000D_
Underlying Values
_x000D_
How to Identify the Values Underlying Your Course
_x000D_
Course Goals and Objectives
_x000D_
Writing Survey/Interview Questions about Underlying Values
_x000D_
Analyzing Data on Underlying Values
_x000D_
Unintended Consequences
_x000D_
How Can You Identify Unintended Consequences?
_x000D_
Writing Survey/Interview Questions about Unintended Consequences
_x000D_
Analyzing Data on Unintended Consequences
_x000D_
How to Enhance the Validity of Your Findings
_x000D_
Recommendations for Course Improvement
_x000D_
Writing the Evaluation Report
_x000D_
Conclusions
_x000D_
8. Findings from Two Authentic Case Studies
_x000D_
Methods and Procedures for Both Studies
_x000D_
Distance Learning: Computing Science 200 (CPSC 200)
_x000D_
E-Learning: Professional Writing 110 (PWRIT 110)
_x000D_
Conclusions
_x000D_
9. Bringing It All Together
_x000D_
Using Messick's Framework to Evaluate Distance and E-Learning Courses
_x000D_
Using the Unfolding Model to Evaluate Your Courses
_x000D_
Conducting Your Evaluation Study
_x000D_
What Have We Learned from Two Case Studies?
_x000D_
E-Learning and Beyond: Is the Unfolding Model the Last Word?
_x000D_
The Future of Distance Education and E-Learning
_x000D_
About the Author :
Valerie Ruhe is an Evaluation Studies Specialist at the University of British Columbia. Previously, she was an Assessment and Evaluation Consultant at the Center for Teaching and Learning, University of Minnesota. She has 10 years of professional program evaluation experience in distance education, K-12, and higher education.
Bruno D. Zumbo is Professor of Measurement, Evaluation, and Research Methodology, and of Statistics at the University of British Columbia. He is widely published in research methodology, validity, and validation processes, as well as statistical science and program evaluation methodology. His work has had wide-ranging influence across many fields in the social, educational, and health sciences.
Review :
"Ruhe and Zumbo have written the premier text for evaluation of distance education and e-learning. This is the first theoretically grounded, comprehensive guide for conducting rigorous process and outcome studies of one of the fastest growing segments of curriculum development and education. It will be valuable to those involved in evaluating innovative educational practices and programs, today and for years to come. The book is unique in presenting both evaluation theory and practice, making it an excellent course text and practical resource." - Christina A. Christie, School of Behavioral and Organizational Sciences, Claremont Graduate University, USA "I found this book intriguing and worthy of thoughtful discussion by students, practitioners, and theorists. It provides a good introduction to program evaluation in the context of distance education and e-learning, and to the authors' unfolding model.!Students need this kind of deep discussion of the role of stakeholder values, how to respond dynamically to continual changes in technology, and how to integrate evaluation into the design process to enhance their distance education and e-learning processes and products." - David Dwayne Williams, Department of Instructional Psychology and Technology, Brigham Young University, USA "While there are resources available to aid in the design of e-learning materials, there is very little out there focusing on how to document their effectiveness. This book presents an approach to evaluation and also provides the theory that underlies that approach!.Anyone involved in designing or evaluating e-learning or technology-supported instruction needs to take a look at this book!" - Dianna Newman, Department of Educational and Counseling Psychology, University at Albany, State University of New York, USA "This book should have a significant impact on the field. Distance education is no longer on the periphery of teaching and learning; rather, it is now widely practiced and applied. This book is well written, clearly explained, and relevant to all those who are interested in best practices in distance education and e-learning." - Michael Simonson, Instructional Technology and Distance Education Program, Nova Southeastern University, USA
