Implementation of Large-Scale Education Assessments: (Wiley Series in Survey Methodology)

Presents a comprehensive treatment of issues related to the inception, design, implementation and reporting of large-scale education assessments. In recent years many countries have decided to become involved in international educational assessments to allow them to ascertain the strengths and weaknesses of their student populations. Assessments such as the OECD's Programme for International Student Assessment (PISA), the IEA's Trends in Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy (PIRLS) have provided opportunities for comparison between students of different countries on a common international scale. This book is designed to give researchers, policy makers and practitioners a well-grounded knowledge in the design, implementation, analysis and reporting of international assessments. Readers will be able to gain a more detailed insight into the scientific principles employed in such studies allowing them to make better use of the results. The book will also give readers an understanding of the resources needed to undertake and improve the design of educational assessments in their own countries and regions. Implementation of Large-Scale Education Assessments: Brings together the editors’ extensive experience in creating, designing, implementing, analysing and reporting results on a wide range of assessments. Emphasizes methods for implementing international studies of student achievement and obtaining highquality data from cognitive tests and contextual questionnaires. Discusses the methods of sampling, weighting, and variance estimation that are commonly encountered in international large-scale assessments. Provides direction and stimulus for improving global educational assessment and student learning Is written by experts in the field, with an international perspective. Survey researchers, market researchers and practitioners engaged in comparative projects will all benefit from the unparalleled breadth of knowledge and experience in large-scale educational assessments gathered in this one volume.

Table of Contents:
Notes on Contributors xv Foreword xvii Acknowledgements xx Abbreviations xxi 1 Implementation of Large?]Scale Education Assessments 1 Petra Lietz, John C. Cresswell, Keith F. Rust and Raymond J. Adams 1.1 Introduction 1 1.2 International, Regional and National Assessment Programmes in Education 3 1.3 Purposes of LSAs in Education 4 1.4 Key Areas for the Implementation of LSAs in Education 10 1.5 Summary and Outlook 16 Appendix 1.A 18 References 22 2 Test Design and Objectives 26 Dara Ramalingam 2.1 Introduction 26 2.2 PISA 27 2.3 TIMSS 34 2.4 PIRLS and Pre?]PIRLS 41 2.5 ASER 45 2.6 SACMEQ 52 2.7 Conclusion 56 References 58 3 Test Development 63 Juliette Mendelovits 3.1 Introduction 63 3.2 Developing an Assessment Framework: A Collaborative and Iterative Process 65 3.3 Generating and Collecting Test Material 68 3.4 Refinement of Test Material 72 3.5 Beyond Professional Test Development: External Qualitative Review of Test Material 81 3.6 Introducing Innovation 86 3.7 Conclusion 90 References 90 4 Design, Development and Implementation of Contextual Questionnaires in Large?]Scale Assessments 92 Petra Lietz 4.1 Introduction 92 4.2 The Role of Questionnaires in LSAs 93 4.3 Steps in Questionnaire Design and Implementation 95 4.4 Questions and Response Options in LSAs 115 4.5 Alternative Item Formats 119 4.6 Computer?]Based/Online Questionnaire Instruments 128 4.7 Conclusion and Future Perspectives 131 Acknowledgements 132 References 132 5 Sample Design, Weighting, and Calculation of Sampling Variance 137 Keith F. Rust, Sheila Krawchuk and Christian Monseur 5.1 Introduction 137 5.2 Target Population 138 5.3 Sample Design 144 5.4 Weighting 148 5.5 Sampling Adjudication Standards 153 5.6 Estimation of Sampling Variance 156 References 166 6 Translation and Cultural Appropriateness of Survey Material in Large?]Scale Assessments 168 Steve Dept, Andrea Ferrari and Béatrice Halleux 6.1 Introduction 168 6.2 Overview of Translation/Adaptation and Verification Approaches Used in Current Multilingual Comparative Surveys 169 6.3 Step?]by?]Step Breakdown of a Sophisticated Localisation Design 174 6.4 Measuring the Benefits of a Good Localisation Design 184 6.5 Checklist of Requirements for a Robust Localisation Design 190 References 191 7 Quality Assurance 193 John C. Cresswell 7.1 Introduction 193 7.2 The Development and Agreement of Standardised Implementation Procedures 194 7.3 The Production of Manuals which Reflect Agreed Procedures 196 7.4 The Recruitment and Training of Personnel in Administration and Organisation: Especially the Test Administrator and the School Coordinator 197 7.5 The Quality Monitoring Processes: Recruiting and Training Quality Monitors to Visit National Centres and Schools 198 7.6 Other Quality Monitoring Procedures 201 7.7 Conclusion 204 Reference 204 8 Processing Responses to Open?]Ended Survey Questions 205 Ross Turner 8.1 Introduction 205 8.2 The Fundamental Objective 207 8.3 Contextual Factors: Survey Respondents and Items 207 8.4 Administration of the Coding Process 214 8.5 Quality Assurance and Control: Ensuring Consistent and Reliable Coding 221 8.6 Conclusion 229 References 229 9 Computer?]Based Delivery of Cognitive Assessment and Questionnaires 231 Maurice Walker 9.1 Introduction 231 9.2 Why Implement Computer?]Based Assessments? 232 9.3 Implementation of International Comparative Computer?]Based Assessments 238 9.4 Assessment Architecture 244 9.5 Item Design Issues 247 9.6 State?]of?]the?]Art and Emerging Technologies 250 9.7 Summary and Conclusion 250 References 251 10 Data Management Procedures 253 Falk Brese and Mark Cockle 10.1 Introduction 253 10.2 Historical Review: From Data Entry and Data Cleaning to Integration into the Entire Study Process 254 10.3 The Life Cycle of a LSA Study 255 10.4 Standards for Data Management 256 10.5 The Data Management Process 258 10.6 Outlook 272 References 274 11 Test Implementation in the Field: The Case of PASEC 276 Oswald Koussihouèdé, Antoine Marivin and Vanessa Sy 11.1 Introduction 276 11.2 Test Implementation 278 11.3 Data Entry 294 11.4 Data Cleaning 295 11.5 Data Analysis 295 11.6 Governance and Financial Management of the Assessments 295 Acknowledgments 296 References 297 12 Test Implementation in the Field: The Experience of Chile in International Large?]Scale Assessments 298 Ema Lagos Campos 12.1 Introduction 298 12.2 International Studies in Chile 302 Annex A 321 References 321 13 Why Large?]Scale Assessments Use Scaling and Item Response Theory 323 Alla Berezner and Raymond J. Adams 13.1 Introduction 323 13.2 Item Response Theory 325 13.3 Test Development and Construct Validation 329 13.4 Rotated Test Booklets 345 13.5 Comparability of Scales Across Settings and Over Time 347 13.6 Construction of Performance Indicators 349 13.7 Conclusion 354 References 354 14 Describing Learning Growth 357 Ross Turner and Raymond J. Adams 14.1 Background 357 14.2 Terminology: The Elements of a Learning Metric 358 14.3 Example of a Learning Metric 360 14.4 Issues for Consideration 360 14.5 PISA Described Proficiency Scales 365 14.6 Defining and Interpreting Proficiency Levels 374 14.7 Use of Learning Metrics 379 Acknowledgement 380 References 381 15 Scaling of Questionnaire Data in International Large?]Scale Assessments 384 Wolfram Schulz 15.1 Introduction 384 15.2 Methodologies for Construct Validation and Scaling 386 15.3 Classical Item Analysis 387 15.4 Exploratory Factor Analysis 388 15.5 Confirmatory Factor Analysis 389 15.6 IRT Scaling 392 15.7 Described IRT Questionnaire Scales 396 15.8 Deriving Composite Measures of Socio?]economic Status 399 15.9 Conclusion and Future Perspectives 404 References 405 16 Database Production for Large?]Scale Educational Assessments 411 Eveline Gebhardt and Alla Berezner 16.1 Introduction 411 16.2 Data Collection 412 16.3 Cleaning, Recoding and Scaling 416 16.4 Database Construction 418 16.5 Assistance 421 References 423 17 Dissemination and Reporting 424 John C. Cresswell 17.1 Introduction 424 17.2 Frameworks 425 17.3 Sample Items 426 17.4 Questionnaires 427 17.5 Video 427 17.6 Regional and International Reports 428 17.7 National Reports 428 17.8 Thematic Reports 429 17.9 Summary Reports 429 17.10 Analytical Services and Support 430 17.11 Policy Papers 430 17.12 Web?]Based Interactive Display 431 17.13 Capacity?]Building Workshops 432 17.14 Manuals 432 17.15 Technical Reports 432 17.16 Conclusion 433 References 433 Index 436

About the Author :
Edited by Petra Lietz and John C. Cresswell, Australian Council for Educational Research (ACER), Australia Keith F. Rust, Westat and the University of Maryland at College Park, USA Raymond J. Adams, Australian Council for Educational Research (ACER), Australia