Calibrating Standards-based Assessment Tasks for English as a First Foreign Language: Standard-setting Procedures in Germany

Table of Contents:
Chapter 1: Setting Standards in Line with the Common (Claudia Harsch and Simon P. Tiffin-Richards): 1.1 Educational Standards and Standards-based Assessment; 1.1.1 Educational Reform in Germany; 1.1.2 The IQB; 1.1.3 Tests to Evaluate the NES for English as the first Foreign Language (EFL); 1.1.4 Test Development; 1.2 Relating IQB Tests to CEF-levels; 1.2.1 Familiarization; 1.2.2 Specification; 1.2.3 Standardization; 1.2.4 Empirical Validation; 1.3 Rationale for Selecting the Item Pool; 1.4 Description of Panelists; 1.5 Standard-setting Procedure at IQB; 1.5.1 Possible Avenues; 1.5.2 Standard-setting Context and Methods Chosen; 1.6 Research into Validity of the Standard-setting Procedure; Chapter 2: Standard-setting Item Pool (Andre A. Rupp and Raphaela Porsch): 2.1 Overview of Pilot Study Design; 2.1.1 Sampling Process for Students; 2.1.2 Test Design; 2.1.3 Administration of Test Booklets; 2.2 Methodology; 2.3 Analyses for Reading and Listening Comprehension Scales; 2.3.1 CTT Analyses; 2.3.2 IRT Analyses; 2.3.3 Item Fit Analyses; 2.3.4 Item Bias Analyses; 2.3.5 Basic Validity Analyses; 2.3.6 Scale Linking Across Designs; 2.4 Analyses for Written Expression Scale; 2.4.1 Rating Criteria; 2.4.2 Rater Training; 2.5 Descriptive Summary Statistics; 2.6 Results from Multi-faceted Rasch Analyses; Chapter 3: The Bookmark Standard-setting Method (Simon P. Tiffin-Richards): 3.1 Background; 3.2 Bookmark Standard-setting Method; 3.2.1 Design; 3.2.2 Participants; 3.2.3 Materials; 3.2.4 Training; 3.2.5 Methodological Issues; 3.2.6 Data Collection; 3.3 Standard-setting Workshop Results; 3.3.1 Cut-score Summaries; Chapter 4: The Standard-setting Criterion Mapping Method (Karen Draney and Cathleen Kennedy): 4.1 Background; 4.2 Preparation; 4.2.1 Preparation of the Software; 4.2.2 Preparation of the Participants; Chapter 5: Procedural Standard-setting Issues (Karen Draney, Cathleen Kennedy, Steve Moore and Linda Morell): 5.1 Criterion Mapping; 5.1.1 Report of Conduct for Criterion Mapping; 5.1.2 Data Collection; 5.1.3 Issues, Problems, Future Recommendations; 5.2 Bookmark; 5.2.1 Training Session; 5.2.2 Data Collection; 5.3 Summary and Conclusions; Chapter 6: Comparison and Synthesis of Multiple Standard-setting (Simon P. Tiffin-Richards and Olaf Koller): 6.1 Setting Educational Standards; 6.2 The Standard-setting Study; 6.2.1 Panel Composition Factor; 6.2.2 Method Modification Factor; 6.3 General Conclusions; 6.4 Synthesizing and Reporting Results

About the Author :
Olaf Koller, geb. 1963, studierte Psychologie an der Universitat Kiel. Nach AbschluA des Studiums arbeitete er zunachst am Institut fur die Padagogik der Naturwissenschaften in Kiel und wechselte dann an das Max-Planck-Institut fur Bildungsforschung in Berlin, wo er bis 2002 zunachst als wissenschaftlicher Mitarbeiter, spater als Forschungsgruppenleiter (C3) tatig war. Im Jahr 2002 den Lehrstuhl fur Padagogische Psychologie (C4) an der Universitat Erlangen-Nurnberg. 2004 wurde Olaf Koller Grundungsdirektor des Instituts zur Qualitatsentwicklung im Bildungswesen (IQB) und Professor (C4) fur Empirische Bildungsforschung an der Humboldt-Universitat zu Berlin. Nach funfjahriger Tatigkeit wechselte er im Herbst 2009 an das Leibniz-Institut fur die Padagogik der Naturwissenschaften. Dort ist er aktuell Leiter der Abteilung Erziehungswissenschaft und Padagogisch-Psychologische Methodenlehre. Gleichzeitig halt er eine Professur (W3) fur Empirische Bildungsforschung an der CAU. Seine Forschungsinteressen liegen im Bereich der Lehr-Lern-Forschung.