Assessment Engineering in Test Design: Methods and Applications

Assessment Engineering in Test Design presents an engineering-inspired, multi-disciplinary approach to the field of applied measurement that bridges the gap between the craft of item writing and test design, the statistical sophistication of psychometric methods for scale development and maintenance, and score interpretations. This book directly addresses the topic of “next generation” assessment design head-on by proposing a new perspective, a new understanding, of the challenge of designing, developing, and implementing large- (and small-) scale educational testing programs. It introduces Assessment Engineering (AE) as a principled approach to test design where human judgment and creative license during test blueprinting and item writing are replaced with carefully developed cognitive task models that can generate highly consistent test forms that support intended statistical score scale properties and interpretations over time. In doing so, it shows how AE integrates key technologies and design principles from industrial engineering, cognitive science, information technology, machine learning and artificial intelligence, data science, statistical quality control, and advanced psychometrics to the practice of test design and development. Ultimately, this integrated approach offers improved and more consistent overall assessment quality; implementation of highly structured designs and scalable automation to reduce item production costs; reduction of item pretesting costs and exposure; improved calibration and scale maintenance over time; and more consistent generation and deployment of very high-quality, low-cost test items to meet both low- and high-volume production demands over time. Written by a major contributor to the literature on test design and scoring, Assessment Engineering in Test Design is an essential read for all students of psychology and related disciplines as well as test developers and psychometricians.

Table of Contents:
1. An Overview of Assessment Engineering 1.1. Some Definitions of AE 1.2. Limitations of Traditional Test Design 1.2.1. A Test Assembly Example 1.2.2. An AE Perspective on Test Design and Specifications 1.3. AE for Comprehensive Test Design Aligned to Proficiency Claims and Score Interpretations 1.4. Chapter Summary 2. Construct Mapping and Evidence Modeling 2.1. What Are Constructs? 2.2. Limitations of Traditional Score Scale Construction and Their Interpretations 2.2.1. Traditional Development of a Score Scale 2.2.2. Content Blueprints and Standards-Based Alignment 2.2.3. Achievement Level Descriptors and Standard Setting 2.2.4. Item Mapping 2.2.5. Limitations of Traditional Blueprints, Standard Setting, and Item Mapping 2.3. Evidence-Centered Design 2.4. Construct Mapping: An Ordered Progression of Proficiency Claims and Evidence 2.4.1. Choosing a Construct Trajectory 2.4.2. Evidence Models and Proficiency Claims as Building Blocks 2.5. Creating a Construct Map 2.6. Chapter Summary 3. Task Models and Task Model Families 3.1. What Is a Task Model? 3.2. Task Modeling and Cognitive Complexity 3.2.1. Task Model Grammars and the Structure of Task Models 3.2.2. Graphical Representations of Task Model Complexity 3.3. Item Scale Location (Difficulty) and Task Complexity 3.4. Complexity Design Layers 3.5. Chapter Summary 4. Task and Item Difficulty Modeling 4.1. The Need for IDM Research 4.2. Isomorphism and Composability in IDM Research 4.3. Characterizing Statistical Item Difficulty for IDM Research 4.4. Foundations of IDM Research 4.4.1 The Roles of Item Features and Complexity Design Layers 4.4.2 Some Methods for Statistically Modeling Item Difficulty 4.5. Phases in Implementing IDM 4.6. Chapter Summary 5. Task Model Mapping 5.1. Some Limitations of Traditional Test Blueprints and Test Assembly Methods 5.1.1. Conditional Measurement Precision in Test Design and Assembly 5.1.2. The Consequences for Test Assembly of Using Fallible of Content- and Cognitive-Coded Constraints 5.2. Building Task Model Maps (TMMs) 5.3. The Alignment of Complexity and Measurement Precision with a TMM 5.4. Chapter Summary 6. Item Model Families and Automatic Item Generation 6.1. Components and Types of Automatic Item Generation 6.1.1. AI and GPT Applications for AIG 6.1.2. Some Limitations of AIG 6.2. AE-Based Item Models 6.2.1. Item Model Structures and Content 6.2.2. Item Model Quality Control 6.3. Chapter Summary 7. AE Analytics and Quality Control 7.1. Object Analytics 7.1.1. Text Analytics 7.1.2. Image Analytics 7.1.3. Analytics for Audio/Visual Segments 7.1.4. Analytics for Tables and Tabular Data 7.1.5. Analytics for Mathematical Expressions and Equations 7.1.6. Analytics for Numbers/Numerical Sets 7.1.7. Analytics for Item Structures 7.2. Complexity Scoring Protocols 7.3. Psychometric QC Using Conditional Residual Analyses 7.4. An Object Analytic Architecture 7.5. Chapter Summary 8. AE Implementation and Future Directions 8.1. What Problems Does AE Actually Solve? 8.2. Implementing AE: A System of Integrated Systems 8.2.1. Versioning and Robust Integration 8.2.2. Quality Improvement Metrics 8.2.3. High-Level Procedures and Systems 8.3. Future Directions References

About the Author :
Richard M. Luecht is a Professor Emeritus of Educational Research Methodology at UNC-Greensboro. He has designed numerous algorithms and software programs for automated test assembly and devised a computerized adaptive multistage testing framework used by several large-scale testing programs.