Observational Measurement of Behavior

This book is a comprehensive textbook designed to introduce graduate students to research methods and techniques used in observational measurement of behavior, along with the theoretical principles underlying these methods and techniques. The aspects of observational measurement addressed are highly relevant to special education, communication sciences, psychology, social work, and related fields. Chapters 1–8 discuss foundational topics in observational measurement of behavior: systematic observation, validity and reliability, measurement theory, the development of coding manuals, coding, common metrics of observational variables, observer training, interobserver reliability, and more. Chapters 9–11 address advanced topics: sequential analysis, research questions involving sequential associations, and generalizability theory. Chapter 12 is a “‘wrap-up” chapter summarizing best practices for observational measurement. Each chapter begins with an overview of the topics covered and ends with the authors’ conclusions and practical recommendations regarding aspects of observational measurement. Throughout, the authors provide real-world examples based on their extensive experience studying communication behaviors. Recommended methods are discussed in relation to currently available observational software. Definitions of key terms are prominently featured in the text and compiled in the Glossary. The online companion materials include exercises and assignments that provide opportunities for readers to apply the principles and practices discussed throughout the book; these comprehensive documents are an excellent resource for course instructors.

Table of Contents:
Table of Contents Draft 10/7/16   1: Introduction and Measurement Contexts Overview Definition of Systematic Observation Using Count Coding Rationale for Systematic Observation Using Count Coding Importance of Falsifiable Hypotheses The Continuum of State-Likeness to Trait-Likeness Context-Dependent Behavior Person Characteristics Generalized Behavioral Tendencies Skills The Relative Scientific Value of Different Objects of Measurement Ecological Validity and Representativeness Conclusions and Recommendations References   2: Validation of Observational Variables Overview The Changing Concept of Validation Understanding Which Types of Validation Evidence Are Most Relevant for Different Research Designs, Objects of Measurement, and Research Purposes  Content Validation  Definition of Content Validation  Different Traditions Vary on the Levels of Importance Placed on Content Validation  Weaknesses of Content Validation Sensitivity to Change Definition of Sensitivity to Change Influences on Sensitivity to Change Weaknesses of Sensitivity to Change Treatment Utility Definition of Treatment Utility Weaknesses of Treatment Utility  Criterion-Related Validation Definition of Criterion-Related Validation Primary Appeal of Criterion-Related Validation Weaknesses of Criterion-Related Validation Construct Validation Definition of Construct Validation Discriminative Validation  Nomological Validation Multitrait, Multimethod Validation An Implicit “Weakness” of Science? Recommendations  References   3: Measuring Person Characteristics Overview Contextual Measurement Error Definition of Measurement Context A Brief Overview of Measurement Theory Definition of Contextual Measurement Error Representativeness Contextual Measurement Error in Measures Of Generalized Behavioral Tendencies Averaging Scores Across Contexts Improves Measures of Generalized Behavioral Tendencies Aggregates Tend to Improve Estimates of Known True Score. Aggregates Tend to Improve Construct Validity. Aggregates Tend to Improve Stability. Controlling Influential Contextual Variables Stabilizes Observed Scores for Highly Variable Person Characteristics Why Naturalistic Observations Are Not Necessarily More Representative Than Contrived Ones Why Skills Are Often Measured in Structured Measurement Contexts Why Skills Are Often Assessed in Clinics or Labs The Link Between Stability and Construct Validity Recommendations and Conclusions References   4: Designing or Adapting Coding Manuals Overview Definition of a Coding Manual Deciding Whether to Use an Existing Coding Manual or to Construct a New One Recommended Steps for Modifying or Designing Coding Manuals Defining When to Start and Stop Coding Conceptually Defining the Context-Dependent Behavior or the Generalized Characteristic Defining the Highest Level of Codable Behavior Determining the Level of Distinction Coders Have to Make Organizing the Coded Categories into Mutually-Exclusive Sets Physically Based Definitions, Socially Based Definitions, or Both? Defining the Lowest Level Categories Source of Conceptual and Operational Definitions A Qualitative Approach to Identifying Definitions Defining Segmenting Rules The Potential Value of Flowcharts Do Coding Manuals Need to be Sufficiently Short to be Included in Methods Sections? Recommendations and Conclusions References   5: Coding Overview The Elements of an Observational Measurement System Behavior Sampling The Superordinate Distinctions: Continuous versus Intermittent The Subordinate Distinctions: Timed-Event versus Event versus Interval Timed-Event Sampling Event Sampling Interval Sampling Types of Interval Sampling Whole Interval Sampling Momentary Interval Sampling Partial Interval Sampling The Importance of Knowing What Metric the Investigator Wants to Estimate Summary of Behavior Sampling Participant Sampling Focal Sampling Multiple Pass Sampling Conspicuous Sampling Reactivity Live Coding versus Recording the Observation For Later Coding Live Coding Coding from Recorded Sessions Recording Coding Decisions Recommendations and Conclusions References   6: Common Metrics of Observational Variables Overview Definition of Metric Quantifiable Dimensions of Behavior Proportion Metrics Proportion Metrics Change the Meaning of Observational Variables Scrutinizing Proportions An Implicit Assumption of Proportion Metrics Testing Whether the Data Fit the Assumption of Proportion Metrics Consequences of Using a Proportion When the Data Do Not Fit the Assumption Alternative Methods to Control Influential Contextual Variables Statistical Control Procedural Control Aggregate Measures of Generalized Person Characteristics Weighted Count Unit-Weighted Aggregates Group Analysis of Observational Variables Transforming the Metric Bootstrapping Recommendations and Conclusions References   7: Observer Training and Preventing Observer Drift Overview Point-by-point Agreement and Disagreement Point-by-point Agreement of Interval Sampled Data Point-by-point Agreement of Timed-Event Data Discrepancy Matrices Discrepancy Discussions Using Discrepancy Discussions to Train Observers Creating Criterion-coding Standards. Remaining Steps to Train Observers Preventing Observer Drift Method of Selecting Sessions for Agreement Checks Remaining Steps to Preventing or Addressing Observer Drift Recommendations References   8: Interobserver Agreement and Reliability of Observational Variables Overview Additional Purposes of Point-by-Point Agreement  Added Principles When Agreement Checks Are Used to Estimate Interobserver “Reliability” of Observational Variable Scores  Exhaustive Coding Spaces Revisited The Effect of Chance on Agreement  Common Indices of Point-by-Point Agreement Occurrence Percentage Agreement Nonoccurrence Percentage Agreement Total Percentage Agreement Kappa  Base Rate and Chance Agreement Revisited  Intraclass Correlation Coefficient (ICC) as an Index of Interobserver Reliability in Group Designs Options for Running ICC With SPSS  Between-Participant Variance on the Variable of Interest Affects ICC Using ICC as a Measure of Interobserver Reliability for Predictors and Dependent Variables in Group Designs The Interpretation of SPSS Output for ICC  The Conceptual Relation Between Interobserver Agreement and ICC  Consequences of Low or Unknown Interobserver Reliability Recommendations References   9: Introduction to Sequential Analysis Overview Definition of Terms Used in this Chapter Sequential versus Nonsequential Variables Sequential Associations are not Sufficient Evidence for Causal Inferences Coded Units and Exhaustiveness Contingency Tables Three Major Types of Sequential Analysis Event Lag Concurrent Interval Event Lag with Pauses (to replace time window method) Explanation for no longer focusing on time window Indices of Sequential Association: Controlling for Chance Existing Indices of Sequential Association: Advantages and Disadvantages Transitional/conditional probabilities Yule’s Q Risk Difference/Operant Contingency Value Other commonly-used indices and why we do not focus on them (e.g., z) Recommendations and Conclusions References   10: Identifying and Addressing Research Questions Involving Sequential Associations Overview Sequential Analysis in Group Designs Types of Research Questions and Methods to Address Them Indices of Sequential Association as Dependent Variables Testing the Significance of a Mean Sequential Association Testing the Between-Group Difference in Mean Sequential Associations Testing the Within-Subject Difference in Sequential Associations Sequential Analysis in Single-Case Designs Types of Research Questions Descriptive Questions Involving Behavior-Environment Associations to Inform Experimental Analyses Descriptive Questions to Inform Temporal Distribution of One or More Target Behaviors Indices of Sequential Association as Dependent Variables in Single-Case Designs Indices of Sequential Association to Inform Procedural Fidelity Methods to Address Sequential Analysis Research Questions in a Single Case Framework Why Significance Testing is Inappropriate at the Level of the Individual Participant More on the Term ‘Operant Contingency’ Summary of Analysis Methods Used in Behavior Analytic Literature Conditional Probabilities Lag Sequential Analysis Contingency Space Analysis  What is “Enough Data” and How Do We Attain It? Proposed Solutions for Insufficient Data Summary and Recommendations References   11: Generalizability Theory Overview Scope of This Chapter Overview of G Theory and Definition of Terms An Example Observer by Context G and D Study Rationale for Preferring the Absolute G Coefficient Example Applications of D Studies An Ongoing Controversy Recommendations and Conclusions References   12: Best Practices in Observational Measurement   Glossary Index