Design, Evaluation, and Analysis of Questionnaires for Survey Research: (Wiley Series in Survey Methodology)

Table of Contents:
Preface to the Second Edition xiii Preface xv Acknowledgments xvii Introduction 1 I.1 Designing a Survey 4 I.1.1 Choice of a Topic 4 I.1.2 Choice of the Most Important Variables 4 I.1.3 Choice of a Data Collection Method 5 I.1.4 Choice of Operationalization 6 I.1.5 Test of the Quality of the Questionnaire 8 I.1.6 Formulation of the Final Questionnaire 9 I.1.7 Choice of Population and Sample Design 9 I.1.8 Decide about the Fieldwork 10 I.1.9 What We Know about These Decisions 10 I.1.10 Summary 11 Exercises 12 Part I The Three-Step Procedure to Design Requests for Answers 13 1 Concepts-by-Postulation and Concepts-by-Intuition 15 1.1 Concepts-by-Intuition and Concepts-by-Postulation 15 1.2 Different Ways of Defining Concepts-by-Postulation through Concepts-by-Intuition 19 1.2.1 Job Satisfaction as a Concept-by-Intuition 19 1.2.2 Job Satisfaction as a Concept-by-Postulation 20 1.3 Summary 27 Exercises 28 2 From Social Science Concepts-by-Intuition to Assertions 30 2.1 Basic Concepts and Concepts-by-Intuition 31 2.2 Assertions and Requests for an Answer 32 2.3 The Basic Elements of Assertions 33 2.3.1 Indirect Objects as Extensions of Simple Assertions 36 2.3.2 Adverbials as Extensions of Simple Assertions 37 2.3.3 Modifiers as Extensions of Simple Assertions 37 2.3.4 Object Complements as Extensions of Simple Assertions 38 2.3.5 Some Notation Rules 38 2.4 Basic Concepts-by-Intuition 39 2.4.1 Subjective Variables 40 2.4.2 Objective Variables 47 2.4.3 In Summary 49 2.5 Alternative Formulations for the Same Concept 49 2.6 Extensions of Simple Sentences 51 2.6.1 Adding Indirect Objects 51 2.6.2 Adding Modifiers 52 2.6.3 Adding Adverbials 52 2.7 Use of Complex Sentences 53 2.7.1 Complex Sentences with No Shift in Concept 54 2.7.2 Complex Sentences with a Shift in Concept 54 2.7.3 Adding Conditions to Complex Sentences 56 2.8 Summary 56 Exercises 57 3 The Formulation of Requests for an Answer 60 3.1 From Concepts to Requests for an Answer 61 3.2 Different Types of Requests for an Answer 63 3.2.1 Direct Request 63 3.2.2 Indirect Request 66 3.3 The Meaning of Requests for an Answer with WH Request Words 69 3.3.1 “When,” “Where,” and “Why” Requests 70 3.3.2 “Who” Requests 70 3.3.3 “Which” Requests 70 3.3.4 “What” Requests 71 3.3.5 “How” Requests 72 3.4 Summary 74 Exercises 75 Part II Choices Involved in Questionnaire Design 77 4 Specific Survey Research Features of Requests for an Answer 79 4.1 Select Requests from Databases 79 4.2 Other Features Connected with the Research Goal 81 4.3 Some Problematic Requests 83 4.3.1 Double-Barreled Requests 83 4.3.2 Requests with Implicit Assumptions 84 4.4 Some Prerequests Change the Concept-by-Intuition 85 4.5 Batteries of Requests for Answers 86 4.5.1 The Use of Batteries of Stimuli 87 4.5.2 The Use of Batteries of Statements 88 4.6 Other Features of Survey Requests 92 4.6.1 The Formulation of Comparative or Absolute Requests for Answers 92 4.6.2 Conditional Clauses Specified in Requests for Answers 93 4.6.3 Balanced or Unbalanced Requests for Answers 93 4.7 Special Components within the Request 95 4.7.1 Requests for Answers with Stimulation for an Answer 95 4.7.2 Emphasizing the Subjective Opinion of the Respondent 95 4.8 Summary 96 Exercises 96 5 Response Alternatives 98 5.1 Open Requests for an Answer 99 5.2 Closed Categorical Requests 101 5.2.1 Nominal Categories 103 5.2.2 Ordinal Scales 104 5.2.3 Continuous Scales 108 5.3 How Many Categories are Optimal? 111 5.4 Summary 112 Exercises 114 6 The Structure of Open-Ended and Closed Survey Items 115 6.1 Description of the Components of Survey Items 115 6.2 Different Structures of Survey Items 118 6.2.1 Open-Ended Requests for an Answer 119 6.2.2 Closed Survey Items 120 6.2.3 The Frequency of Occurrence 124 6.2.4 The Complexity of Survey Items 125 6.3 What Form of Survey Items Should Be Recommended? 126 6.4 Summary 127 Exercises 128 7 Survey Items in Batteries 130 7.1 Batteries in Oral Interviews 131 7.2 Batteries in Mail Surveys 134 7.3 Batteries in CASI 138 7.4 Summary and Discussion 142 Exercises 144 8 Mode of Data Collection and Other Choices 146 8.1 The Choice of the Mode of Data Collection 147 8.1.1 Relevant Characteristics of the Different Modes 148 8.1.2 The Presence of the Interviewer 149 8.1.3 The Mode of Presentation 151 8.1.4 The Role of the Computer 152 8.1.5 Procedures without Asking Questions 155 8.1.6 Mixed-Mode Data Collection 155 8.2 The Position in the Questionnaire 156 8.3 The Layout of the Questionnaire 158 8.4 Differences due to Use of Different Languages 158 8.5 Summary and Discussion 159 Exercises 160 Part III Estimation and Prediction of the Quality of Questions 163 9 Criteria for the Quality of Survey Measures 165 9.1 Different Methods, Different Results 166 9.2 How These Differences Can Be Explained 173 9.2.1 Specifications of Relationships between Variables in General 173 9.2.2 Specification of Measurement Models 175 9.3 Quality Criteria for Survey Measures and Their Consequences 178 9.4 Alternative Criteria for Data Quality 181 9.4.1 Test–Retest Reliability 181 9.4.2 The Quasi-simplex Approach 182 9.4.3 Correlations with Other Variables 183 9.5 Summary and Discussion 184 Exercises 185 Appendix 9.1 The Specification of Structural Equation Models 187 10 Estimation of Reliability, Validity, and Method Effects 190 10.1 Identification of the Parameters of a Measurement Model 191 10.2 Estimation of Parameters of Models with Unmeasured Variables 195 10.3 Estimating Reliability, Validity, and Method Effects 197 10.4 Summary and Discussion 201 Exercises 202 Appendix 10.1 Input of Lisrel for Data Analysis of a Classic MTMM Study 205 Appendix 10.2 Relationship between the TS and the Classic MTMM Model 205 11 Split-Ballot Multitrait–Multimethod Designs 208 11.1 The Split-Ballot MTMM Design 209 11.1.1 The Two-Group Design 209 11.1.2 The Three-Group Design 210 11.1.3 Other SB-MTMM Designs 211 11.2 Estimating and Testing Models for Split-Ballot MTMM Experiments 212 11.3 Empirical Examples 213 11.3.1 Results for the Three-Group Design 213 11.3.2 Two-Group SB-MTMM Design 215 11.4 The Empirical Identifiability and Efficiency of the Different SB-MTMM Designs 218 11.4.1 The Empirical Identifiability of the SB-MTMM Model 218 11.4.2 The Efficiency of the Different Designs 221 11.5 Summary and Discussion 221 Exercises 222 Appendix 11.1 The Lisrel Input for the Three-Group SB-MTMM Example 222 12 MTMM Experiments and the Quality of Survey Questions 225 12.1 The Data from the MTMM Experiments 226 12.2 The Coding of the Characteristics of the MTMM Questions 229 12.3 The Database and Some Results 230 12.3.1 Differences in Quality across Countries 231 12.3.2 Differences in Quality for Domains and Concepts 234 12.3.3 Effect of the Question Formulation on the Quality 235 12.4 Prediction of the Quality of Questions Not Included in the MTMM Experiments 237 12.4.1 Suggestions for Improvement of Questions 239 12.4.2 Evaluation of the Quality of the Prediction Models 240 12.5 Summary 241 Exercises 242 Part IV Applications in Social Science Research 243 13 The SQP 2.0 Program for Prediction of Quality and Improvement of Measures 245 13.1 The Quality of Questions Involved in the MTMM Experiments 246 13.1.1 The Quality of Specific Questions 246 13.1.2 Looking for Optimal Measures for a Concept 250 13.2 The Quality of Non-MTMM Questions in the Database 252 13.3 Predicting the Quality of New Questions 256 13.4 Summary 261 Exercises 262 14 The Quality of Measures for Concepts-by-Postulation 263 14.1 The Structures of Concepts-by-Postulation 264 14.2 The Quality of Measures of Concepts-by-Postulation with Reflective Indicators 264 14.2.1 Testing the Models 265 14.2.2 Estimation of the Composite Scores 268 14.2.3 The Quality of Measures for Concepts-by-Postulation 270 14.2.4 Improvement of the Quality of the Measure 274 14.3 The Quality of Measures for Concepts-by-Postulation with Formative Indicators 276 14.3.1 Testing the Models 278 14.3.2 Estimation of the Composite Score 281 14.3.3 The Estimation of the Quality of the Composite Scores 282 14.4 Summary 283 Exercises 284 Appendix 14.1 Lisrel Input for Final Analysis of the Effect of “Social Contact” on “Happiness” 284 Appendix 14.2 Lisrel Input for Final Analysis of the Effect of “Interest in Political Issues in the Media” on “Political Interest in General” 285 15 Correction for Measurement Errors 287 15.1 Correction for Measurement Errors in Models with only Concepts-by-Intuition 287 15.2 Correction for Measurement Errors in Models with Concepts-by-Postulation 292 15.2.1 Operationalization of the Concepts 292 15.2.2 The Quality of the Measures 294 15.2.3 Correction for Measurement Errors in the Analysis 297 15.3 Summary 298 Exercises 299 Appendix 15.1 Lisrel Inputs to Estimate the Parameters of the Model in Figure 15.1 300 Appendix 15.2 Lisrel Input for Estimation of the Model with Correction for Measurement Errors using Variance Reduction by Quality for all Composite Scores 301 16 Coping with Measurement Errors in Cross-Cultural Research 302 16.1 Notations of Response Models for Cross-Cultural Comparisons 303 16.2 Testing for Equivalence or Invariance of Instruments 307 16.2.1 The Standard Approach to Test for Equivalence 307 16.3 Problems Related with the Procedure 309 16.3.1 Using Information about the Power of the Test 309 16.3.2 An Alternative Test for Equivalence 315 16.3.3 The Difference between Significance and Relevance 317 16.4 Comparison of Means and Relationships across Groups 318 16.4.1 Comparison of Means and Relationships between Single Requests for Answers 318 16.4.2 Comparison of Means and Relationships Based on Composite Scores 319 16.4.3 Comparison of Means and Relationships between Latent Variables 321 16.5 Summary 324 Exercises 325 Appendix 16.1 The Two Sets of Requests Concerning “Subjective Competence” 326 Appendix 16.2 ESS Requests Concerning “Political Trust” 327 Appendix 16.3 The Standard Test of Equivalence for “Subjective Competence” 328 Appendix 16.4 The Alternative Equivalence Test for “Subjective Competence” in Three Countries 329 Appendix 16.5 Lisrel Input to Estimate the Null Model for Estimation of the Relationship between “Subjective Competence” and “Political Trust” 331 Appendix 16.6 Derivation of the Covariance between the Composite Scores 333 References 336 Index 352

About the Author :
WILLEM E. SARIS, PHD, is Emeritus Professor in Methodology of the University of Amsterdam and the Universitat Pompeu Fabra, Barcelona. He is Laureate of the 2005 Descartes Prize for “Best Collaborative Research” as member of the Central Coordinating Team of the European Social Survey (ESS) and Recipient of the World Association of Public Opinion Research’s “Helen Dinerman Award” in 2009 for his lifelong contribution to the methodology of Opinion Research. Dr. Saris also received the “2013 Outstanding Service Prize” of the European Survey Research Association. IRMTRAUD N. GALLHOFER, PHD, is a linguist and was senior researcher on projects of the ESS, Research and Expertise Centre for Survey Methodology at the Universitat Pompeu Fabra, Barcelona. She is Laureate of the 2005 Descartes Prize for “Best Collaborative Research” as a member of the Central Coordinating Team of the ESS.