Validity and Limitations of College Student Self-Report Data: New Directions for Institutional Research, Number 150(112 J-B IR Single Issue Institutional Research)

Critics of student self-reported data claim that the accumulated corpus of research documenting student learning on the basis of survey responses stands on shaky ground. This volume argues that scholarship on proper use of student self-report data is woefully underdeveloped and contributing authors offer several important insights to assist IR practitioners in identifying potential limitations associated with self-report data. Volume editors Serge Herzog, director of institutional analysis at the University of Nevada, Reno, and Nicholas A. Bowman, postdoctoral research associate in the Center for Social Concerns at the University of Notre Dame, have assembled contributing authors who are leading scholars in the field of college student self-reports. Combined, the chapters draw on data from a mix of colleges and universities, capturing student growth at different stages of the undergraduate experience, and even beyond graduation. This is the 150th volume of the Jossey-Bass quarterly report series New Directions for Institutional Research. Always timely and comprehensive, New Directions for Institutional Research provides planners and administrators in all types of academic institutions with guidelines in such areas as resource coordination, information analysis, program evaluation, and institutional management.

Table of Contents:
Editors' Notes 1 Serge Herzog, Nicholas A. Bowman 1. Examining Systematic Errors in Predictors of College Student Self-Reported Gains 7 Nicholas A. Bowman A potential concern with using self-reported gains as a proxy for longitudinal growth is that predictor variables might provide consistently different results depending on how the outcome is measured. This chapter explores that possibility using a longitudinal sample of 8,615 fi rst-year college students at forty-nine institutions; it includes selfreported gains and longitudinal measures of the same constructs. The analysis examines differences associated with student characteristics and institution type, and the chapter uses cognitive and social psychological theory to anchor the fi ndings. 2. Gauging Academic Growth of Bachelor Degree Recipients: Longitudinal vs. Self-Reported Gains in General Education 21 Serge Herzog Few institutional research practitioners enjoy access to longitudinal data that measure changes in student mastery of general education skills. This chapter shows to what degree student self-reported gains in an alumni survey correspond to objective longitudinal changes in scores on widely used high-stakes standardized tests. Controlling for student sociodemographic background and academic experience, the analyses explore the infl uence of postgraduate employment status, field of study, and interaction with faculty on perceived versus longitudinally measured learning gains. 3. Using College Students' Self-Reported Learning Outcomes in Scholarly Research 41 Gary R. Pike The debate over validity of student self-reported learning should be informed by established theory on the role of student personality and academic discipline, and how self-reported learning is infl uenced by these two factors, as argued in this chapter. Using Holland's personenvironment fi t theory of vocational and educational behavior, the author examines the association between four types of self-reported learning outcomes and corresponding academic disciplines. 4. The Tie That Binds: The Role of Self-Reported High School Gains in Self-Reported College Gains 59 Tricia A. Seifert, Ashley M. Asel This chapter analyzes data from the Research on the Iowa Student Experience study to determine the extent to which students' retrospective self-reported gains from high school are associated with college self-reported gains. Given previous research on self-reported gains, the study hypothesizes a significant link between these measures and explores whether that relationship varies between fi rst-year and senior students. 5. Measuring How College Affects Students: Social Desirability and Other Potential Biases in College Student Self-Reported Gains 73 Nicholas A. Bowman, Patrick L. Hill Social desirability may be one of several potential sources of bias in student self-reports. The authors of this chapter weigh the infl uence of this and other factors that may compromise the validity of student survey responses, including satisfaction with college, narcissism, sense of personal growth, and self-esteem. The impact of these factors is measured for both fi rst-year and advanced students, taking into account retrospectively perceived gains attained prior to entering college. 6. The Validity of Student Engagement Survey Questions: Can We Accurately Measure Academic Challenge? 87 Stephen R. Porter, Corey Rumann, Jason Pontius Our understanding of whether students feel academically challenged typically hinges on what they report in popular surveys such as the National Survey of Student Engagement. To examine the accuracy of what students report, this study compares survey responses with actual assignments documented in course syllabi for the courses that students completed. Specifi cally, the authors determine whether self-reported assignments for the most recent semester are signifi cantly associated with assignments documented in the syllabi. 7. Clearing the AIR About the Use of Self-Reported Gains in Institutional Research 99 Robert M. Gonyea, Angie Miller This chapter clarifi es and conceptualizes student self-reported gains from the perspective of two researchers affi liated with the National Survey of Student Engagement. The authors present new fi ndings on the degree to which social desirability bias may (or may not) skew NSSE data. They furnish advice on proper use of self-report data, discuss efforts to improve the NSSE as a suitable instrument to gauge student impressions of their learning, and describe how that information is a valuable complement to other sources of data used by institutional researchers. 8. Reconciling (Seemingly) Discrepant Findings: Implications for Practice and Future Research 113 Nicholas A. Bowman, Serge Herzog Synthesizing results from the previous chapters, the authors discuss the various conditions examined in this volume that may infl uence the congruence between student self-reported and objective longitudinal measures of growth. Limitations and proper use of self-report data are discussed in the context of improving the practice of institutional research. Index 121