Date of Completion

9-9-2019

Embargo Period

3-7-2020

Keywords

Generalizability, Randomized Experiment, Multilevel Data Structure

Major Advisor

Christopher Rhoads

Associate Advisor

H Jane Rogers

Associate Advisor

Hariharan Swaminathan

Associate Advisor

Tania Huedo-Medina

Associate Advisor

Aarti Bellara

Field of Study

Educational Psychology

Degree

Doctor of Philosophy

Open Access

Open Access

Abstract

Over the past decade, the generalizability of randomized experiments, defined as the level of consistency between an estimated treatment effect in a sample and the true treatment effect in a target population, has received increasing attention from the educational research community. Existing methods focus on either: (a) prospectively preventing or (b) retrospectively adjusting away the bias caused by the nonrandom selection of institutions, such as schools, into a study sample. This study explores methods to adjust away the bias caused by both the between-institution and within-institution selection processes. For instance, in educational studies we desire to account for both nonrandom school-level selection and non-random selection of students and/or teachers. I conduct a simulation study to evaluate the bias reducing properties of different methods for estimating and utilizing inverse probability of participation (IPP) weights in this two-level context. The simulation study found that methods that incorporated both student and school IPP weights reduced more bias than methods that only incorporated the school IPP weights. Using data from a cluster randomized trial of a math professional development intervention, this study finds that within participating schools, the participating sample of students were more “advantaged” than the non-participating students, suggesting the need to adjust for within school nonrandom selection. The study estimated and applied IPP weights to reduce bias in the estimated population average treatment effect for the intervention.

Download
COinS