Abstract This study used integrative data analysis (Curran & Hussong, 2009), which allows combining data from different studies, to examine the generalizability of a research-based mathematics program, schema-based instruction (SBI), with a focus on proportional reasoning. Data were pooled from existing SBI studies spanning three U.S. states in which teachers and their classrooms were randomly assigned to either a SBI or control condition. Results of a series of multilevel (i.e., two-level, students within teachers) models indicated the effectiveness of SBI in improving students’ mathematical problem-solving performance. The pooled data findings clarify that SBI improves students’ mathematical problem solving more so for proximal than distal measures. These findings also show Black and Hispanic students continue to lag behind White students, that teachers with more experience tend to be associated with higher performing classrooms, and that increasing concentrations of students eligible for free or reduced price lunch and students receiving special education services are generally associated with weaker performances. Implications of these findings are discussed.
Design-based methods have recently been developed as a way to analyze randomized controlled trial (RCT) data for designs with a single treatment and control group. This article builds on this framework to develop design-based estimators for evaluations with multiple research groups. Results are provided for a wide range of designs used in education research, including clustered and blocked designs. Because analysis in the multi-armed setting involves pairwise contrasts across the research groups, the key methodological question addressed is: How do the estimators for the two-group design need to be adjusted for multi-armed trials? The critical insight is that in multi-armed trials where the goal is to identify the most effective treatments, the samples for each pairwise contrast are representative of the full set of randomized units, not just of themselves. The implication is that variance terms need to be adjusted slightly under the finite-population framework that can reduce precision, and blocks need to be weighted to reflect the full randomized sample in the block or biases can result. An empirical example using data from a multi-armed education RCT demonstrates the issues.
