A three-year study finds that students who had been exposed to more reform-oriented teaching performed slightly better in both math and science than those who had experienced less. Evidence also suggests that reform-oriented teaching may help improve students' problem-solving skills.

Presents the findings of a multiyear study of the effectiveness of reform-oriented science and mathematics teaching (instructional practices for engaging students actively in their own learning and enhancing the development of complex cognitive skills) — specifically, whether such practices are associated with higher student achievement and whether that association is sensitive to the aspects of achievement that are measured. (CD-ROM enclosed.)

In a three-year study, RAND researchers examined the relationship between reform-oriented instruction and student performance in mathematics and science. At the end of the study, students who had been exposed to more reform-oriented teaching performed better in both math and science than those who had experienced less, but the differences in scores were small. The relationship between instructional practice and student performance was stronger when performance was measured on items that required problem-solving skills rather than procedural skills, an outcome that is generally consistent with the goals of reform-oriented teaching. These results illustrate the importance of matching performance measures to reform goals in evaluating instructional innovations. In science, students might be expected to generate scientifi c questions, plan research, collect data, analyze relationships, and write reports. More generally, reform-oriented instruction encourages inquiry-based activities and the skills of intellectual conversation: asking questions, discussing alternative approaches to problems, presenting reasons for answers to questions, and making connections between old and new knowledge and between superfi cially disparate topics. Th e study of reform-oriented instruction described here, called Mosaic II, was an extension of Mosaic I, an earlier investigation in which a team of RAND researchers found “a weak but positive relationship” between reform-oriented instructional practices and students’ scores on standardized tests. Intrigued by these results, they undertook a new study focused on the same question, but designed to yield a more detailed picture of this relationship. Th e new study involved three key methodological features: a longitudinal research design, multiple measures of instructional practices, and multiple measures of student performance. Student performance on standardized tests was measured over a three-year period At the outset of the study, the RAND team selected fi ve cohorts of elementary and middle-school students in three school districts. A cohort consisted of all the students in a selected grade in one of the two subject areas (for example, third-grade mathematics students or sixth-grade science students).1 Each cohort was followed for three years, and their performance on standardized tests was measured at the end of each year (current-year performance), as well as aggregated at the end of the three-year period (cumulative performance). Multiple measures were used to assess instructional practice To obtain data regarding teachers’ choices and classroom activities as they occur, the Mosaic II team used the following methods to assess instructional practices: • Teacher Surveys: Th e surveys gathered information about teachers’ educational background, experience, and classroom practices. • Classroom Vignettes: Teachers were presented with written vignettes, consisting of hypothetical classroom situations, and asked to indicate how likely it was that they would choose each of a list of possible actions that could be taken in response. Th e actions refl ected a range of reform-oriented and non-reform-oriented alternatives. Unlike the survey, which required teachers to recall their actions over a wide range of situations over a long period, this method provided teachers with a realistic context in which to situate their responses, potentially off ering insights into their preferred teaching style and increasing the generalizability of their responses. • Teacher Logs: Each day during a twoto fi ve-day period, teachers were asked to fi ll out a log by checking off items on a list describing specifi c activities that occurred during mathematics or science lessons. Th e logs focused on both teacher and student activities each day during the data-collection period. Th e immediacy of this procedure was intended to increase accuracy in reporting of classroom activities. • Classroom Observations: Th e RAND team observed 52 teachers, recording behaviors on a protocol designed to capture important elements of reform-oriented instruction. Because the results of these observations were independent of teacher reports, they were free of any biases due to forgetting or other reporting limitations. Both multiple-choice and open-ended measures were used to assess student performance on standardized tests Two measures—scores on both multiple-choice and openended items on standardized tests—were used as indicators of student learning in mathematics and science. In addition, performance on items that require problem-solving skills was diff erentiated from performance on items that required only the application of procedural skills.

Urban school systems face immense challenges in trying to meet the goals of the No Child Left Behind Act (NCLB) and in trying to erase the achievement gap between minority and nonminority students. A new RAND study, funded by the Hewlett Foundation, off ers insights and lessons learned from the eff orts being made by three urban districts to improve the instructional quality and performance of their schools. Th ese districts had partnered with an intermediary organization, the University of Pittsburgh’s Institute for Learning (IFL), as part of their reform strategy. Th e study sought to answer four broad questions: • What strategies did districts use to promote instructional improvement? How did these strategies work? • What constrained or enabled district instructional improvement eff orts? • What was the impact of the IFL? What constrained or enabled the district-IFL partnerships? • What are the implications for district instructional improvement and district-intermediary partnerships? Th e study sample consisted of three urban school districts that had made districtwide instructional improvement a high priority, had worked with the IFL for at least three years, and were willing to participate in the study. Th e districts varied in district size, union environment, and state. Data were collected through extensive fi eld interviews and focus groups conducted over a two-year period; RAND-developed surveys of elementary, middle, and high school principals and teachers; district and IFL documents; and demographic and student achievement databases.