January 1, 2005

LinkingData_GrowStudy.pdf

During the last decade and in the wake of the No Child Left Behind legislation, standards, assessments, and accountability have emerged as three prongs of a national education reform movement that has asked district and school administrators to think very differently about educational decision-making and the use of data. However, research about data-driven decision-making is limited. With funding from the Carnegie Corporation, in the spring of 2002, EDC's Center for Children and Technology began a two-year exploratory study that examined how educators and administrators within the New York City public school system are using data-made available to them through the print and web-based reporting system of the Grow Network- -- to inform decisions about teaching and learning and about educational practices.

As a tool, the Grow Report® brings what are often disparate parts of the educational system into conversation and alignment with each other, reinforcing the notion that standards are meant to inform instructional practices not just serve as a means to accountability while also identifying the standards the tests are highlighting (because every standard cannot be tested). By design, it provides a format that builds a bridge between standards, testing results, and instructional strategies and appears to be highly successful in creating a navigational framework for educators.

Using a mix of qualitative and quantitative methodologies, the two-year study unfolded in three phases. Phase One focused on understanding the ways in which central office personnel, along with district superintendents and their education teams, thought about using data to inform decision- making. Phase Two emphasized ethnographic research in 15 schools across four school districts in New York City that represented various neighborhoods, student populations, and overall performance levels. Phase Three involved the development and administration of two separate surveys across the New York City public school system that asked teachers and administrators about how they interpret data and conceptualize the use of the Grow Reports® for instructional planning.

Teachers' Attitudes Toward Data, Decision-making and the Grow Reports
What we have learned from this study is that teachers, in particular, are open to data, but also scrupulous about its use. Rather than accept a narrowing interpretation of their students' strengths and weaknesses based on a single test, they instead rely on multiple sources of data - impressionistic, anecdotal, and experiential - accrued over the long term and based on many experiences with their students to make most instructional decisions. Yet having been introduced to the possibilities of using systematic data to make instructional decisions, teachers are eager for more and better data. The Grow Report® is an important first step in this process and shows the promise of making the transition to data-driven decision-making in our schools.

No matter how teachers viewed state-mandated, standardized testing, whether skeptically or acceptingly, they recognized, across the board, that part of their job is to prepare students to take the test. But they have questions about the test. In interviews and survey responses, they expressed concern about the accountability environment's impact on instruction, with three quarters of teachers feeling that the tests lead them to teach in ways that contradict their own ideas of good teaching. The majority of teachers also questioned the test's accuracy in measuring students' academic abilities and in measuring the "life skills" that students need to succeed as well as the test's culturally sensitivity or developmentally appropriateness to all students.

Based on these concerns, teachers frequently seek to monitor student learning and triangulate assessment data in a variety of different ways. All of the teachers surveyed reported using multiple assessment strategies, either always, often, or sometimes. When interviewed, the teachers discussed mixing various assessment strategies to provide a fuller picture of student understanding and learning. Overall, the teachers felt that teacher-made assessments and strategies were more useful than the external assessments. Only a handful of teacher said that they rely solely on formal assessments.

Throughout the survey and in the interviews, teachers across New York City reported using the Grow Reports® in various ways to meet their overall classes' as well as their diverse students' academic needs. Grow-using teachers discussed making decisions within several specific areas of their instructional practice: (1) targeting instruction, with decisions about class priorities, lesson plans, and the academic year; (2) meeting the needs of diverse learners, seen in strategies such as grouping, creating Individualized Education Plans (IEPs), and giving individualized assignments and materials appropriate to the students' levels; (3) supporting conversations with parents, students, fellow teachers and administrators about students' learning; (4) shaping teachers' professional development by reflecting on their own practice; and , (5) encouraging self-directed learning by giving the data to students.

Teachers noted that the Grow Reports® provided them with more information about students than what they had access to previously, both as a class and as individuals. When asked how they use the Grow Reports®, several teachers responded that they use the data when deciding where to target their instruction from creating a more extensive yearlong pacing calendar to planning mini-lessons that review and reinforce certain concepts or skills. Most teachers also felt that having timely data that showed individual student performances helped support their decisions to differentiate instruction according to individual students' needs or to group students, based on how they performed on specific skills to target certain skills. Despite the high-stakes climate in which they teach, Grow-using teachers felt less forced to exclude anything not on the test from their teaching and estimated that they spent less time on explicit test preparation activities than was true for a national sample of teachers. Compared to the national sample, these New York City teachers can be considered "data-friendly." In other words, the skepticism many expressed about the value of standardized test data in educational decision-making was often more about its incompleteness and the stakes involved, than a rejection of the whole idea of using standardized tests to measure individual achievement. Many teachers were careful to note that the Grow data was one "small piece" in a wide array of assessments they use, including observations, in-class assignments, daily quizzes, unit pre- and post-tests.

Moreover, teachers added that the data on the Grow Reports® has "declining value" as the year progresses, in part because students take the exam roughly six months before the reports are distributed.

Administrators' Attitudes Toward Data, Decision-making and the Grow Reports Administrators' attitudes about high-stakes testing are not markedly different from teachers. Administrators clearly feel pressure to improve test scores; they discussed how they support teachers' testing needs and how the growing accountability culture has influenced their schools and their own decisions. With its adoption becoming more widespread through NCLB, standardized testing is not only being used to directly measure students' academic progress but also to indirectly assess administrators' leadership.

However, administrators, like teachers, have some reservations about what the tests are measuring. Administrators questioned the validity and reliability of the test in their responses to our survey; the majority of administrators do not consider the test as accurate as a teachers' judgment of what students know and can do. Administrators were divided over most other issues related to statemandated tests, such as whether or not the state-mandated test were an accurate measure of what students know, or whether test pressure narrowed the scope of the curriculum. Despite this ambivalence about the test's effect on curriculum scope and teaching practice, administrators did feel that the state-mandated test is aligned to what teachers teach in the classroom. And though many expressed a desire to assess student progress from different angles in an ongoing fashion, a vast majority of the administrators surveyed reported that, in order to prepare students for the test, they encourage teachers to "teach the students test-taking skills."

New York City district- and building-level administrators reported using the Grow Reports® to gain a greater understanding of the educational and instructional concerns particular to their level of the education system. Administrators explained that the Grow Reports® helped them to identify class-, grade-, and school-wide strengths and weaknesses that could then be used to make decisions about planning, shaping professional development activities, and determining student performance and demographics. In interviews, many of the administrators spoke about how the Grow Reports® helped frame conversations they had with teachers, parents, or other administrators related to student learning, professional development for teachers, or addressing school or district challenges. For example, since the Grow Reports® and associated instructional resources are approved by the New York City Department of Education and aligned to state standards, administrators found that the reports were often a good fit for shaping professional development activities.

Technology's Role in Balancing Tension of Practice and Policy While policymakers have embraced the notion that a single assessment can measure students, educators in this study acknowledge that high-stakes testing communicates only a piece of what they need to know about the complex repertoire of skills and talents that children need to succeed. Herein lies the gap between what national policymakers and local practitioners see as important. Digital technology has already played an important role in bridging this gap by giving teachers access to high-stakes test data. We believe that digital technologies will expand on this role by helping teachers engage in data-gathering and data-analysis processes inclusive of the multifaceted ways in which children show evidence of learning, allowing teachers and administrators to track performance data, observational data, informal conversations, portfolios of student work, self-assessment and reflection, the stuff of daily instructional decision-making in the classroom. The inclusion of these organized, diagnostic, and authentic, performance-based data in the decision- making process would also go a long way toward refining and supporting the practices that educators routinely engage in.

The Grow Report#174; represents an important initial step on this path. By creating a lens through which the relationship between standards, assessments, and instruction can be explored, it helps educators to find reason in and navigate the tensions that prevail in the high-stakes contexts in which they work.