April 1, 2006

Data_AERA06.pdf

One of the hallmarks of the No Child Left Behind Act (NCLB, 2001) is the requirement that states develop annual assessments to measure school and student progress and that educators use data to help improve the learning of all students. As a result, the administrators and teachers are being confronted with complex and diverse sources of data from which they must make informed instructional decisions. Increasingly school districts are turning toward technology-based solutions that they believe will help them to use data more effectively and there are a growing number of technology-based products that enable districts to provide data to many levels of the system - the teachers, administrators, parents, and policy makers - as a means to improve instruction, student learning, and communication.

Examining how technology-based tools can facilitate decision making, and how administrators and teachers use such tools and data to enhance instruction is therefore essential if we are to understand how assessment data can be used effectively to inform educational decision making. The Creating an Evaluation Framework project brings together complimentary evaluation techniques, using systems thinking as the primary theoretical and methodological perspective, to examine the implementation and use of data-driven applications in school settings. The project described in this paper has two goals: (a) to build a knowledge base about how schools use data and technology tools to make informed decisions about instruction and assessment; and (b) to develop an evaluation framework to examine the complexities of dynamic phenomena that will inform the field and serve as a knowledge building enterprise (Mandinach, 2005; Mandinach & Cline, 1994).

The purpose of this work, and central focus of this paper, is to examine technology-based, data-driven instructional decision making tools, their implementation, and impact on different levels of school systems (i.e., administrative and classroom). Examining different tools in diverse settings enables us to develop and validate an evaluation framework that will be sensitive to the dynamic and interacting factors that influence the structure and functioning of schools as complex systems. The framework includes: (a) the use of a systems perspective; (b) examining the system with multiple methodologies at multiple levels; and (c) recognizing its complex nature, and the need for the technology tools to become instantiated so that both formative and summative methods can be used. The research not only examines a methodological framework using systems thinking, but also presents a theoretical framework on how data-driven decision making occurs in school settings, and a structural framework that outlines the functionality of the tools that either facilitate or impede data-driven decision making.