September 1, 2008

eMINTS rpt08_final_short.pdf

The eMINTS professional development programs are designed to help teachers learn how to integrate technology into their teaching, using instructional strategies that promote inquiry‐based learning and encourage collaboration and community building amongstudents and teachers. The eMINTS programs considered in this evaluation include: (1)eMINTS Comprehensive Professional Development (Comp PD), a two‐year programconsisting of approximately 250 hours of teacher professional development and support, for teachers in school‐designated grades, including 10-‐12 Classroom Visits each year; (2) eMINTS4ALL, a two‐year, 90‐hour professional development program built for teachers in the grades above and below eMINTS Comp PD teachers, to support students for multiple years, including 8-9 Classroom Visits per year; and (3) Professional Development for Educational Technology Specialists (PD4ETS), a two‐year program that prepares local district staff members to become eMINTS instructional specialists.

Since 1999 the eMINTS National Center has engaged external evaluators to examine the program to understand its impact on schools, teachers, and students. In the 2005-2006 school year, the eMINTS program added new professional development programs and expanded beyond the state of Missouri, where it began (as "enhancing Missouri's Instructional Networked Teaching Strategies"). In order to help eMINTS ensure a high level of program quality as it scaled up, the Education Development Center, Inc.'s Center for Children and Technology (EDC/CCT) was brought in as the external evaluator in 2006 and charged with designing an evaluation that would create instruments and procedures to assess Program Fidelity and Teacher Mastery of eMINTS core concepts, and would examine the relationships among Program Fidelity, Teacher Mastery, and student achievement.

Program Fidelity consists of two components:

1. PD Fidelity, or how well the professional development addressed the key conceptual constructs of eMINTS

1. Classroom Visits, or how much time instructional specialists spent on certain activities during their regular visits to participant classrooms.

EDC/CCT's evaluation strategy was designed to answer the following questions about the eMINTS Comprehensive and eMINTS4ALL professional development programs: Regarding Program Fidelity:

• Are the current eMINTS professional development sessions faithfully addressing the core program constructs (PD Fidelity)?

• Are there differences in the level of PD Fidelity between professional development sessions delivered by eMINTS staff and those delivered by participants or graduates of the PD4ETS program

• What activities are specialists spending the most time on during their Classroom Visits?

• Are there differences in the amount of time eMINTS staff and participants or graduates of the PD4ETS program spend on different activities during Classroom Visits?

Regarding Teacher Mastery:

• What are the baseline levels of Teacher Mastery of eMINTS concepts?

• Are eMINTS participants mastering some concepts more successfully than others?

Regarding Program Impact:

• Does the level of eMINTS Program Fidelity have an impact on participating teachers' mastery of the concepts presented in the professional development sessions?

• Does the level of eMINTS Program Fidelity have an impact on the achievement of students in the classrooms of eMINTS teachers?

• Does teachers' level of mastery of the program concepts have an impact on the achievement of student in their classrooms?

EDC/CCT evaluators worked closely with eMINTS program staff to design Fidelity and Teacher Mastery instruments that were closely aligned with the core concepts about effective instruction and technology integration that serve as the foundation for all eMINTS programming. Data collected for the evaluation included:

• Observational data from eMINTS professional development sessions

• Records of Classroom Visits by eMINTS instructional specialists

• Teacher artifacts (Lesson Plans, WebQuests, and Classroom Websites) submitted as part of eMINTS teachers' portfolios

• Data from interviews conducted with 16 teachers one year after they completed the eMINTS program

• Student assessment data from the Missouri Assessment Program (MAP) tests in Mathematics (MA), Communication Arts (CA), Science (SC), and Social Studies (SS)

KEY FINDINGS FROM THIS EVALUATION OF EMINTS

Regarding Program Fidelity, the findings suggest that there was a high level of fidelity to the core eMINTS concepts:

• The majority of instructional specialists addressed many of the key concepts.

• The majority of instructional specialists used the recommended instructional practices.

• There were few differences in PD Fidelity between eMINTS staff and participants or graduates of the PD4ETS program.

• During Classroom Visits, instructional specialists spent the most time working with teachers on Lesson Planning and Modeling Instruction and the least amount of time providing Technology Assistance and Problem Solving.

• eMINTS staff spent more time than participants or graduates of the PD4ETS program on Lesson Planning during Classroom Visits.

Regarding Teacher Mastery, the findings indicate a wide range in the levels of mastery of the eMINTS concepts, with certain concepts more successfully mastered than others:

• Teachers displayed a high level of mastery of some core eMINTS concepts, such as integrating technology to support student learning and having students create authentic products to demonstrate their learning.

• Teachers displayed a lower level of mastery of other core eMINTS concepts, such as designing instruction to address diversity and having students generate their own questions to guide their inquiry.

Regarding Program Impact, our findings suggest that higher PD Fidelity is associated with greater Teacher Mastery of eMINTS concepts, and more time spent Lesson Planning in Classroom Visits is associated with greater Teacher Mastery as reflected in the Lesson Plans:

• There was a significant, positive correlation between PD Fidelity and Teacher Mastery scores on the Lesson Plans teachers submitted in their portfolios.

• There was a positive trend between PD Fidelity and Teacher Mastery on the WebQuests teachers submitted in their portfolios.

• There was a significant, positive correlation between the amount of time teachers spent on Lesson Planning during Classroom Visits and the scores on the Lesson Plans they submitted as part of their portfolios.

Also regarding Program Impact, our findings suggest that higher levels of Teacher Mastery of eMINTS concepts are associated with greater student achievement, higher levels of PD Fidelity are associated with greater student achievement, and more time spent on Lesson Planning during Classroom Visits is associated with greater student achievement:

• There were significant, positive correlations between student MAP scores and Teacher Mastery on the Lesson Plan (in grades 3, 4, and 7), on the WebQuest (in grades 3 and 7), and on the Classroom Website (in grades 4, 5, and 7).

• There were significant, positive correlations between PD Fidelity and student MAP scores in grades 3, 4, 5, and 8.

• There were significant, positive correlations between student MAP scores and Lesson Planning during Classroom Visits in grades 4, 5, and 8.

Overall, this evaluation provides evidence that the eMINTS program is being implemented with a high level of fidelity by both the eMINTS staff and participants and the graduates of the PD4ETS program; that teachers are mastering some, but not all, of the core eMINTS concepts; and that maintaining a high level of Program Fidelity is important for ensuring that teachers are mastering the core program concepts, which may then result in higher levels of student achievement.