January 1, 2005

IntelLeadForum04.pdf

Consistent with the longitudinal evaluation of the Intel Teach to the Future Essentials Course that Education Development Center's Center for Children and Technology (CCT) has been conducting since the program's inception in 2000, the Intel Foundation commissioned CCT to conduct a formative evaluation of the Leadership Forum Pilot program. This report presents findings and recommendations from this formative research.

Methods
Data collection for this formative research was conducted using mixed methodologies:

• Survey. EDC developed and advised Intel in the deployment of an online survey that all Leadership Forum attendees were asked to complete at the end of each training session.
• Observations. An EDC evaluator attended and observed six Leadership Forums from September through November 2004.

Key Findings
Below is a list of key findings from the formative evaluation of the Pilot.

• From September 27 to November 19, 2004, Intel Innovation in Education held 18 Leadership Forums, training 345 administrators, 267 of whom responded to an online survey (77% response rate).
• Twenty-six percent of survey respondents were district-level administrators (superintendent, curriculum director, etc.) and 56% were school-level administrators (principal, assistant. principal, tech coordinator, etc.). There was very little difference in the survey responses submitted by administrators working at the district and state levels, suggesting the Forums reached both groups effectively.
• Nearly all survey respondents (95%) reported the ideas and skills they learned from the Forum either will probably (54%) or will definitely (41%) help them improve teacher effectiveness and student achievement by supporting and promoting the integration of technology.
• Almost all of the survey respondents (95%) reported the Forum helped them either to a moderate extent (43%) or to a great extent (51%) create a prioritized list of leadership behaviors that impact the integration of technology as a tool to improve student learning. Equally high were the number of respondents who reported the Forum did the following: examined the critical role educational leaders play in the effective integration of technology into teaching and learning; presented an analysis of the ISTE NETS-A standards and performance indicators; helped me to develop a personalized action plan ready for implementation in my school or district; illustrated concrete strategies that were new to me for improving student achievement through the integration of technology into the classroom.
• A solid majority of survey respondents found all five of the core modules of the Leadership Forums - review of available resources, creation of personal action plan, discussion of case studies and leadership behaviors, discussion of best practices and exploration of NETS-A standards - either moderately or very useful.
• Three-quarters of respondents reported feeling either well prepared (16%) or adequately prepared (59%) to act on the steps outlined in their Action Plans.
• The Leadership Forums helped strengthen administrators' perceptions of and interest in Intel Teach to the Future, specifically: 1) 70% of survey respondents were from schools/districts participating in Intel Teach to the Future; 2) the Leadership Forums helped strengthen administrators' perceptions of and interest in Intel Teach to the Future; 3) many participants said they were interested in providing additional training for themselves and other administrators; and 4) most of the RTAs and trainers interviewed reported they had little or no followup planned after the Forums' completion.
• The Leadership Forums encountered several implementation challenges: 1) core technologies - both Internet connectivity and Turning Point - were sometimes unreliable; 2) the curriculum script was too restrained and the format was not interactive enough for some administrators; and 3) a small minority of administrators mistakenly expected the Forum to focus on concrete skills and tools for them rather than focusing on technology integration at the classroom level.

Recommendations
The report contains recommendations based on the findings represented in Sections I and II.

Continue to fill the void of administrator technology training by expanding the program. Administrator responses to this program confirm the program team's expectation that little professional development regarding instructional uses of technology is yet available for administrators. Consequently, administrators were very receptive to this program, but also came either with very narrowly defined goals and expectations (wanting technical training) or with so many needs it was difficult for them to focus in on the specific goal-setting activities presented in the current curriculum. Although Intel is unlikely to choose to respond to administrators' interest in skills training, there are many opportunities to expand this training to provide further support to administrators who are just beginning to learn how to be effective leaders in this area. Possibilities include the following:

• Partnering with administrator certificate/degree programs offered by schools of education. For instance, one of the Forums was embedded in a course administrators were taking at a state university as part of a longer training program.
• Offering a series of Forums that focus on strategies for implementing Action Plans and opportunities for administrators to expand and deepen their goals. Parts of the series - discussion of the various Action Plan implementation steps, for instance - could occur online after administrators had the initial opportunity to train face-to-face.
• Building an administrator component into teacher training thereby facilitating administrator-teacher communication around technology integration. For instance, the future iteration of Teach to the Future could include a Leadership Forum segment, requiring teachers to collaborate with principals and other instructional leaders within their schools.

Continue to use Leadership Forums to strengthen the positive perception of and participation in Intel Teach to the Future. While all participants are introduced to Intel's teacher training program during the course of the Forums and receive a copy of the "Intel Teach to the Future: Meeting the Challenge of No Child Left Behind" booklet, they could absorb more detailed information about how teachers in their schools and districts can pursue additional training opportunities. Past evaluation of the Essentials Course has indicated that Intel Teach to the Future often complements existing, more basic professional development offerings within school districts. The Leadership Forums could include an activity in which administrators conduct an initial needs assessment regarding their teachers' professional development related to educational technology, and consider where the Essentials Course might fit within their overall sequence of trainings. More concretely, administrators could be given hard copies of information about how to become involved in Intel Teach to the Future, and contact information for other districts in their area who have already been involved in the program and could provide insight into how to make the best use of it.

Give participants greater opportunity to exchange ideas with one another by building in additional paired and group activities. Many participants expressed a desire to hear from their colleagues about technology integration within their local settings and to use the Forum as an opportunity to develop concrete ways to work together within their schools and districts. In the Essentials Course, the extended training period gave teachers time to build these relationships and develop plans for future collaboration. This kind of peer learning and collegial support are consistent with the spirit of all of Intel's professional development programs, and should be emphasized in the Leadership Forums as well. However, their brief timeframe means these kinds of activities must be structured into the program itself, because there is little or no time for informal discussion or follow-up conversations among participants. The Leadership Forums should emphasize the importance of learning from one another's experiences and the value of future collaboration among participants by building time for structured, focused planning around these topics into the Forum agenda.

Reduce the level of "scripting" in the training experience by encouraging facilitators to familiarize themselves with the substance of the curriculum rather than relying on specific language. As the program expands and the need to recruit trainers grows it will be important to communicate clearly what is expected of each trainer and to model these expectations in the training they receive. This will be true of both newly recruited trainers as well as administrators who initially attend a Forum and then have an interest in training fellow administrators once they return to their schools and districts. Although requiring all trainers to read a curricular script ensures the delivery of the same content in every Forum, it does not give the trainer the flexibility to adapt to local needs and interests, potentially making the training less valuable to participants. Therefore, future training and support materials, such an online resource constructed for trainers in the field, may include a simple description of key points worth emphasizing in each curriculum module. Another useful resource would be a list of "tried-and-true" adaptations that trainers have made depending on various local conditions and particular kinds of participant groups.

Conduct detailed technical run-throughs prior to each training session. Trainers as well as an on-site technical support staff will minimize the number of technical delays and glitches by doing the following: running Turning Point software checks, conducting software inventories, including Flash, preloading relevant Internet sites, and testing the connection between the laptop and projector. Ideally, the same tech support person, who is familiar with the facility and other demands that are likely to occur on infrastructure at the time of the training, will remain present during the training to troubleshoot any difficulties that arise.

Use the next phase of evaluation to identify concrete ways participants use their training, tracking the implementation of their Action Plans and how they support technology integration at the classroom level based on their Forum experience. Nearly all of the participants interviewed expressed a strong desire to implement their Action Plans but relatively few (16%) described themselves as being "well prepared" to do so. Tracking if and how administrators take action to accomplish these goals is likely to yield valuable information about the local and systemic challenges administrators encounter, specific knowledge or information they may need in order to act effectively, or different decision-making or policysetting practices that prove to be more or less successful. Learning more about all of these topics will inform further development of the training experience and help to define the scope and depth of impact the program can expect to have.

Revise the online survey - or administer two versions - to capture a broader range of responses. Just as the Forum curriculum will be updated, so too must the survey to reflect the changes and newly refined goals. While this survey provided useful feedback on participants' responses to the Forum, responses were highly consistent across questions and across different groups of respondents. It is appropriate to develop a more focused survey at this point that would allow for more discrimination among participants' different experiences with the program.