June 1, 2004

LIVE@EXPL_final2004.pdf

In the summer of 2000, Education Development Center's Center for Children and Technology (CCT) began a multiyear evaluation of the LIVE @ THE EXPLORATORIUM: Origins project as part of the Exploratorium's three-year funding from the National Science Foundation. Evaluation planning and work began in August of 2000 and continued through December 2003. This report summarizes the evaluation activities, analyzes the findings, and reflects on the challenges and opportunities facing evaluations of this type in subsequent initiatives.

One goal for the evaluation was to produce evidence and analysis of how LIVE @ THE EXPLORATORIUM: Origins impacts visitors at the Exploratorium and those who visit online. The two main audiences for this work are the project staff and the broader community of museums and science centers. One significant objective of the evaluation work explores what tools, assessment processes, and promising evaluation practices might be useful in ongoing webcast and website development for promoting science learning online.

The evaluation work had two strands. The first provided rapid feedback during, and immediately after, each webcast to allow the project team to make modifications and adjustments. The second strand looked at the relative utilization patterns over time and the impact of the various project components

In the course of collecting data for the webcasts produced by the Exploratorium during the Origins project, we learned that it was important to have a stand-alone version of the online survey so that we could provide continuity in the collection and analysis of the responses to the survey. We also found that introducing the survey via a pop-up screen improved the response rate. For those webcast series where the studio audience was surveyed along with the online audience, we found that the visitors to the studio tended to be more spontaneous. This finding, however, was complicated by the fact that the studio survey polled a much more complete subset of studio visitors compared to the subset of respondents to the online survey (less than 1% of the virtual visitors). Because of the Exploratorium's privacy policy, the path and history of individual visitors to the webcasts was not tracked.

Our major findings for all the webcasts were the following:

• The Origins webcasts are a part of the near recent history of science. For example, the DNA webcasts with the pioneers of the Human Genome Project is of historical significance since it recorded the history, viewpoint, and vision of the pioneers of the genetic revolution at a critical juncture of history when the Human Genome Project had been completed by the same individuals who started it. The importance of the Origins project lies not just in the fact that it brought the ideas of science to the public, but rather that it brought the actual scientists doing the science at the moment that they were doing it to the public.




• The nature of the audience changed with the webcast series. Data from the surveys attached to the individual series of webcasts suggest that interest in the specific content of each series of webcasts were a stronger motivation for drawing oline viewers to the webcasts than interest in anything the Exploratorium had to offer. Audiences for the different webcasts differed in terms of the principal way in which they identified themselves, gender, country of origin, and reactions to webcast. The audiences of the webcasts did not differ significantly in terms of age.




• When the Origins website and webcast worked closely together, the online audience gained a stronger understanding. Responses to the Antarctica survey suggested that the audience understood best those topics that were addressed in both the website and the webcast. Conversely where a topic was addressed in only either one form, their understanding was more limited. An alternative explanation, however, is that those visitors with greater interest and science acumen tended to watch the webcasts as well as visit the website.




• By using onsite manipulatives, facilitation, and online media, the Origins staff demonstrated a deep ability to explain science in terms that could be understood by the layperson as well as an ability to design new media representations of science information. In the Speaking of DNA webcast, for example, annotation of Crick & Watson's original paper on DNA made it possible for lay people to read and make some sense out of it. This also enabled online viewers to understand the terminology that the pioneers of the genetic revolution used during the webcast interviews. During the Antarctica webcasts the Exploratorium had to mediate the experience of the webcast because the audience lived in the temperate climate and brought its conceptions of temperate zone life to the Antarctic images.




• Webcasting on location raised the level of science literacy because the Remote Team was able to do more than simply describe what they saw. Since so many of them were also explorers, they were able to compare their own past experiences, whether it was scuba diving, mountain climbing or spelunking in ice caves to the explorations in the various remote locations to give webcast viewers an additional scale by which to judge the uniqueness of the experience. In addition to the enriched personal reaction, the remote team was also able to apply their critical analytical skills to draw out the full implications of what they experienced. Creating a realtime connection to researchers in the field also opened a window into the current thinking of a live scientist or member of a Remote Team and provided a unique sense of place through realtime visual and aural signals. This sense of immediacy contributed to a feeling of actually being at the remote location.




• The webcasts provoked many, many thoughtful reactions from the audience. A small sampling includes: a child asking whether the lasers used to map Antartica were measuring the height of the snow or the land; children asking a series of questions about ice, water and flow during a discussion about the movement of glaciers in the Dry Valleys; a researcher wondering about the genetic explanation of how a free swimming organism shifts to a sessile existence during the Unwinding DNA webcast; a viewer observing that the source of heat in the universe is stars but wondering what the source of cold is in the universe during the Astrobiology webcast.




• The most interactive and engaging webcasts were helped by certain conditions. These included: the personality and facilitation ability of the scientist (being knowledgeable, passionate, but also honest and humble); a sharp, engaged audience (providing interactive connectivity does not guarantee interaction because there has to be someone on the receiving end who engages the speaker -middle-school kids seem to be the perfect age.); moderation by the Origins staff that constantly corrects, explains, and amends what the scientist says so that the audience can understand scientist; promotion of a sense of exploration and discovery so as to engage the interest of the webcast viewer.

In our analysis of the log files for the Origins file we found that:

Over the three year period of the Origins project, there were over 3 million visits to the Origins section of the Exploratorium's website. A majority of the visitors (85.9%) came one time; 7.8% came twice and 2.2% came three times. For the vast majority of visitors (96%, 2,898,575 visits) the duration of their visit was 5 minutes or less. Only 4% of the visitors (118,601 visits) visited the Origins website for more than 5 minutes.

For the webcasts, the Hubble series were the most popular with 38,085 visits to the Hubble webcast page. The CERN webcasts were the second most popular with 12,705 visits. The Belize- London webcasts were the third most popular with 11,049 visits. The Antarctica webcasts were the fourth most popular with 9,649 visits. The country with the most active visitors was the United States with 549,053 visits (87.3%). The United Kingdom had the second most active visitors with 13,271 visits (2.1%). The top referring site was the Exploratorium's own main home page with 2,371, 283 visits (78.6%) of the total 3,017,176 visits recorded in the log file. 410,956 searches led to the Origins home pages. This represented 13.6% of the referrals to Origins.

We found some issues to be specific to the Origins project. They are the following:

• The Exploratorium's web audience is drastically different from its studio audience. The live studio audience included many families with young children. The web audience, on the other hand, seemed to be primarily male, older, with a large (30%-45) international component. However, the Web Trends analysis of the log files suggests those viewers who actually watched webcasts long enough to fill out a survey about them were in the top 4% of visitors in terms of the duration of their visit. Thus, the difference in character between the physical and the online visitors may simply reflect the fact that the onsite surveys and the webcast cameras captured a broader segment of the studio audience than the online surveys were able to.




• In order to acclimate the studio audience to concepts and terminology that was to be presented during the Web casts, The Origins staff developed preparatory sessions for the Hubble webcasts in Year One. These preparatory sessions included activities that engaged the audience through direct perception or physical activity that provided concrete experience from which they could extrapolate to understand the concepts that would be detailed during the webcast.

Our recommendations were:

As a result of this study, we recommended the following for those institutions that decide to use webcasting as a way to reach a broad clientele beyond the confines of their local geographic region:

• A privacy policy should be dynamic, comprehensive, and flexible while reflecting the overall needs and philosophy of the organization, technical changes in tracking methods, and the needs of individual departments to secure rapid and widespread feedback on site design. The privacy policy should be periodically reviewed to balance what may be opposing needs of preserving privacy and ensuring broadly-based timely evaluation.




• Utilize the unique strengths of webcasting as a medium of communication. These include the following: ability to be intensely interactive; ability to create a window into the current thinking of onsite researchers; ability to create a sense of place; ability to depict motion that reflects the behavior of local life.




• Build an online community of regulars that can be used to: inform museum patrons about upcoming webcasts, solicit opinions about project ideas, mockups, etc.; offer a higher level of membership that offers a greater inside look of the museum's work; encourage creative work around science to be shared with other Exploratorium patrons; provide the 'critical mass' needed to establish a robust level of interaction for online forums.




• Prepare for the future of webcasting by redesigning it so that the greater capacity for interactive interaction implicit in the growing penetration of broadband access can be tapped. Since broadband users are more likely to be creators and managers of online content than narrowband users, more likely to search for information online, and more likely to engage in multiple Internet activities on a daily basis, the format in which the webcast occurs should be even more interactive by creating a digital stage on which the broadband user can multitask - watching the webcast at the same time that he/she views high resolution images, sends questions via email, etc.




• Design the interactive content of the site in such a way that visitors can produce written or verbal artifacts (as in postings or questions or comments) or make decisions that are recorded (polls, games, branching, etc.) that are designed to reveal indicators of some interests or understandings of the participants.




• Embed the webcasts in surrouding material that contextualizes the webcasts and helps make them useful over a much longer term.




• Build a network of affiliated places (school classrooms, after school centers, adult education programs, senior centers, etc.) in which the institution can run real or virtual focus groups that can provide detailed feedback that can illuminate understandings and challenges to aid formative evaluation.