ITEST Annotated Report Excerpts

Excerpt 1 [Urban Ecology, Information Technology, and Inquiry Science for Students and Teachers, Boston College]

Summarizes goals, results, conclusions

Urban Ecology, Information Technology, and Inquiry Science for Students and Teachers is a project of the Urban Ecology Institute (UEI) at Boston College (BC). Under a previous NSF grant, UEI developed an Urban Ecology Field Studies program for high school teachers, counselors, and students. The current project is a three-year effort to integrate information technology (IT) and career education content into the existing program.

As of September 2007, the project has completed Year 2 of the grant. The project goals for Year 2 included the development of technology-based, urban ecology modules, teacher professional development, and testing the modules in local classrooms. As evaluators for the ITEST project, EDC is providing summative evaluation of the teacher professional development sessions, and the teachers’ use of the modules in the classroom. To this end, EDC developed, administered, and analyzed four evaluation instruments. For evaluation of teacher professional development, a pre-post teacher survey, and a focus group were used. For classroom implementation, a classroom implementation survey and a classroom observation protocol were used.

For the teacher professional development, EDC's evaluation goal is to determine: What effect do the project's professional development strategies have on the skills and content knowledge of participating teachers specific to conducting information technology-enhanced field studies? To address this goal, EDC administered a pre-post teacher survey before and after the summer institute, and conducted focus group evaluations after the summer institute.

In Year 1, EDC had developed and administered a pre-post teacher survey at the 2006 summer institute, and in Year 2, EDC analyzed and reported on the data from that survey to the BC team in November 2007. The findings were encouraging but for some items on the survey there was a great deal of variability. Therefore, for Year 2 the survey was revised – creating, modifying, and deleting items for some scales in order to achieve greater reliability. The survey was tested for reliability with a pilot group of 79 undergraduate education students and adjusted based on the results from Chronbach's alpha and factor analyses. The instrument was sufficiently reliable to be administered during the Year 2 summer institute training in July and August, 2007. The findings of the pre-post teacher survey are summarized as follows (survey included in Appendix A):

• Participants reported statistically significant levels of skill improvement in their skill with classroom uses of IT (teaching students to use IT; helping students to use IT in class as part of a lesson; and designing lessons that make use of IT to teach science) and their use of software tools specific to the UEI summer institute (bioacoustics and GIS software). Their general levels of IT use and their skill levels with PowerPoint and Excel did not increase significantly.
• Participants demonstrated improvement in their ability to define the term “urban ecology” with more complexity, recognizing physical, biological and human components to urban ecology; but remained consistent in describing the primary benefit to society of studying urban ecology as helping solve urban problems and improve urban planning.
• Participants also remained unchanged in their ideas about the potential benefits to students of participating in field-based studies, factors to consider in choosing a site of study, and safety precautions that should be followed in the field.
• Participants did report statically significant change over the course of the summer institute in their level of knowledge about how to guide students into STEM careers; self-efficacy in teaching science field investigations; belief in the usefulness of IT to engage students with scientific content; confidence about teaching students how to formulate explanations, models, and arguments; and confidence about teaching students how to design and conduct scientific investigations.
• The only construct for which the change was not statistically significant was educators' perception of the importance of their own role in providing STEM career information to students, which was already high at the pretest.
• Participants did report statically significant change over the course of the summer institute in their level of knowledge about how to guide students into STEM careers; self-efficacy in teaching science field investigations; belief in the usefulness of IT to engage students with scientific content; confidence about teaching students how to formulate explanations, models, and arguments; and confidence about teaching students how to design and conduct scientific investigations.
• The only construct for which the change was not statistically significant was educators' perception of the importance of their own role in providing STEM career information to students, which was already high at the pretest.

Also for the Year 2 summer institute, EDC conducted teacher focus group evaluations of the summer institute, with the goal of evaluating the teachers' perceived level of usefulness of the professional development activities, what could be improved, and what support they needed in order to implement the UEI curriculum during the school year. The findings from the focus groups are summarized as follows (protocol included in Appendix B):

• Most participants were satisfied with the institute and left feeling competent to implement the UEI curriculum.
• Repeated hands-on practice, urban ecology content, and tangible resources were considered the most helpful aspects of this professional development offering.
• Some teachers found working with students helpful, e.g., for practicing field supervision or anticipating problems with the content or software. The bioacoustics project was burdened by specific disruptive students, and suggested better student applicant screening next year. The ecoscenario group, on the other hand, remarked that they had a particularly cooperative group of students this year.
• A common suggestion was that administrative procedures and communication surrounding the institute could be improved.
• The tree inventory group suggested pre-assessing student participants so that teachers could anticipate their ability levels and work with them accordingly.
• The ecoscenario group felt that the environmental justice content had been taught in too cursory a way and deserved its own strand.

To evaluate the classroom use of the modules, EDC's evaluation goal is to determine: What effect do the project's professional development strategies have on participating teachers' use of the field-based teaching modules and the associated IT elements of those modules in their classrooms? To address this goal, EDC distributed an implementation survey and conducted classroom observations.

The implementation survey that EDC developed asks teachers about their use of the modules, including time spent, how much was covered, student engagement, teacher comfort with the technology, how well things worked, any barriers to use, and any adaptations made to the module. The survey was sent to all participations, about fifteen teachers, and responses were received from eight teachers. Overall, the survey results indicate that teachers felt positive about their experiences with both of the Urban Ecology modules. Teacher responses to the implementation survey (included in Appendix C) are summarized as follows:

• The modules were at about the right difficulty level for their students.
• Students found the modules to be quite engaging.
• The modules worked with few problems.
• Teachers felt reasonably comfortable with the technology.
• The most commonly reported problems related to lack of time due to scheduling issues and the end of the school year.

For classroom observations, EDC developed a classroom observation protocol that focused on student data analysis and presentation tasks – the protocol looks at how teachers frame and reference the technology throughout the lesson; teacher scaffolding of the technology; student engagement in the lesson; the nature of the questions students are asking; the level of use and kind of evidence used in presentation; and the presence of career components in the lessons taught. Teachers were interviewed as to their prior use of the materials, how well they thought the class went that day, and how typical was the day's class. Eleven classroom observations were conducted during May-June, 2007 - four classrooms, with two to three observations each. Two of the four classrooms were using the bird bioacoustics module, and two were using the tree inventory module. The four classrooms presented a range of experiences with the materials, but teachers were generally pleased with the results. Some common findings of the classroom observations (observation protocol included in Appendix D) were as follows:

• Time pressure and the end of the year was an issue for all four classrooms, and teachers found themselves having to rush through or skip some of the activities.
• Teachers need time to become proficient in both the technology and the content. Having project members available for in-class support, as noted above, was very valuable to all four teachers.
• Access to enough computers was very important. The classrooms with one laptop for every two students were the most effective at having students accomplish the data analysis and presentation tasks.
• Technology makes some tasks easier, but care needs to be taken that the tasks not become rote. Students entering in data may be learning important urban ecology topics, or they may just be following step-by-step instructions.
• Content learning was sometimes disconnected from technology process. Teachers need the time to bring content and process together and check for student understanding.

The report to follow describes in detail the development, administration, and data analysis of the four evaluation instruments for Year 2: the pre-post teacher survey, the focus group, the implementation survey, and the classroom observations. Then we look for themes across the data to present conclusions.

Excerpt 2 [EcoScienceWorks (ESW), Foundation for Blood Research]

Provides overview related to project goals.

Overview

During this past year, the EcoScienceWorks (ESW) project has focused on fulfilling some of our goals and advancing others. Our partners at SimBiotic Software produced the first phase of the software content around which our 23 teachers developed a draft curriculum. The teachers piloted this curriculum with our summer students and field-tested it in their classrooms. Further content development by SimBiotic Software, MIT Teacher Education Program and two graduate students enrolled in University of Southern Maine’s GIS program will provide new elements for our field-tested curriculum. At the time of this report, our evaluator is assessing teacher satisfaction with the curriculum and self-perception of technological and pedagogical change over the first year of the project through phone interviews. Currently, we are in the midst of planning the summer sessions for year 2. Only one teacher has withdrawn from the project because she was assigned to a different grade level for which the content is inappropriate, and she felt she would not be able to field-test the curriculum.

Excerpt 3 [SRI Build IT]

Executive Summary

ALL STARS is an after-school program for middle school girls in the Northern California Bay Area. Developed and coordinated by Girls Incorporated of Alameda County, ALL STARS serves a predominately black and Latina population of young girls in the Oakland Unified and San Leandro Unified School Districts by providing academic enrichment, homework assistance, social support, and health and fitness development to participants. During the 2005-06 academic school year, Build IT was added as the technology component to the ALL STARS after-school program at two of the sites: EXPLORE and Muir middle schools. Through a grant from the National Science Foundation and working with staff at Girls Inc., a team of researchers at SRI have been in the process of developing and implementing the Build IT curriculum during the past year. During the first year of its development SRI and ALL STARS staff has implemented Unit 1 and Unit 2 of the curriculum in the fall and spring semesters, and Unit 3 during the summer session. The curricula engage students in hands-on activities that encourage understanding of the design process, technology underlying the Internet, HTML, and other information technology components. The goals of Build IT are to motivate middle school girls to use technology and to strengthen and build their technology fluency; to increase middle school girls' interest in and desire to take high school algebra and geometry courses in preparation for postsecondary STEM [Science, Technology, Engineering and Mathematics] education and/or IT careers, and to encourage middle school girls to pursue IT careers. Evaluators at Hatchuel Tabernik & Associates were contracted by SRI to perform a summative evaluation of the Build IT program. In order to assess the effectiveness of Build IT, the evaluation utilized pre-post survey information from Build IT participants and a comparison cohort of girls who did not participate in the program. In addition, interviews were conducted with ALL STARS staff in order to assess the strengths, successes, challenges and opportunities of incorporating the Build IT curriculum into the ALL STARS program. In this report we present the findings of thisYear 1 evaluation. The evaluation seeks to answer four central research questions:

• Are Build IT participants more likely to become interested in IT careers and make plans to take courses in high school to help prepare them for those careers?
• Do Build IT participants become more knowledgeable about IT careers and learn basic IT concepts?
• Do Build IT participants develop more contemporary IT skills and intellectual capabilities for IT?
• Does the ALL STARS program grow in its capacity to offer curriculum focused on developing IT fluency over the course of the project?

The evaluation findings highlight the following:

• Declines in negative sentiments toward mathematics and computer courses. A number of Build IT participants entered the program believing that they would not take computer or upper-level mathematics courses in high school. By the end of the school year, however, participants exhibited substantial decline in these negative sentiments. Much of this movement was in the direction of uncertainty (e.g. 'I don’t know'), which suggests that, although positive, the Build IT developers may want to add more of a focus on course planning and boosting perceived mathematics self-efficacy and achievement.

• Positive changes in career attitudes. Build IT participants demonstrated positive changes in IT career attitudes. Build IT participants demonstrated substantially higher positive changes in their sentiments toward career attitudes than did girls in the comparison group.
• Positive growth in IT career knowledge. In contrast to girls in the comparison group, Build IT participants demonstrate large gains in positive sentiments toward IT careers; both in understanding what these careers entail and in their perceptions of the desirability of working in these careers.
• Positive gains in computer and technology usage and IT skills. Build It participants not only used IT technology to a greater extent than most girls in the comparison group, but they also demonstrated substantial growth in their perceived ability to learn new programs, use the computer to communicate, and describe how information travels through the Internet. Results indicate that increased computer use is significantly associated with gains in perceived IT skills. Thus, as the Build IT curriculum continues to influence participants' IT engagement, we expect to witness even greater gains in perceived skills.
• No effect on conceptual knowledge. With respect to conceptual knowledge, the results do not support the claim that participation increases girls’ basic IT knowledge. There was very little change in knowledge for both the comparison and Build IT groups. On average, respondents in both groups answered approximately half of the questions correctly. Build IT may need to find additional ways to reinforce the knowledge components of the activities.
• Capacity development for ALL STARS is tied to important adjustments in the Build IT curriculum. ALL STARS staff reported that they developed greater IT knowledge and skills over the course of the project, and that they were able to encourage and enhance participant interest and engagement in a number of the Build IT activities. They recommend that Build IT developers can aid capacity development and provide a more appropriate fit to the ALL STARS program by shortening some of the activities, providing more hands-on professional development before each unit, providing in-class assistance during activities, constructing projects with an eye to implementation in residentially dense and urban areas, making the curriculum relevant to urban, non-white middle school girls, and providing better ethnic and racial representation among the chosen IT professionals. We conclude the report by highlighting the limitations to the Year 1 evaluation findings. These include reduction in sample size due to attrition in the ALL STARS program, differences in adapting Build IT curriculum to specific sites, and potential limitations in the survey questions and evaluation design that will need to be reassessed for Year 2 of the project.