Teacher Education Annotated Report Excerpts

Excerpt 1 [Los Angeles Collaborative]

Interpretations & Conclusions:
Presents formative evaluation findings

Findings for Year One are discussed under four general categories:

• Development of the Collaborative,
• Faculty Recruitment and Development,
• Curriculum Development; and
• Student Participation.

[below are some of the findings for the faculty recruitment and development section]

Introduces conclusions

A series of faculty workshops served as the centerpiece for faculty development and recruitment in Year One. Participants attended workshops on cooperative learning techniques, education theory, and classroom assessment. Additionally, faculty were offered a wide variety of other educational programs and symposiums throughout the year.

Reiterates formative evaluation questions

ETI considered the following issues in assessing faculty recruitment and development in Year One:

• To what extent did workshops develop faculty knowledge of student learning/thinking processes and cooperative learning techniques?
• How effective were the faculty workshops in changing faculty teaching practices and attitudes toward cooperative learning?
• How well did LACTE workshops establish a collaborative setting for teachers, community college faculty, and four-year institution faculty?
• To what extent were faculty recruited to participate in LACTE?

Presents balanced conclusions based on quantitative and qualitative data

Specific findings related to faculty recruitment and development are discussed below.

• Faculty development is one of the strongest aspects of LACTE in Year One.Faculty development retreats were generally held every two-to-three months and involved faculty from the various LACTE campuses. Workshops explored cooperative learning techniques, student learning/thinking processes, curriculum development, and classroom assessment. Faculty were given time to work in small discipline-specific groups. In addition, a wide variety of other educational programs and symposiums were offered to LACTE faculty throughout the year. Almost 80 percent of faculty surveyed indicated that faculty workshops met or exceeded their expectations.

"Out of all of LACTE's goals for the first year, faculty development has come along the farthest." (Faculty member)

"LACTE has quite well provided professional development opportunities for science and math faculty in LACTE institutions to become acquainted with alternative styles of teaching and methods of assessment."(Faculty member)

• Overall, faculty indicated on surveys and in focus groups that they felt the workshops to be helpful and that they intended to employ the techniques and information learned during the workshops in their classrooms.
  
  
• Over 85 percent of faculty surveyed indicated that they were "likely" or "definitely likely" to use the information and techniques acquired during the workshops in their classrooms in the future.

"I was not using cooperative learning before LACTE. I never bought into it until it was modeled at the LACTE conference. When I was done [with the conference], I didn't think of doing it any other way."(Faculty member)

"The workshops were very helpful in terms of teaching and learning skills. This was good for me because I didn't have an educational background." (Faculty member)

"I came back after the conference and used some of the ideas the following week. I liked the practical aspects of the ideas." (Faculty member)

• Faculty at the development retreats found their interactions with colleagues from other colleges rewarding.The faculty workshops provided an important opportunity for faculty from different campuses and from different disciplines to meet. After the earlier workshops, faculty indicated that more time needed to be set aside for informal discussions. Later faculty workshops addressed this need.
  
  
• Junior faculty are concerned about the effect of joining LACTE on their tenure efforts. In ETI focus groups, junior faculty expressed concern about the effect of involvement in LACTE on tenure. Faculty suggested that a more visible involvement and commitment of senior campus administrators in LACTE and at faculty conferences would encourage junior faculty participation. Over 70 percent of faculty surveyed at the Spring faculty retreat were tenured professors.

"Experimenting with the way one teaches, regardless of involvement with LACTE, is risky. This experimentation is a lot safer for tenured professors." (Faculty member)

"I would like to see more deportment chairs, associate deans, vice presidents show up more often at LACTE events. This would really propel the program forward." (Faculty member)

"There is enough tension on campus regarding retention, tenure, and promotion that faculty may shy away from LACTE or anything else that may diminish their chances of tenure." (Faculty member)

• Some LACTE campuses and faculty groups are not well-represented at faculty workshops. In addition, College of Education faculty are not being recruited to the faculty workshops. A select group of four-year institutions and community colleges are consistently well represented at the faculty workshops. Other LACTE campuses do not have a strong presence at the workshops. Science and math faculty interviewed at four-year campuses without a strong presence at the workshops suggested that there were not enough discipline-specific experts presenting information to be useful to them. The science and math faculty wanted to hear more about classroom reform from scientists and mathematicians rather than from those involved in education reform more generally.

"More than a dozen science people on my campus are into it [LACTE] but they're not interested in the workshops. The science folks don't feel they can learn much about changing science pedagogy from sociologists, for example." (Science Faculty member)

Presents information on project responsiveness to stakeholders

• Ongoing feedback from faculty members is obtained during the workshops which allows for continuous program improvement. LACTE workshop organizers collect a variety of feedback forms over the course of the faculty workshops. This information is incorporated into the planning of future workshop programs. Faculty expressed that LACTE workshop coordinators had been responsive to their input.

"This year, in response to your stated needs, we will begin to merge teaching/learning strategies with discipline-based course development." (Fall LACTE faculty retreat memorandum.)

Excerpt 2 [Philadelphia Collaborative]

Interpretations & Conclusions:
Describes project implementation progress

Process Results

Most tasks are on target within the established timelines. Each of the tasks has evolved and some have been implemented ahead of schedule although a few will need to move forward more quickly.

Describes project impact on participants

The focus group results indicated that most students liked the increased participation encouraged by the revised courses. They found working in groups helped to reduce anxiety and encouraged students to ask more questions. Students found that discussing content material during the CETP course meetings helped them to understand the material better than a lecture format. Students also responded favorably to the "hands on," application oriented projects which were part of the revised courses. Emphasis on practical aspects of Mathematics and Science theories helped students understand better why they needed to learn the material. CETP courses were more interesting to CETP students, in part, because the CETP students felt that the course material was more applicable to them personally.

Presents project strengths

Students who entered the revised courses feeling competent in mathematics and science seemed to benefit the most from the small group format. These students felt more comfortable asking questions in the discussion groups. Several of these students commented that in lecture-oriented classes, they did not ask questions for fear of saying something "dumb" or "stupid." In addition, the applied nature of the projects demonstrated to students that the concepts they were learning had "real-world" applications. For many students, this was the first time they had found math and science relevant to their lives.

Teacher involvement and accessibility were key influences on students' view of revised courses. The revised courses which students responded to most favorably were those in which the professor mingled with the students during small group activities and provided "clues" on how to approach a problem, took the time during lectures to answer questions, and expressed enthusiasm about the topic.

Describes project impacts on participants

Describes stakeholders' use of formative findings

The surveys of faculty and other grant participants generally had low response rates. Of those responding, most individuals were convinced of the importance of the CETP and felt they wanted to participate. However, many expressed a feeling of lack of direction and lack of interaction with other CETP participants. Based on these reactions, five evaluation/assessment subcommittees were formed in summer of 1996. Of the five, only the Math/Math Ed. subcommittee met on a regular basis. Some of the others did not meet a second time because of lack of participation by those invited. Others appear to break into sub-groups which continued to meet on their own.

Interpretations & Conclusions:
Presents conclusions based on quantitative and qualitative data

Quantitative Evaluation Results

Students in CETP courses responded more favorably to the Science and Mathematics Course Survey at both Temple and Philadelphia Community College and within both mathematics and sciences.

Analysis of the open-ended items illustrated differences favoring the CETP course. In response to the question, "What in this course did you find helpful?," students in CETP courses were more likely to cite as the "Methods Used," interaction with classmates, presentation of subject, and instructor characteristics. Students in traditional courses more often mentioned homework and the textbook as the Methods Used for learning. In terms of "Skills Gained," students in CETP courses were more likely to cite: the ability to apply the subject matter to practical problems, feeling more comfortable with the subject, and learning to work in groups. Responses from students in the traditional courses were less likely to cite the "Skills Gained" by CETP students.

For the last three years, students in CETP courses have expressed more favorable attitudes about mathematics and science courses than students in traditional courses. Students in the CETP courses respond positively to the features of the course which reflects the CETP goals (e.g., use of practical examples, small group work, etc.). In addition, CETP students can identify and seem to appreciate the specific teaching techniques that are promoted by the project. In general, students in CETP courses tend to be more engaged with the faculty member, other students, and the course content.

In Year 3, the Evaluation Committee began to investigate the effects that enrollment in CETP courses had upon enrollment patterns, grades, and retention in subsequent mathematics and science courses. Preliminary analyses indicate that CETP courses may help to retain students. The retention rates, through the third semester, were slightly higher for freshmen who enrolled in CETP as compared to non-CETP mathematics and science courses.

Excerpt 3 [Arizona Collaborative]

Interpretations & Conclusions:
Describes project impact on participants based on quantitative data

Implementation of Modeling Instruction

The first three sections of the participant survey (Method, items 1-13; Technology, items 14-22; and Model-based content, items 23-30) assess participants' implementation of the various components of Modeling Instruction. Based on their responses in these sections, participants were divided into three groups. Group 1 consists of 10 teachers (22%) who claimed to have tried to implement all modeling components addressed in the survey systematically during the 1995-96 academic year, Group 2 consists of 21 teachers (43%) who claimed to have tried to implement some modeling components systematically and others erratically. Group 3 consists of 17 teachers (36%) who claimed that they could implement various components only erratically. No participant claimed not to have tried to implement any modeling component at all, except for one teacher who had no computers and who subsequently answered "never" on related questions.

Interprets respondents' comments

From personal comments written by teachers in groups 2 and 3 at the end of each section of the survey, two major reasons appeared to be behind their inability to implement the modeling approach systematically. The first reason was that most of these teachers argued that in their first year of implementing the modeling approach, following only their first summer workshop in 1995, they were still trying to understand the new approach and figure out how they should go about implementing it in their classrooms. They admit to have fumbled in the process, but believe to have learned how to do things better in this and coming years.

Identifies contextual influences on implementation

The second major reason given by many teachers in the same two groups was that they stumbled on some logistics problems that were beyond their control, and that disrupted their attempt to implement the modeling approach. Among those reasons were administration interference; lack of, or problems with, needed equipment; scheduling problems; class disruption; and student quality and resistance to a non-traditional approach.

Presents conclusions based on quantitative and qualitative data

Triangulates findings with external evaluation

Modeling instruction being only in its first year of implementation at participating high schools, it is understandable that teachers run into such problems. What is encouraging is that all participants have embraced the modeling approach, and that, as illustrated in Figure 3, and as will be further discussed in the evaluation of the summer workshop, teachers in groups 2 and 3 are committed to work on their problems. The summer workshops have fostered an esprit-de-corps among participating teachers that enabled them to pinpoint their individual problems and figure out ways to solve them through discussions with workshop peers. These findings were also supported by <name of person> in her external evaluation report.

Describes impacts based on quantitative data

Impact on Students Reported by Teachers

Some questions on the first three sections of the survey asked participants about the impact of the various components of the modeling approach on their students' understanding of physics. In this respect, and by comparison to traditional instruction, about 91% of respondents consistently said that the following modeling components had "better" or "much better" impact on student understanding:

• The interactive modeling cycle method by comparison to traditional lecturing
• MBL/CBL activities by comparison to traditional laboratory activities
• Model-based content by comparison to traditional course content.

Furthermore, the proportion of respondents who reported that their students' reaction to each of the above modeling components was "favorable" or "very favorable" ("neutral"):

• 78% (15%) with regard to the modeling method
• 93% (3%) with regard to the use of MBL/CBL
• 74% (24%) with regard to model-based content.

Presents balanced conclusions

Evaluation of Summer Workshop

The response distributions on the first three questions of the survey (Table 4) reveal that the overwhelming majority of participants reacted favorably to getting them directly involved in the development of basic models of physics outside of mechanics. Respondents' written comments were also favorable, despite the struggle that some had to go through in the development of materials, as indicated in the excerpts given in Figure 5.

The next four questions dealt with other educational projects that were introduced during the workshop. In their written comments, respondents expressed appreciation of being exposed to a variety of educational projects, but, and as Table 4 shows, participants had mixed reactions about these projects. Virtually all respondents had favorable opinions about the CASTLE project, but the reaction was not as favorable on the other projects, especially InfoMall. One group in each site worked on revising CASTLE materials and aligning them with the modeling philosophy so that interested participants (90%) could use them in their own classrooms.

86% of respondents expressed that their original expectations about the current workshop were either "very well" or "well" fulfilled. As expressed by their comments on the survey (Figure 6), most participants indicated that they developed during this workshop new insights on how they need to modify their teaching practice to help students better understand physics. This was especially true for those teachers in groups 2 and 3, though a few teachers in these groups felt that they still need some time to better understand and implement the modeling approach, especially with regard to the topics addressed in the current workshop.

Presents impact based on quantitative data

Conceptual Understanding of Physics

The overall average gain factor g of .36 of our participants' students is about half a standard deviation above traditional courses' average, and within the range of interactive courses whose threshold is .34.

95-96 FCI results of the three groups of teachers distinguished according to modeling instruction implementation

*These are teachers who have records for both FCI administrations in 95-96, and three of whom had no 94-95 records.
**Mean values are given /100 for pretest and posttests, and /1.00 for the gain factor g.

Compares results with national averages

Addresses intended outcomes

Table 3 above shows that the average gain factor of group 1 teachers (who claimed to have implemented all components of the modeling approach systematically) is .61 which is about one standard deviation above the national average of interactive courses, and close to .69, the highest value of g ever reported in the literature. Comparing 1994-95 and 1995-96 posttest means in Table 3, as well as the gain factor g in Table 3 and in Figure 7, reveals that the more systematic teachers are in implementing the modeling approach, the more their students' conceptual understanding of physics improve as measured by the FCI.

Interprets quantitative results in light of related literature and national averages

A few more points are worth noting in Figure 7:

• All but the students of one teacher in group 1, and those of only one teacher in group 2, averaged above the Newtonian threshold of 60% on the FCI posttest. This is the threshold for students to begin thinking like Newtonians about the motion of physical objects (Hestenes et al., 1992, 1995). Furthermore, the 74% posttest of this group is more than one standard deviation higher than the overall average of all participants, and close to two standard deviations above the national average (national FCI SD=15% according to Hake, 1996).
• The average g of students of every teacher in group 1 falls above the interactive threshold. Among the 10 teachers in this group, there were 7 for whom g=.60 (.48 is the national average for interactive courses), including one teacher for whom g=.73 is the highest FCI gain factor ever reported in the literature.

Presents conclusion

These results indicate that all components of the modeling approach need to be implemented systematically in order for students to reach satisfactory understanding of Newtonian mechanics.

(…)

Compares quantitative results and addresses statistical significance

Views About Knowing and Learning Physics

Within all three cognitive dimensions, students expressed, on average, more expert views than mixed or fold views on both the pretest and posttest. However, Figure 8 shows that students expressed 4% less expert views on the posttest than on the pretest within these dimensions. This decrease is actually not significant, being only about a quarter of a standard deviation (SD=16%), and especially being far smaller than decreases on similar issues reported in the literature.

Compares results reported in the literature

Within all three scientific dimensions, students also expressed, on the average, more expert views than mixed or folk views on both the pretest and posttest. Figure 8 shows that students expressed 1% more expert views on the posttest than on the pretest within the dimensions in question. Although this increase is not significant (SD=18%), it is promising especially when compared to the decline consistently reported in the literature in student views about similar issues.

Relates results about impact to the goals of the project

Compared to FCI results, VASS results show that it is more difficult to have a positive impact on student views about knowing and learning physics than on their conceptions about specific physics topics covered in instruction, like Newtonian mechanics. However, given the commitment of participating teachers, and, hopefully, their improved readiness to align their teaching closer with the modeling approach, the coming years should show better VASS results.

Stakeholder Review & Utilization:
Presents results to participants and stakeholders

Putting Evaluation Results Into Action

All evaluation data summarized above are being continuously shared with the project staff and participating teachers. Furthermore, every teacher is being provided with parallel data about her/his own students.

Presents information useful to participants

Student data were presented to participants during the 1996 summer workshop, and their implications discussed in detail with them. Many participants indicated that the data and subsequent discussions opened their eyes to issues they could not envisage before, and that they will consequently consider taking their teacher practice into new directions.

Describes feedback loop between participants and the project

The project staff are using the data to provide appropriate assistance to participating teachers, and continuously refine the summer workshops. Furthermore, continuous feedback is solicited from participants to enhance the project, and appropriate actions are taken accordingly.

Excerpt 4 [Rocky Mountain Collaborative]

Interpretations & Conclusions:
Interprets results about implementation and impact from quantitative data

Presents balanced conclusions

Frequencies. In general, responses were quite positive on the Student Course Checklist. Students tended to perceive that proposed course characteristics were implemented in RMTEC courses and that they facilitated their learning. Twenty-six out of the 28 items received responses from 50% of the respondents (or more) indicating that the course characteristics was implemented and was helpful to some degree (see Appendix C). Students reported that the following features were implemented in the class and helped them to learn course concepts and content (items receiving endorsement from 80% or more of the respondents, specifically reporting that the strategy was implemented in the course, and it was helpful to some degree):

• cooperative learning groups
• a supportive atmosphere for learning new ideas
• being asked to take responsibility for their own learning
• in-class activities that encouraged them to evaluate their understanding of course concepts
• in-class discussions with other students
• activities that involved reading, writing, and other methods of communication to increase their learning
• assignments that required them to think about course concepts and critically analyze information
• connecting ideas in this course with other scientific and mathematical areas
• being asked to build on things they already knew
• being given the opportunity to ask questions in class.

The three items that received the highest ratings (20% or more of the respondents checked that the strategy "happened and was extremely helpful") were the following:

• cooperative learning in groups
• in-class discussions with other students
• being given the opportunity to ask questions in class.

Eleven of the items received ratings from 20% of the respondents (or more) that indicated they perceived that the classroom strategy was not implemented:

• solving problems related to everyday life or professional significance use of several different types of teaching methods
• having experienced public school teachers involved in the teaching of this course
• frequent feedback about my learning from other students in the course and from the course instructor(s)
• chances for me to influence what goes on in the course
• connecting ideas in this course with other (non-science/math fields)
• having an opportunity to explore ideas in which I am interested
• having a chance to manipulate concrete objects to help me learn course concepts
• laboratory experiences that enhanced my knowledge of course concepts
• field experiences that enhanced my knowledge of course concepts.

Presents conclusions

Interpretation of Frequencies. Students tended to report that courses associated with the RMTEC Project were implemented effectively and consistently with intended course characteristics. It appears that the cooperative learning groups and other opportunities for students to discuss course concepts with other students during class time were effective in facilitating learning. Students also benefited from invitations to ask questions during class time. Synthesizing across items, it seems that the classes were effective in taking into account students' prior understandings and helping students to perceive the wider utility and meaning of course concepts within scientific and mathematical fields of inquiry.

Notes inconclusive results

It remains unclear whether instructors were able to achieve inquiry-oriented instruction without manipulation of concrete manipulatives (35.8% of the students overall responded that they did not have a chance to manipulate concrete objects).

Describes data limitations

Interpretation of Sub-Group Analyses. Men and women did not seem to differ overall in their perceptions of RMTEC course strategies. However, students preparing to teach and those not preparing to teach appeared to hold different perceptions of the courses. It is possible that the teaching students were more attentive to the nature of instructional strategies used by their college instructors, or that they were simply more aware of educational terms included in the items. Even so, when they did perceive that strategies were implemented, non-teaching students were more likely to rate the strategies as being less helpful than were the pre-service teachers (at a statistically significant level in chemistry but not in mathematics courses). This finding is worth considering further—students' previous exposure to particular instructional strategies and their beliefs about appropriate instructional modes may affect their acceptance of the legitimacy of the techniques as well as their assessment of their contribution to learning. Some of the qualitative comments also revealed concerns with particular teaching methods that were foreign to students. Discrepancies of goals and background knowledge between teaching and non-teaching students may render RMTEC instruction differently effective for the two groups. Finally, the data suggests that students from different ethnic backgrounds may have had different experiences in the courses. However, this conclusion is offered guardedly given the small numbers of students from African American, Hispanic American, Asian/Pacific Islander American, and Native American backgrounds.

Presents conclusions based on qualitative data

Qualitative comments provided support for the contention that project instructors taught in a manner consistent with project objectives. Students occasionally expressed concern with initiatives such as cooperative learning, with teaching methods that made it difficult to understand course concepts (e.g., pace of instruction, lack of depth in explanations, failure to answer questions), and with assessment methods. However, comments overall tended to imply that instructors were making an effort to implement project-endorsed strategies such as cooperative learning, inquiry-oriented approaches, etc.

Presents balanced quantitative data

Highlights from the aggregated student course checklist across institutions:

• 94% (Fall, 1995) and 87% (Spring, 1996) of the students stated that they had engaged in cooperative groups and that they were helpful
• 93% (Fall, 1995) and 87% (Spring, 1996) of the students stated that there was a supportive atmosphere for learning new ideas and that it was helpful
• 89% (Fall, 1995) and 78% (Spring, 1996) of the students stated that there had been use of technology to support learning and that it had been helpful
• 89% (Fall, 1995) and 84% (Spring, 1996) of the students stated that their assignments required them to think about concepts and critically analyze information
• 85% (Fall, 1995) and 77% (Spring, 1996) of the students stated that they were in learning settings which showed respect for diversity
• 20 out of 30 items on the questionnaire were viewed by at least 80% of the students as happened and helpful (Fall, 1995).

Areas that needed improvement were as follows:

• only 49% (Fall, 1995) and 46% (Spring, 1996) of the students experienced public school teachers involved in teaching of the course and found it helpful
• 64% (Fall, 1995) and 54% (Spring, 1996) of the students obtained frequent feedback about their learning from other students
• 68% (Fall, 1995) and 64% (Spring, 1996) of the students obtained frequent feedback about their learning from the course instructor(s).

Results of Faculty Survey. Seven faculty members (70% of those responding) indicated that it would be beneficial for them to have help in developing alternative assessments. Three indicated that they do not need help. One faculty member said that it would be beneficial to have sessions for the whole department, not just RMTEC faculty.

Changes in assessment the faculty are interested in making in RMTEC courses in the future include (1) more attention to application of research in personal rationale and teaching strategies, (2) encouragement of action research in classroom observation, (3) more systematic collection of data, (4) development of a better set of questions for the personal interview, and (5) information about alternative assessments that are available and are being developed.

Describes project impacts

Research is being conducted by the RMTEC faculty at all 3 institutions to assess if the instructional methods have been successful. Faculty also have observed that enrollment in later chemistry classes has risen at UNC, although fewer students took Introduction to Chemistry. As MSCD a faculty member noted that students' interest is higher and class discussions seem more authentic. Another faculty member at UNC noted that student attitudes toward the RMTEC courses are very positive and that scores on tests are significantly higher than they were in the Spring 1995 traditional course.

Compares project with traditional practices

Faculty were asked what changes in RMTEC courses, if any, were made in instructional methods that differed from the traditional format of other courses they have taught. Responses are grouped with respect to Chemistry, Science/Math Education, or Mathematics in Table 2.

Summarizes qualitative data about results in table format

Describes impacts

In the past faculty indicated that they used written assessments (tests) or written and oral presentations of information or both. Assessment methods have changed in all RMTEC courses these faculty teach. The "usual" tests are a thing of the past. Faculty now require more writing, include projects, portfolios, discussion participation, reaction to research, journals, group assessment, exams with more critical thinking questions, proficiency-based assessments; tests include hands-on activities and most use rubrics.

Excerpt 5 [Los Angeles Collaborative]

Recommendations:
Presents recommendations for project improvements

Recommendations

ETI believes that there are specific recommendations that, when implemented, will enhance the various program dimensions of LACTE and strengthen LACTE overall in Years Two through Five. Based on the findings outlined above, the Evaluation and Training Institute offers the suggestions listed below:

[Recommendations were provided for the four general categories of activities. Below are recommendations for the faculty recruitment and development.]

• K-12 teachers should be included as participants in future workshops and other LACTE activities. The inclusion of K-12 teachers would ensure educational continuity and that materials developed are germane to the needs of teachers and prospective educators at that level. Since a primary focus of LACTE is K-12 teacher preparation, it stands to reason that educators at this level could contribute a great deal to the faculty workshops and to curriculum development in terms of both insight and experience.
• LACTE needs to continue efforts to enlist a visible involvement and commitment of senior campus administrators to encourage faculty participation, and particularly junior faculty participation. Administrators at four-year and community college campuses should continue to be made aware of the impact of the collaboration in order to generate high-level support for LACTE on the various campuses.
• LACTE should emphasize the presence of discipline-specific experts presenting information at faculty conferences. Faculty are particularly interested in getting more discipline-specific information on instructional techniques and curriculum revision. At the same time, to further the goals of LACTE, it is also important to continue to have opportunities for faculty from different disciplines to work together.
• LACTE faculty development workshops should provide information on how new technologies can be incorporated into revised or new courses. Faculty are interested in learning about the use of new technologies in math and science classrooms. LACTE may be able to attract new faculty to workshop sessions that look at how new technologies, including Web sites, are being incorporated into courses.

Excerpt 6 [Philadelphia Collaborative]

Recommendations

The Evaluation Committee recommends that course directors examine the specific features of each CETP course (e.g., use of practical examples, small group work, multi-discipline collaboration, student and faculty characteristics) to determine which features are having the most impact and which features may need revision. As the CETP program grows, it will be important to provide new CETP faculty with the skills, documentation, and training needed to teach CETP classes.

Excerpt 7 [Rocky Mountain Collaborative]

Interpretations & Conclusions:
Presents conclusions

Recommendations

The fact that students were unaware of the contribution of the experienced public school teachers (also a finding with the Fall 1995 administration of the survey) raises questions about the roles and responsibilities of the classroom teachers and their day-to-day duties in the classes in which they did participate, there was substantial variation in the amount of time they spent in class sessions. Also, instructors may wish to consider ways in which connections can be strengthened in students' minds between course concepts they teach and aspects of students' everyday lives and professional goals.

Recommendations:
Addresses intended outcomes

Conclusion. Data were limited in many respects, but they do suggest that RMTEC courses were offered in a manner largely consistent with intended course characteristics. Instructors may benefit from additional professional development experiences with a few instructional strategies.

Excerpt 8 [Oklahoma Collaborative]

Interpretations & Conclusions:
Presents conclusions

Across all Summer Academy participants, desired attitudinal changes occurred on nearly every scale. As summarized in Table 2.3, nearly all scales showed change in the desired direction. Furthermore, science-based curricula showed unexpected effects on attitudes towards math. Participants increased in their self-efficacy toward teaching math and decreased in their feelings of math anxiety.

Addresses intended and unintended outcomes

However, several patterns of change contradicted our expectations. Participants did not change in their attitude toward inquiry-based teaching techniques and participants showed an increase on the Pupil Control Ideology scale, indicating increased authoritarian values as a result of their summer academy experience. Overall, the pattern of change was in line with expectations for the Summer Academies.

Summarizes results about statistical significance in table format

Italics denote scales where the direction of change or lack thereof was unexpected. Change was assessed with paired T-test at significance level of p < .05, N = 131.

Presents quantitative results and measurement criteria

To assess the degree of change these measurements were standardized on a common scale. The degree of change was divided by the standard deviation of the scores at Time 1 to create a d statistic (shown in Table 2.4.). D-scores greater than + .10 or - .10 were statistically significant at the .05 level of probability.

The greatest change occurred in attitudes related to Hands-on and Inquiry-based education, Science education, and participants' motivation to continue to learn about teaching methods. The results are displayed graphically in Chart 1. As expected, the scales for science attitudes changed more than attitudes towards math.

Italics denote scales where the direction of change or lack thereof was unexpected. Change was assessed with paired T-test at significance level of p < .05, N = 131.

Presents conclusions that address intended and unintended outcomes

Conclusions

• The summer academies had the desired effect of increasing positive attitudes towards the teaching profession, science, and toward using reform techniques in the classroom.
• The summer academies had the unexpected effect of ameliorating math anxiety, and increasing self-efficacy toward teaching math.
• The summer academies had the unexpected, albeit small, effect of increasing authoritarian attitudes in the classroom.

Summarizes results in table format

2.4. Did Pre-service and In-service Teachers Show Different Patterns of Attitudinal Change?

To evaluate whether pre-service and in-service teachers differed in their experience of the summer academies, we examined differences by service status of the participants (high school student, college student, and in-service teacher). In general, high school students and in-service teachers both changed in desired ways and in line with overall results. College students showed less positive attitudinal change, and showed more change in unexpected directions. In most cases, high school students showed the greatest magnitude of change, and college students showed the least change. These findings are shown in Tables 2.5 and 2.6. Group sizes were similar for all three groups.

Italics denote scales where the direction of change or lack thereof was unexpected.

Italics denote scales where the direction of change or lack thereof was unexpected.

(…)

Interprets results in context of project

Standardized change scores are given for each service status group in Table 2.8, and the scores are plotted in Chart 2. From the chart it is clear that the High School group showed relatively large increases on the majority of attitudinal measures, especially learning motivation, and self-efficacy measures. Most importantly, the High School group showed the largest increase in positive attitudes towards the teaching profession. This finding was comforting in light of the fact that one of the primary goals of the summer academies was to recruit new teachers.

Presents conclusions and speculates on causation

The college students showed some change in the desired direction. They increased on self-efficacy attitudes toward inquiry-based teaching and the field of science. They also decreased on science and math anxiety, and did not increase on the Pupil Control Ideology scale. College students did not show increasing positive attitudes toward the teaching profession, learning motivation, hands-on self-efficacy, or science outcome, and they showed a decrease in attitudes toward inquiry-based teaching. The mixed results for the college students could be the result of several factors. One likely factor is that these students were already exposed to many of the reforms being presented in the summer academies in their current curriculum in college. Thus, these students may already have relatively positive attitudes toward many of O-TEC related goals.

Describes intended and unintended outcomes

The in-service group showed many of the changes that we expected. They increased in their feelings of self-efficacy toward using hands-on and inquiry-based teaching techniques and in their feelings that good teaching will lead students to understand science better (Science Outcome). The in-service group also showed marked decreases in science and math anxiety. Interestingly, the in-service group showed the largest increase in authoritarian attitudes in the classroom. We have no explanation for this finding and will pursue a replication of these results during the next round of summer academies.

Lists conclusions from findings

The analysis of differences across service status suggest:

• The summer academies had the largest influence on changing attitudes toward the teaching profession in the most important group-high school students.
• It appears that the summer academies had the greatest influence on high-school pre-service teachers and in-service teachers, and had mixed effects on college students.
• In-service teachers showed increases on many of the desired outcomes and an unexpected increase in authoritarianism.

(…)

Interpretations & Conclusions:
Lists conclusions

2.9. Overall Conclusions and Recommendations for Summer Academies

• The experience of the summer academies appeared to have strong effects on participants. Most participants showed increases in their confidence to employ reform teaching techniques and decreases in science and math anxiety.
• High school participants showed the greatest increase in attitudes toward teaching. Future summer academies may want to focus more of their efforts on high school participants, as this is where the greatest 'recruitment' effect was found.
• There were no significant differences between the experience of men and women. There were no racial differences on quantitative indices. Follow-up qualitative interviews revealed no racial issues or bias in the delivery of the summer academy topics.
• Although some sites showed less change than others did, the overall pattern of change was consistent with ideal expectations.
• Participants were enthusiastically positive about their experience in the summer academies.

Excerpt 9 [Maryland Collaborative]

Interpretations & Conclusions:
Reiterates evaluation question

Research question 1: Is there a difference between the MCTP teacher candidates' and the non-MCTP teacher candidates' attitudes and beliefs about mathematics and science?

Describes quantitative analysis and data

Describes sample

Interprets results

Sections of our 45 item survey instrument that were verified by factor analysis dealt with beliefs about mathematics and science (=.76); attitudes toward mathematics and science (=.81); beliefs about teaching mathematics and science (=.69); attitudes toward learning to teach mathematics and science (=.79); and attitudes toward teaching mathematics and science (=.60). Data were obtained (total instrument responses, N=931; teacher candidates, n=609; MCTP teacher candidates, n=286) during the 1995/96 academic year from 33 reform-based mathematics, science, or pedagogy undergraduate college classes taught in 7 higher education institutions in Maryland. A key finding is that there are differences between the attitudes and beliefs of MCTP and non-MCTP teacher candidates as they initially encounter reform-based instruction in their teacher preparation classes. The evidence suggests that non-MCTP teacher candidates in MCTP classes become less enthusiastic about taking more college courses in mathematics and science, and less enthusiastic about learning how to use technology to teach these subjects.

Presents generalizable conclusion

The attitudes and beliefs of the MCTP teacher candidates do not experience this negative impact. This finding contributes to a better understanding of the struggles reform minded college-level science and mathematics content and pedagogical specialists face in instituting curricular transformation in teacher education programs.

Reiterates evaluation question

Research question 2: Do MCTP teacher candidates' attitudes toward and beliefs about mathematics and science change over time as they participate in the MCTP classes?

Describes sample

Provides response rate

Presents key finding about change over time

Data were obtained in the 1995/96 academic year, at the end of the 1996 summer semester, Fall 1996, Spring 1997, summer 1997, and Fall 1997. The 1995/1996 sample (N=386) were all MCTP teacher candidates in 33 MCTP classes taught in seven different institutions of higher education in Maryland. The summer sample (N=24) were competitively selected summer intern MCTP teacher candidates. The subsequent sample consisted of a representative sample (N=100) of the second-year cohort of MCTP teacher candidates who were mailed a survey at the end of each semester with directions to complete and return it (80%+ response rate). A key finding is that in the 1995/96 academic year the means of the attitudes and beliefs of the MCTP teacher candidates (moderately high) tended to remain the same. However, in the summer of 1996, the means of the attitude and beliefs of the MCTP teacher candidates showed a sharp increase improvement, which over time has steadily approached a ceiling effect in all five scales.

Draws implications for further research

These data suggest that to document the extent of the positive impact of the MCTP on the attitudes and belief means of the MCTP teacher candidates it requires a longitudinal data collection strategy extending throughout the undergraduate program.

Excerpt 10 [Louisiana Collaborative]

Interpretations & Conclusions:
Presents conclusions about strengths

Status of Reform on Louisiana Campuses—General

Substantial progress was made again this past year in expanding and institutionalizing reform on campuses. Major progress has been made toward campus-wide systemic reform at many of the regional universities, with pockets of reform more likely to be found in the larger, more traditional research-based universities. Even in those universities, however, campus-wide progress was made.

Much of the reform progress at all universities has been focused on preparing pre-service elementary school teachers. The specified mathematics and science curriculum for elementary pre-service teachers, and the comparatively lower number of students preparing to be high school mathematics and science teachers have made the reform focus on pre-service elementary teacher preparation predictable.

Role of State Support

Consistent feedback from campuses is that LaCEPT has played a critical role in bringing about reform on the campuses. The national and state support provided by NSF, the Board of Regents, and LaCEPT provided a jump start for mathematics and science pre-service and undergraduate reform by

• allowing faculty members time and resources to reform courses particularly through release time,
• giving administrators confidence that this effort was important and credible since it had the support of the Board of Regents and the National Science Foundation,
• increasing the awareness of various administrators and faculty members on the campuses and focusing their efforts.

Presents conclusions about stakeholder involvement

The Louisiana Systemic Initiatives Program (LaSIP) also has provided a helpful foundation for LaCEPT reform efforts. Many of the professors who have been leaders in LaCEPT were first involved in LaSIP activities. Many of those have stayed involved in LaSIP projects in addition to their work with LaCEPT.

Recommendations:
Recommends further involvement by key stakeholder group

The continued leadership and support of the Board of Regents is critical if the momentum of the reform effort is to be maintained. While much of the progress brought about by LaCEPT has been institutionalized and will continue on many campuses, campus administrators will be looking closely at the level and nature of Board of Reagents support as LaCEPT activities draw to a close. Administrators will be making choices about class size, laboratory space and lab class for pre-service teachers, and a number of important cost-related issues. The degree to which the Board continues to maintain its interest and support for reform and the nature of its policies, including its financial support decision, will in large measure dictate how many of these university level decisions will be made.

Summarizes stakeholder attitudes and perspectives

Various university personnel expressed the importance of Board leadership in convincing university administrators and others that pre-service teacher education, especially in mathematics and the sciences, should be a priority. University personnel consistently indicated that continuing clear and concrete expressions of support by the Board would attract the attention of university administrators and other policymakers. They also indicated that the Board should be cautious about mandating changes beyond setting standards upon which programs will be evaluated and funds distributed. As they conclude, top-down support and incentives for change are superior to top-down requirements for change.

(…)

Interpretations & Conclusions:
Presents conclusions about area of concern

Despite remarkable progress, however, the issue of rewarding and promoting faculty is still important to address. Universities have basically used different criteria for counting mathematics or science education development and research for mathematics or science professors. Universities generally count such work as scholarly work only if there are publications or other evidence of success, including presentations at national or regional workshops or conferences, and successful grant applications. The level of evidence varies. However more research-oriented universities tend to weigh significantly such work only for a few people specifically hired to teach undergraduate survey courses or only if the professor has content research publications.

Excerpt 11 [Louisiana Collaborative]

Interpretations & Conclusions

Board of Regents

Following are some of the state-level recommendations and conclusions derived from comments of faculty and administrators and the results of other formative evaluation activities over the last three years. They have already been reviewed and validated by many participants in the reform Process. Accompanying these recommendations and conclusions are comments about what results have occurred relating to, and possibly influenced by, these recommendations and conclusions.

Recommendations:
Weaves findings into conclusions and recommendations

Board of Regents

There is a critical need for support and leadership at the state level, particularly from the Board of Regents. Various university personnel expressed the importance of Board leadership in convincing university administrators and others that pre-service education, especially in mathematics and the sciences, should be a priority. University personnel consistently indicated that clear and concrete expressions of support by the Board will attract the attention of university administrators and other policymakers. At the same time, the Board should be cautious about mandating changes beyond setting standards upon which programs will be evaluated and funds distributed. Generally, top down support and incentives for change is superior to requirements for change.

Presents results of evaluation recommendations

Result: The board of Regents has taken decisive leadership by adopting far-reaching policy statements.

LaCEPT

LaCEPT should continue to promote sharing across campuses by promoting more regional and subject matter workshops and possibly developing a clearinghouse of successful ideas, approaches, curriculum plans, and materials. LaCEPT could consider organizing and paying the expenses for a three-day retreat for sharing among the campuses.

Result: Additional emphasis was placed on sharing among campuses at the January conference. Also, LaCEPT has set aside funds for additional statewide workshops in 1997 to accomplish others of these purposes.

Excerpt 12 [Oregon Collaborative]

Interpretations & Conclusions:
Identifies contextual influences

OCEPT involves every higher education institution in the state of Oregon; the necessity of keeping it decentralized and flexible also creates the possibility that its impact will be dissipated and lack focus. This section discusses this issue and others which will affect OCEPT's success, and identifies strategies we are developing to confront these issues which make OCEPT somewhat unique among Collaboratives.

Presents project challenges

The first visit of the National Visitors Committee in November 1997 confirmed and reinforced our beliefs that OCEPT has some major challenges in front of it and needed to devise new strategies for overcoming these. These challenges, and other issues that have been identified by the Management Team and Mentor Teams, are outlined here, along with our approach in addressing them.

The scope of reform efforts

From the beginning OCEPT has had a formal structure to promote cross-institutional, discipline-based collaboration: the Faculty Mentor Teams. These teams are connecting with and strengthening existing networks, such as professional societies, in their disciplines. We had been less inclined to create formal structures for promoting change within individual institutions, partly because of the scope of the problem (34 institutions) and because of the concern that OCEPT-created structures might not last beyond the period of funding. We were heartened to hear confirmation from the NVC that we did indeed need to target our efforts toward institutional change in some, not all, participating institutions. As discussed earlier in the section on inter- and intra-institutional collaboration, it has made sense to focus on the nine subgrant institutions as a starting point.

Recommendations:
Suggests project improvements endorsed by stakeholders

The Management Team, consisting of the co-PIs at each of these institutions, has agreed that we need more focused recruiting of Faculty Fellows, aimed at encouraging reforms tied directly to OCEPT goals, and at creating teams of Mentors and Fellows who will have a more lasting impact within the institution. The co-PIs are all clearly interested in working both with their Mentor Teams and with faculty on their own institutions, and have the professional stature to accomplish this.

Presents results related to broader educational issues

Focusing on teacher preparation

Often the educational experiences in our colleges and universities that need the most revision are introductory mathematics and science courses. In a system such as Oregon's, in which most students receive a bachelor's degree in a discipline and then go on to a graduate teacher education program, introductory courses for both majors and non-majors must generally serve a greater variety of students than just future teachers. The faculty who teach these courses are not likely to initiate changes that are appropriate for only one small population in their class. Fortunately, it is easy to make the argument that what will be good for future teachers will be good for all students; for example, making sure students gain a sound understanding of fundamental math and science concepts, the role of science in society, and its strengths and limitations as a form of inquiry. Problems of science literacy in the general population are well known to most math and science faculty, and it is a short leap to see the importance of the science and math preparation of future teachers in solving this problem.

Presents initial project weaknesses and improvements over time

Suggests project improvements based on evaluation results

While we have been able to use these arguments to recruit many new participants to OCEPT and give them a new view of the importance of science and mathematics in teacher preparation, Faculty Fellows in Year 1 did not automatically make the connection between their own teaching and the needs of future teachers. Many made no overt effort to identify students in their courses who were interested in teaching, or discussed with their students issues related to teaching as they might have discussed issues related to other careers. In the application and selection process of Year 2 Faculty Fellows, the need to connect their projects directly to OCEPT's goals for teacher preparation was made much more clear, and it will continue to be emphasized much more strongly in the Summer Institute and the work of the Mentor Teams with their Fellows. We do need to be able to support change in many kinds of courses, both those for committed future teachers who can identify themselves as such, and those for potential future teachers who may not. The Management Team has agreed, however, that unless special efforts are made to focus on courses and programs in which future teachers can be identified and their special needs addressed, our efforts are likely to lack real impact.

Presents project weaknesses

Recommendations:

Changing requirements vs. changing courses: which comes first?

The Teacher Education team's study of teacher education admissions requirements is providing evidence that most elementary programs require little or no mathematics or science. Most programs make recommendations, but we do not have evidence yet that many students heed these recommendations. There is little incentive to require more mathematics and science when the requirements for teacher licensure specify only passing scores on standardized tests. Our discussions with teacher educators also have uncovered their often negative view of college mathematics and science. Echoing general criticisms of undergraduate science and math courses, they see large, impersonal, competitive courses which emphasize the memorization of disconnected facts, as the source of their students' anxiety about math and science and their reluctance to teach these subjects. Requiring more courses like these would only make their jobs harder. We need to convince teacher education faculty that good mathematics and science courses do exist, that OCEPT is committed to improving others, and that requiring or recommending these courses will improve their students' ability and interest in teaching science and mathematics. At the same time, we need to convince college mathematics and science faculty that making changes in their courses to meet the needs of future teachers will in fact be worth their while—that future teachers do take these courses, and will do so in greater numbers if we can convince teacher education programs to require or recommend them.

Interpretations & Conclusions:
Presents conclusions about project weaknesses

Recommendations:
Presents recommendations and actions taken

Focusing the diversity agenda

As with so many other aspects of OCEPT, the task of increasing the diversity of the teacher workforce involves multiple constituencies and stakeholders, many of which do not currently talk to or even know of each other. A meeting of representatives from special programs for underrepresented groups uncovered the fact that many of these programs do not communicate with one another, even when they serve the same populations. Further, the disconnect between pre-college and college-level programs was striking. One major problem that contributes to this disconnect is that these programs are almost all severely marginalized. Unlike regular academic departments, funding for these programs is dependent on grant funding or other "soft money" sources. Their leaders are not seen as regular faculty members, and faculty from other departments do not know or work with them. Not only does OCEPT need to find out which strategies used by these programs are most effective, we need to share these strategies with mathematics, science, and education faculty members so they can also get involved. Most believe they lack the special expertise to work on diversity issues. Not only do we need to increase faculty's connection with these special programs, we need to devise ways of meeting diversity goals within regular classrooms and programs for all students, not just underrepresented populations.

This wide-ranging and complex agenda has been difficult for us to get a handle on. One strategy we are putting in place this year is to fund two regular Faculty Fellows (not special consultants) to gather and synthesize information about special programs as well as strategies faculty can use to meet the needs of all students in their classes, addressing the real diversity that exists there, among learning styles, interests, and professional goals, as well as gender and ethnicity.

Presents conclusions about implementation progress

Successful progress

While dealing with these challenges, we feel confident in pointing out that OCEPT is already on the road to success in many areas. We are changing many faculty's view of teacher education and themselves as teacher educators. Participants in last Summer's Institute reported great changes in their awareness of the needs of future teachers, particularly in mathematics and science, and in their own feeling of responsibility for encouraging and supporting future teachers. OCEPT leaders have learned some valuable lessons about making sure these changes in beliefs are translated into changes in practice, and are planning more effective ways of addressing this issue with new Institute participants, most of whom have already indicated they know "nothing" about teacher education and are involved "not at all."

We are also raising the level of dialog about science/math teaching and learning among OCEPT participants, and will continue this effort. For example, several of our Faculty Mentors are engaged in research on the nature of inquiry in science education and how it is represented in classroom activity. The idea is that current notions of "science as a process" are inadequate, and need to include thorough description of how students think and question, not just what they do in "hands-on" laboratories.

Presents conclusions about best practices

OCEPT participants also are focusing more on change in faculty's beliefs about teaching and learning, rather than course change, in teaching improvement. Faculty who modify their beliefs about teaching and learning will modify their teaching practice in all their courses and other interactions with students, and can serve as catalysts of change for their colleagues. Attempting to make courses "teacher-proof" so that they can be taught by anyone, if the original developer leaves or changes his/her teaching assignment, is unrealistic, since the teacher's beliefs about teaching and learning will impact his/her practice in ways that cannot be prescribed by a syllabus or laboratory handbook.

Recommendations:
Presents future project directions

OCEPT participants have embraced the notion of "teaching as scholarship," that faculty need to make their own decisions about teaching and how they determine its effectiveness, within a context of collaboration, dissemination, and peer review. We are now working to make sure the needs of "teaching scholars" in all kinds of institutions, community colleges as well as research universities, are addressed. We are committed to placing the responsibility for decision-making about teaching, assessment, and dissemination in the hands of the practitioners themselves, rather than prescribing any one set of methods. This will produce a more richer and more varied picture of teaching improvement in Oregon, and one that will initially look more ambiguous as well. We will be focusing on building a case for the effectiveness of our efforts, rather than assuming that relying on one type of data alone will produce results that speak for themselves.

Excerpt 13 [Montana Collaborative]

Interpretations & Conclusions:
Presents conclusions while acknowledging data limitations

Implications and Conclusions

Student responses on this survey provide valuable, but limited, information on the course impacted by the STEP Project. Surveys offer student perspectives while ignoring instructor perspectives. By virtue of attempting to revise and improve their courses, instructors inevitably are put into disequilibruim, and it may take time to integrate instructional strategies into their repertoire. Thus, for example instructors may struggle with ways in which they can best integrate experimentation, inquiry modes of instruction, and cooperative learning into their existing methods. Depending on experiences of the instructor, preliminary efforts may not be completely successful. Hence, survey responses are summarized here with caution as to their meaning and interpretation. Generally, responses were quite positive. Students perceived that course characteristics were implemented that are consistent with goals of STEP. Areas to target for improvement include technology use and assessment methods.

Some evidence exists that varied assessment methods are used.

Provides findings useful for project improvement

Percentages for education courses are considerably higher than survey aggregate data. Students are not routinely asked to explain their thinking or understanding or to evaluate their own work.

Presents project strengths

Highlights from the aggregated course survey across institutions:

Student affect questions were given high ratings (86 to 95% positive):

• Course work was challenging and interesting
• Students felt free to talk with instructors individually about their work or progress
• Students encounter materials or activities that provoke curiosity and do problems or projects that they find interesting.

83% of students gave details on how these courses were different from other math/science courses they have taken.

79% of students reported that they see a connection between course content and the world outside of school.

In Education courses, students reported they get to verbally explain thinking and understanding. They are also given opportunities to evaluate their own work.

Presents conclusion

In summary, responses were quite positive. Students indicated that instructional characteristics, consistent with the goals of STEP, were implemented in the STEP courses Spring 1996.

Excerpt 14 [Montana Collaborative]

Stakeholder Review & Utilization:
Overviews report structure

Describes stakeholder involvement in and dissemination of report

The introduction of the report will include background and goals generated by the persons involved in the activity as well as the original proposal. The evaluation plan, procedures, and the results section of the reports are guided by the evaluation questions. These reports are 'stand alone' documents which include complete information about an activity, how it was evaluated, and the results and/or discussion. Report drafts are to be circulated to the staff member(s) responsible for developing and/or carrying out the activity to get input, especially on the description of the purposes and procedures of the activity.

Indicates how reports will be utilized

The reports or portions of them are used as a basis for an overall summative evaluation of the project. They are also used for other purposes such as developing papers for presentation at local or national meetings, developing publications, or inclusion in project reports to NSF. Reports and data are available to all project staff for these or other purposes. The annual report will be completed by March 15 each year.

Excerpt 15 [City Science Workshop, City College of New York]

Interpretations & Conclusions:
Presents coding categories used to examine results

A goal of the City Science Workshop was to have teachers involve their students in cooperative work. The coding categories reflect whether it occurred as well as the nature of that work. As can be seen from the graph, students worked in groups in most classes, although the results of the group work are not usually the focus of later class work.

Presents findings in graph format

In reviewing the data on this aspect of the program, it was seen that twelve percent of the participants did not use cooperative learning to any significant degree. The remaining participants made use of cooperative learning in different degrees. Twenty-three percent used groups, but did not allocate sufficient time for group discussions. Forty-one percent of the participants who used small groups allocated time for discussions and presentations at the end of the period. Ten percent allocated time for class discussions following group sharing. Fourteen percent created the environment for investigations to continue based on class discussions.

Presents conclusions

Participants have shown a willingness towards the use of cooperative learning. This is reflected in classroom organization and activities. Prior to their involvement in the program, many teachers arranged their classrooms in rows and spent a greater percentage of the instructional period at the front of the classroom. Following their involvement in the program participants have shown an increased tendency to allow their students to work in groups, make presentations and have discussions about their projects. There are now many participants who have rearranged their classroom so that group work is facilitated. Many teachers said that although this arrangement was initially established for science investigations they are now using it for other subject areas. Participants have also devoted a greater percentage of the instructional period towards group investigations.

Excerpt 16 [Teachers Leading Teachers, University of South Carolina]

Interpretations & Conclusions:
Relates results to project goals

A. Project Goal 1: To strengthen the PHYSICAL SCIENCE KNOWLEDGE base of Grades 4-9 teachers in South Carolina schools.

Summary Results for Project Goal 1: Lead Teachers have made substantial cognitive gains in both physical science subject matter and in the related pedagogy. The physical science concepts that were studied were germane to their needs. The hands-on activities that they restructured were appropriate for their respective grade levels and were effective teaching tools. Confidence levels were significantly elevated and LTs described their teaching as more effective as a result of participating in TLT.

Excerpt 17 [Anonymous 1]

Interpretations & Conclusions:
Presents quantitative results in table format

Presents qualitative data from multiple methods of inquiry to support quantitative results

Written comments illustrate this level of satisfaction and describe the prevailing climate in these sessions:

It has been by far the best workshop I have attended. I walked away wishing we could continue! We learned something new daily that we could use in the classroom. What more can we ask for!

Every Program A session was useful. All could be adapted to the state curriculum goals for my grade level.

The resources and teacher materials will truly help in preparing lessons.

The program was wonderful! The presenter was excellent. His knowledge and enthusiasm was a plus. He made our workshops fun and never dull. This is so important when we come tired and overworked and leave feeling energized, excited and inspired.

(…)

Presents corroborative results from observations

HRI observations corroborated the teacher reports. Observed sessions were characterized by an atmosphere of respect and collaboration. Individuals were encouraged but not forced to participate in discussions and hands-on activities, and most did so willingly even when it came to pronouncing difficult Latin plant names in front of their peers. Facilitators provided new information and followed an organized plan for each session, artfully drawing on participants' creativity and past experiences by use of questioning strategies. It appears that the workshop experience itself has been fine-tuned over several years of implementation by the Museum.

Recommendations:
Suggests project improvements

As was true with Cohort 1 in 1996, the one aspect of the workshops which seemed somewhat less satisfactory to Cohort 2 respondents was the amount of time allocated for workshops. Written responses to the question about suggesting ways to improve the Program A workshops provide some insight into this complex problem for school staff.

Though only a few isolated comments indicated dissatisfaction with the use of workday or after-school hours, more prevalent was the sentiment that the sessions were rushed. Quite a few teachers commented that more time was needed to complete or prepare the areas or complete tasks. It appeared that teachers found themselves excited and enthusiastic time to complete the work necessary to be ready for the next phase of the professional development in approximately four weeks time.

(…)

Interpretations & Conclusions:
Interprets statistically significant results

Impacts on Preparedness

Participants were queried pre- and post- about their preparedness to teach various environmental science concepts. Results are shown in Table 7 below. Substantial and significant changes were found throughout.

*=Statistically significant at the .05 level.

Presents balanced conclusions

The most obvious increase in confidence was in the participants' feelings of preparedness to use the outdoor environment of the school yard and to bring outdoor elements inside the classroom for teaching science and other subjects. As indicated from workshop evaluation results, the Museum does a good job of teaching about the environment and pointing out existing outdoor resources while helping school to create new such resources. In addition, it is clear that the workshops model sound science teaching methods, which benefit teachers by building their confidence in using investigation and discovery learning in their classrooms.

Additional questions sought feedback about participants' preparedness to teach not only environmental science, but other subject areas as well. Possible responses were: not well qualified, adequately qualified, and very well qualified. Figure 1 below shows that (1) there is significant change in the percent of respondents who feel very well qualified to teach each of the subjects in question except mathematics; and (2) both before and after Program A, participants were more likely to feel well-qualified to teach reading, mathematics, and social sciences than the natural sciences and technology.

See Figure 1

Post tests in all of the above subject areas except mathematics were statistically significant at the .05 level.

Addresses intended and unintended outcomes

Certainly the gain in confidence in teaching earth, life, and integrated sciences is consistent with the subject focus of Program A. Other gains may reflect the emphasis Program A places on integrating environmental science with reading, writing, and other subject areas. Classroom artifacts indicate that children do write and draw about their Program A experiences. In fact, one school has developed the Program A Newsletter ("By Kids for Kids") with stories about birds and other wildlife.

(…)

Presents qualitative results to support conclusion

Sustainability—Interviews with Cohort 1 Schools

HRI's interviews revealed that collaboration can be the key in maintaining enthusiasm. In a small school were nearly every staff member received professional development (including the principal, media specialist and PE teacher) the vision seems to be alive and well. The gardens and pond at this school are frequently used (and, coincidentally, were being visited by a class at the moment of one of the phone interviews), and maintenance does not seem to be a problem in the second year as evidenced by this interview excerpt:

As spring was approaching and the wildflowers were starting to return, some of the classes have been out weeding. And things like that have been going on without anybody being instructed, "Hey, we need to weed the garden." Everybody has just sort of assumed responsibilities for projects started. It's just continuing on.

Presents conclusions about the importance of stakeholders to the project and contextual influences

It is noteworthy that the principal of this school is actively involved, and helped plan and implement the second year professional development activity which was to be open to all teachers and assistants at the school. This event was to be a three-day, two-night field trip to a barrier island for workshops with a Museum leader as well as other resource persons.

Principal support was also key in a larger school where only one-third of the staff members opted to participate. Again, the principal participated in the program when possible, and plans were made for the second year of the program early in the year. Unfortunately, due to school closing for inclement weather, this session was delayed a year. In spite of the delay, the program seems to be maintaining itself. In fact, this school has become a nationwide award-winner in recycling, the recipient of a $5000 grant that the school intends to use to construct a nature trail and a larger pond.

In contrast, there are schools where the impact of Program A was less noticeable. Many factors were cited as contributing to this, among them: changes in school leadership; lack or organization; several lead teachers, but no single person in charge; failure to schedule the second year professional development workshops; state curriculum mandates emphasizing other areas of particular reason in any given school, it is a fact that staff turnover occurs from time to time, and some populations have lower parental involvement than others. A number of these obstacles could be addressed in the Program A program to prepare lead teachers and others by suggesting road blocks to anticipate and how to navigate around them.

(…)

Interpretations & Conclusions:
Presents conclusions about project strengths

Summary and Recommendations

In the second year of operation as an NSF-funded project, Program A is continuing its steady progress toward meeting its goals. At the midway point in this funding cycle, there are obviously a number of outstanding strengths that can be built upon to increase the overall impact of the project in years to come. Included are the following:

• Program A offers a series of in-service experiences that project participants find to be extremely high quality professional development. Participants are especially pleased with the resources and teaching materials used; the careful planning and organization of the sessions, the expertise and preparedness of workshop leaders, and the opportunity to conduct hands-on, experiential activities.
  
  
• The project models the use of experiential teaching for participants throughout the workshop sessions. Modeling outdoor investigation versus simply describing the types of things that could be done was extremely effective.
  
  
• The project effectively works to recruit schools committed to the project's philosophy and through an excellent series of workshop experiences, the project has been able to maintain participant interest and enthusiasm throughout the first year's sessions. Follow-up sessions, successfully scheduled in most of the schools, have proved to be an extender of this enthusiasm and interest.
  
  
• The project provides participating schools with a wealth of resources that will enable them to significantly enhance the value of their school grounds for teaching and learning.
  
  
• The project's professional development experiences seem to be yielding the intended impacts of increasing the use of the environment in teaching science and other subjects; positively impacting the attitudes of these school populations; building the teachers' confidence in their ability and preparedness to teach environmental science and other related subjects; and enhancing the value of the school grounds as a learning resource.
  
  

Recommendations:
Suggests project improvements

While the in-service sessions have been highly successful in working towards the stated goals, the following recommendations are suggested to further strengthen the program:

• Consider ways to support and maintain initial enthusiasm of Program A participants by providing additional opportunities for networking among Program A schools beyond the first year for the purpose of sharing lessons, activities and strategies for success.
  
  
• Consider ways to ease the time constraint felt by a number of participants, either by adjusting the workshop schedule or at least initiating discussion of this issue in sessions throughout the year.
  
  
• Continue to model approaches being advocated for classroom implementation, being explicit in pointing out the specific strategies that are being modeled, and discussing ways to share these techniques with other adult non-participants in the school.
  
  
• Continue to offer participants opportunities for input and reflection and, where possible, work to increase opportunities to center discussions around questions such as "How can this activity be used in the classroom? How can others in the school community be involved as well?"
  
  
• Continue to look for ways to strengthen the leadership-development component of the project or develop other strategies to increase the likelihood that the project will realize its intended long-term project impact.

Excerpt 18 [Educational Cooperative Service Unit, MN]

Interpretations & Conclusions:
Presents results related to project goals and evaluation questions

EVALUATION QUESTIONS and STATUS REPORT

GOAL 1. To show teachers how computational science can greatly enhance the teaching of mathematics and science in the high school classroom.

1. What training activities were provided to participants?

All training activities were logged and categorized as to instructional strategy, facilities used, and amount of time by topic and curriculum area. Profiles of the activities follow:

The primary instructional strategy was hands-on training using a micro-computer lab. On a composite basis, sixty-one per cent (61%) of the instructional time was spend in a hands-on setting, eleven (11%) per cent was on field trips, and eight per cent (8%) was in a lecture setting. A new category for describing the instructional strategies was added for the second year of instruction. This category is referred to as "Group Discussion." This category was added because a significant amount of time was spent on a team basis where the members of the teams collaborated on the development of their team's project. From these small group discussions, the information was shared by the entire cadre through the use of large group presentations and discussions. Twenty-three per cent (23%) of the instructional time for Cadre 1 in its second year was spent on this type of group discussion and thirty-one per cent (31%) of the time for Cadre 2 was spent in group discussion.

Presents results in table format

In addition to these classroom strategies, a computational scientist from the business world was identified to work with each project team. Each project team met at least once with their computational scientist to discuss concerns and strategies related to their individual projects.

(…)

2. What were the teachers' evaluations of the activities?

A comprehensive evaluation questionnaire was administered to the teachers at the end of each week's activities. The responses to two of the items at the end of the third week which give an overall perception of the participants, are displayed on the following tables.

The first question read, "The session provided learning opportunities that support the stated outcomes." Table 5 displays the responses from Cadre 1.

Presents results in graph format

The average of the responses to this item by Cadre 1 was 4.7 on a scale of one to five, with five being the highest possible score.

Table 6 displays the responses from Cadre 2. The average of the responses to this item by Cadre 2 was 4.5 on a scale of one to five, with five being the highest possible score.

The second item was, "I feel that I know enough to get started (with the help of my colleagues)." Table 7 displays the responses from Cadre 1.

The average of the responses to this item by Cadre 1 was 4.4 on a scale of one to five, with five being the highest possible score.

Table 8 displays the responses from Cadre 2. The average of the responses to this item by Cadre 2 was 4.6 on a scale of one to five, with five being the highest possible score.

GOAL 2. To develop a cadre of teachers who know what computational science is, understand its relevance in the contemporary work place, and can use some of the computational science techniques in their classrooms to teach existing curriculum.

1. How many teachers have been trained? Where are they from? What is their background and training? Are minorities represented? What is the diversity represented?

Descriptive profiles were developed for the participants to identify the categories represented in the program. The profiles are displayed in the following ables.

Presents participant characteristics in table format

As anticipated, most participants came from mathematics and science. On a composite basis, thirty-seven per cent (37%) of the participants were mathematics teachers, forty-four per cent (44%) were science teachers, fifteen per cent (15%) taught technology, three per cent (3%) English, and one per cent (1%) social studies.

The experience level of the participants was determined based on the experience stated in the application forms. On a composite basis, sixty-six per cent (66%) of the participants had multiple years of experience and use of multiple software packages, twenty-nine per cent (29%) had moderate experience, and six per cent (6%) had very little previous experience with computers.

On a composite basis, fifty-nine per cent (59%) of the participants came from senior high schools, thirty-four per cent (34%) came from middle schools, and seven per cent (7%) came from alternative schools which provide secondary education.

On a composite basis, fifty-one per cent (51%) of the participants were male; forty-nine (49%) per cent female.

On a composite basis, ninety-six percent (96%) of the participants were white, four percent (4%) were minorities.

Identifies future evaluation activities

FUTURE EVALUATOR ACTIVITIES

1. Finalize paper-pencil indicators to assess teacher and student changes in attitude, knowledge, and behavior.
2. Finalize observation checklists that can be used as indicators of behavior changes.
3. Schedule the collection of all future evaluation data.
4. Collect and compile all future evaluation data.
5. Prepare a final report that can be used for summative evaluation and dissemination.

The final evaluation report will assess the projects impact on both teacher effectiveness and student learning. By documenting the activities of the project as it develops, the evaluation will be able to relate the factors that contribute to the effectiveness of the project.

Excerpt 19 [New York City Collaborative]

Stakeholder Review & Utilization:
Presents follow-up stakeholder actions in table format

Excerpt 20 [Arizona Collaborative]

Interpretations & Conclusions:
Compares results with other studies

Hake (1996) has conducted a national survey of student performance on the FCI and established some national FCI standards to which our results are henceforth compared. The 1994-95 posttest mean of 47% shown in Table 2 is typical of national high school FCI results. The 1995-96 pretest average of 26% is about 4% below the national average, and the corresponding posttest of 51% is about 4% above the national average. The FCI gain factor g (defined as Posttest%-Pretest%/100-Pretest%) is a better measurement of the effectiveness of instruction. According to Hake, the national average value of g is about .23 in physics courses taught following the traditional approach of lecture and demonstration, and .48 in interactive physics courses that engage students actively in "heads-on (always) and hands-on (usually) activities which yield immediate feedback through discussion with peers and/or instructors."

Excerpt 21 [Virtual Economics, National Center for Research in Education]

Interpretations & Conclusions:
Interprets key findings about groups

Table 2.3 shows that there is remarkable consistency in the high ratings of Virtual Economics given by the beta and final user groups in spite of any differences that might affect the overall assessment. Both groups gave excellent or good ratings to Virtual Economics in terms of its usefulness, assistance for learning economics, as a resource guide, and all the other factors. The similarity in positive responses lend more credibility to the reliability and validity of the results from the survey evaluations.

On most features, however, the teachers using the final version of Virtual Economics rated it higher than did the beta sample perhaps because of the improvements in the final version. Most of these differences were statistically significant. The group of teachers who rated the final version assessed the overall usefulness of Virtual Economics a quarter of a point higher than the beta teacher group (t-value: 2.40). The same significant difference was found for the features of identifying standards, raising awareness, user friendliness, and helpfulness of the manual.

*Means are calculated using the following scale:

(4) E=excellent; (3) G=good; (2) F=fair; (1) P=poor.

**Standard deviations are in parentheses.