Teacher Education Annotated Report Excerpts

Excerpt 1 [Los Angeles Collaborative]

Interpretations & Conclusions:
Presents formative evaluation findings

Findings for Year One are discussed under four general categories:

• Development of the Collaborative,
• Faculty Recruitment and Development,
• Curriculum Development; and
• Student Participation.

[below are some of the findings for the faculty recruitment and development section]

Introduces conclusions

A series of faculty workshops served as the centerpiece for faculty development and recruitment in Year One. Participants attended workshops on cooperative learning techniques, education theory, and classroom assessment. Additionally, faculty were offered a wide variety of other educational programs and symposiums throughout the year.

Reiterates formative evaluation questions

ETI considered the following issues in assessing faculty recruitment and development in Year One:

• To what extent did workshops develop faculty knowledge of student learning/thinking processes and cooperative learning techniques?
• How effective were the faculty workshops in changing faculty teaching practices and attitudes toward cooperative learning?
• How well did LACTE workshops establish a collaborative setting for teachers, community college faculty, and four-year institution faculty?
• To what extent were faculty recruited to participate in LACTE?

Presents balanced conclusions based on quantitative and qualitative data

Specific findings related to faculty recruitment and development are discussed below.

• Faculty development is one of the strongest aspects of LACTE in Year One.Faculty development retreats were generally held every two-to-three months and involved faculty from the various LACTE campuses. Workshops explored cooperative learning techniques, student learning/thinking processes, curriculum development, and classroom assessment. Faculty were given time to work in small discipline-specific groups. In addition, a wide variety of other educational programs and symposiums were offered to LACTE faculty throughout the year. Almost 80 percent of faculty surveyed indicated that faculty workshops met or exceeded their expectations.

"Out of all of LACTE's goals for the first year, faculty development has come along the farthest." (Faculty member)

"LACTE has quite well provided professional development opportunities for science and math faculty in LACTE institutions to become acquainted with alternative styles of teaching and methods of assessment."(Faculty member)

• Overall, faculty indicated on surveys and in focus groups that they felt the workshops to be helpful and that they intended to employ the techniques and information learned during the workshops in their classrooms.
  
  
• Over 85 percent of faculty surveyed indicated that they were "likely" or "definitely likely" to use the information and techniques acquired during the workshops in their classrooms in the future.

"I was not using cooperative learning before LACTE. I never bought into it until it was modeled at the LACTE conference. When I was done [with the conference], I didn't think of doing it any other way."(Faculty member)

"The workshops were very helpful in terms of teaching and learning skills. This was good for me because I didn't have an educational background." (Faculty member)

"I came back after the conference and used some of the ideas the following week. I liked the practical aspects of the ideas." (Faculty member)

• Faculty at the development retreats found their interactions with colleagues from other colleges rewarding.The faculty workshops provided an important opportunity for faculty from different campuses and from different disciplines to meet. After the earlier workshops, faculty indicated that more time needed to be set aside for informal discussions. Later faculty workshops addressed this need.
  
  
• Junior faculty are concerned about the effect of joining LACTE on their tenure efforts. In ETI focus groups, junior faculty expressed concern about the effect of involvement in LACTE on tenure. Faculty suggested that a more visible involvement and commitment of senior campus administrators in LACTE and at faculty conferences would encourage junior faculty participation. Over 70 percent of faculty surveyed at the Spring faculty retreat were tenured professors.

"Experimenting with the way one teaches, regardless of involvement with LACTE, is risky. This experimentation is a lot safer for tenured professors." (Faculty member)

"I would like to see more deportment chairs, associate deans, vice presidents show up more often at LACTE events. This would really propel the program forward." (Faculty member)

"There is enough tension on campus regarding retention, tenure, and promotion that faculty may shy away from LACTE or anything else that may diminish their chances of tenure." (Faculty member)

• Some LACTE campuses and faculty groups are not well-represented at faculty workshops. In addition, College of Education faculty are not being recruited to the faculty workshops. A select group of four-year institutions and community colleges are consistently well represented at the faculty workshops. Other LACTE campuses do not have a strong presence at the workshops. Science and math faculty interviewed at four-year campuses without a strong presence at the workshops suggested that there were not enough discipline-specific experts presenting information to be useful to them. The science and math faculty wanted to hear more about classroom reform from scientists and mathematicians rather than from those involved in education reform more generally.

"More than a dozen science people on my campus are into it [LACTE] but they're not interested in the workshops. The science folks don't feel they can learn much about changing science pedagogy from sociologists, for example." (Science Faculty member)

Presents information on project responsiveness to stakeholders

• Ongoing feedback from faculty members is obtained during the workshops which allows for continuous program improvement. LACTE workshop organizers collect a variety of feedback forms over the course of the faculty workshops. This information is incorporated into the planning of future workshop programs. Faculty expressed that LACTE workshop coordinators had been responsive to their input.

"This year, in response to your stated needs, we will begin to merge teaching/learning strategies with discipline-based course development." (Fall LACTE faculty retreat memorandum.)

Excerpt 2 [Philadelphia Collaborative]

Interpretations & Conclusions:
Describes project implementation progress

Process Results

Most tasks are on target within the established timelines. Each of the tasks has evolved and some have been implemented ahead of schedule although a few will need to move forward more quickly.

Describes project impact on participants

The focus group results indicated that most students liked the increased participation encouraged by the revised courses. They found working in groups helped to reduce anxiety and encouraged students to ask more questions. Students found that discussing content material during the CETP course meetings helped them to understand the material better than a lecture format. Students also responded favorably to the "hands on," application oriented projects which were part of the revised courses. Emphasis on practical aspects of Mathematics and Science theories helped students understand better why they needed to learn the material. CETP courses were more interesting to CETP students, in part, because the CETP students felt that the course material was more applicable to them personally.

Presents project strengths

Students who entered the revised courses feeling competent in mathematics and science seemed to benefit the most from the small group format. These students felt more comfortable asking questions in the discussion groups. Several of these students commented that in lecture-oriented classes, they did not ask questions for fear of saying something "dumb" or "stupid." In addition, the applied nature of the projects demonstrated to students that the concepts they were learning had "real-world" applications. For many students, this was the first time they had found math and science relevant to their lives.

Teacher involvement and accessibility were key influences on students' view of revised courses. The revised courses which students responded to most favorably were those in which the professor mingled with the students during small group activities and provided "clues" on how to approach a problem, took the time during lectures to answer questions, and expressed enthusiasm about the topic.

Describes project impacts on participants

Describes stakeholders' use of formative findings

The surveys of faculty and other grant participants generally had low response rates. Of those responding, most individuals were convinced of the importance of the CETP and felt they wanted to participate. However, many expressed a feeling of lack of direction and lack of interaction with other CETP participants. Based on these reactions, five evaluation/assessment subcommittees were formed in summer of 1996. Of the five, only the Math/Math Ed. subcommittee met on a regular basis. Some of the others did not meet a second time because of lack of participation by those invited. Others appear to break into sub-groups which continued to meet on their own.

Interpretations & Conclusions:
Presents conclusions based on quantitative and qualitative data

Quantitative Evaluation Results

Students in CETP courses responded more favorably to the Science and Mathematics Course Survey at both Temple and Philadelphia Community College and within both mathematics and sciences.

Analysis of the open-ended items illustrated differences favoring the CETP course. In response to the question, "What in this course did you find helpful?," students in CETP courses were more likely to cite as the "Methods Used," interaction with classmates, presentation of subject, and instructor characteristics. Students in traditional courses more often mentioned homework and the textbook as the Methods Used for learning. In terms of "Skills Gained," students in CETP courses were more likely to cite: the ability to apply the subject matter to practical problems, feeling more comfortable with the subject, and learning to work in groups. Responses from students in the traditional courses were less likely to cite the "Skills Gained" by CETP students.

For the last three years, students in CETP courses have expressed more favorable attitudes about mathematics and science courses than students in traditional courses. Students in the CETP courses respond positively to the features of the course which reflects the CETP goals (e.g., use of practical examples, small group work, etc.). In addition, CETP students can identify and seem to appreciate the specific teaching techniques that are promoted by the project. In general, students in CETP courses tend to be more engaged with the faculty member, other students, and the course content.

In Year 3, the Evaluation Committee began to investigate the effects that enrollment in CETP courses had upon enrollment patterns, grades, and retention in subsequent mathematics and science courses. Preliminary analyses indicate that CETP courses may help to retain students. The retention rates, through the third semester, were slightly higher for freshmen who enrolled in CETP as compared to non-CETP mathematics and science courses.

Excerpt 3 [Arizona Collaborative]

Interpretations & Conclusions:
Describes project impact on participants based on quantitative data

Implementation of Modeling Instruction

The first three sections of the participant survey (Method, items 1-13; Technology, items 14-22; and Model-based content, items 23-30) assess participants' implementation of the various components of Modeling Instruction. Based on their responses in these sections, participants were divided into three groups. Group 1 consists of 10 teachers (22%) who claimed to have tried to implement all modeling components addressed in the survey systematically during the 1995-96 academic year, Group 2 consists of 21 teachers (43%) who claimed to have tried to implement some modeling components systematically and others erratically. Group 3 consists of 17 teachers (36%) who claimed that they could implement various components only erratically. No participant claimed not to have tried to implement any modeling component at all, except for one teacher who had no computers and who subsequently answered "never" on related questions.

Interprets respondents' comments

From personal comments written by teachers in groups 2 and 3 at the end of each section of the survey, two major reasons appeared to be behind their inability to implement the modeling approach systematically. The first reason was that most of these teachers argued that in their first year of implementing the modeling approach, following only their first summer workshop in 1995, they were still trying to understand the new approach and figure out how they should go about implementing it in their classrooms. They admit to have fumbled in the process, but believe to have learned how to do things better in this and coming years.

Identifies contextual influences on implementation

The second major reason given by many teachers in the same two groups was that they stumbled on some logistics problems that were beyond their control, and that disrupted their attempt to implement the modeling approach. Among those reasons were administration interference; lack of, or problems with, needed equipment; scheduling problems; class disruption; and student quality and resistance to a non-traditional approach.

Presents conclusions based on quantitative and qualitative data

Triangulates findings with external evaluation

Modeling instruction being only in its first year of implementation at participating high schools, it is understandable that teachers run into such problems. What is encouraging is that all participants have embraced the modeling approach, and that, as illustrated in Figure 3, and as will be further discussed in the evaluation of the summer workshop, teachers in groups 2 and 3 are committed to work on their problems. The summer workshops have fostered an esprit-de-corps among participating teachers that enabled them to pinpoint their individual problems and figure out ways to solve them through discussions with workshop peers. These findings were also supported by <name of person> in her external evaluation report.

Describes impacts based on quantitative data

Impact on Students Reported by Teachers

Some questions on the first three sections of the survey asked participants about the impact of the various components of the modeling approach on their students' understanding of physics. In this respect, and by comparison to traditional instruction, about 91% of respondents consistently said that the following modeling components had "better" or "much better" impact on student understanding:

• The interactive modeling cycle method by comparison to traditional lecturing
• MBL/CBL activities by comparison to traditional laboratory activities
• Model-based content by comparison to traditional course content.

Furthermore, the proportion of respondents who reported that their students' reaction to each of the above modeling components was "favorable" or "very favorable" ("neutral"):

• 78% (15%) with regard to the modeling method
• 93% (3%) with regard to the use of MBL/CBL
• 74% (24%) with regard to model-based content.

Presents balanced conclusions

Evaluation of Summer Workshop

The response distributions on the first three questions of the survey (Table 4) reveal that the overwhelming majority of participants reacted favorably to getting them directly involved in the development of basic models of physics outside of mechanics. Respondents' written comments were also favorable, despite the struggle that some had to go through in the development of materials, as indicated in the excerpts given in Figure 5.

The next four questions dealt with other educational projects that were introduced during the workshop. In their written comments, respondents expressed appreciation of being exposed to a variety of educational projects, but, and as Table 4 shows, participants had mixed reactions about these projects. Virtually all respondents had favorable opinions about the CASTLE project, but the reaction was not as favorable on the other projects, especially InfoMall. One group in each site worked on revising CASTLE materials and aligning them with the modeling philosophy so that interested participants (90%) could use them in their own classrooms.

86% of respondents expressed that their original expectations about the current workshop were either "very well" or "well" fulfilled. As expressed by their comments on the survey (Figure 6), most participants indicated that they developed during this workshop new insights on how they need to modify their teaching practice to help students better understand physics. This was especially true for those teachers in groups 2 and 3, though a few teachers in these groups felt that they still need some time to better understand and implement the modeling approach, especially with regard to the topics addressed in the current workshop.

Presents impact based on quantitative data

Conceptual Understanding of Physics

The overall average gain factor g of .36 of our participants' students is about half a standard deviation above traditional courses' average, and within the range of interactive courses whose threshold is .34.

95-96 FCI results of the three groups of teachers distinguished according to modeling instruction implementation

*These are teachers who have records for both FCI administrations in 95-96, and three of whom had no 94-95 records.
**Mean values are given /100 for pretest and posttests, and /1.00 for the gain factor g.

Compares results with national averages

Addresses intended outcomes

Table 3 above shows that the average gain factor of group 1 teachers (who claimed to have implemented all components of the modeling approach systematically) is .61 which is about one standard deviation above the national average of interactive courses, and close to .69, the highest value of g ever reported in the literature. Comparing 1994-95 and 1995-96 posttest means in Table 3, as well as the gain factor g in Table 3 and in Figure 7, reveals that the more systematic teachers are in implementing the modeling approach, the more their students' conceptual understanding of physics improve as measured by the FCI.

Interprets quantitative results in light of related literature and national averages

A few more points are worth noting in Figure 7:

• All but the students of one teacher in group 1, and those of only one teacher in group 2, averaged above the Newtonian threshold of 60% on the FCI posttest. This is the threshold for students to begin thinking like Newtonians about the motion of physical objects (Hestenes et al., 1992, 1995). Furthermore, the 74% posttest of this group is more than one standard deviation higher than the overall average of all participants, and close to two standard deviations above the national average (national FCI SD=15% according to Hake, 1996).
• The average g of students of every teacher in group 1 falls above the interactive threshold. Among the 10 teachers in this group, there were 7 for whom g=.60 (.48 is the national average for interactive courses), including one teacher for whom g=.73 is the highest FCI gain factor ever reported in the literature.

Presents conclusion

These results indicate that all components of the modeling approach need to be implemented systematically in order for students to reach satisfactory understanding of Newtonian mechanics.

(…)

Compares quantitative results and addresses statistical significance

Views About Knowing and Learning Physics

Within all three cognitive dimensions, students expressed, on average, more expert views than mixed or fold views on both the pretest and posttest. However, Figure 8 shows that students expressed 4% less expert views on the posttest than on the pretest within these dimensions. This decrease is actually not significant, being only about a quarter of a standard deviation (SD=16%), and especially being far smaller than decreases on similar issues reported in the literature.

Compares results reported in the literature

Within all three scientific dimensions, students also expressed, on the average, more expert views than mixed or folk views on both the pretest and posttest. Figure 8 shows that students expressed 1% more expert views on the posttest than on the pretest within the dimensions in question. Although this increase is not significant (SD=18%), it is promising especially when compared to the decline consistently reported in the literature in student views about similar issues.

Relates results about impact to the goals of the project

Compared to FCI results, VASS results show that it is more difficult to have a positive impact on student views about knowing and learning physics than on their conceptions about specific physics topics covered in instruction, like Newtonian mechanics. However, given the commitment of participating teachers, and, hopefully, their improved readiness to align their teaching closer with the modeling approach, the coming years should show better VASS results.

Stakeholder Review & Utilization:
Presents results to participants and stakeholders

Putting Evaluation Results Into Action

All evaluation data summarized above are being continuously shared with the project staff and participating teachers. Furthermore, every teacher is being provided with parallel data about her/his own students.

Presents information useful to participants

Student data were presented to participants during the 1996 summer workshop, and their implications discussed in detail with them. Many participants indicated that the data and subsequent discussions opened their eyes to issues they could not envisage before, and that they will consequently consider taking their teacher practice into new directions.

Describes feedback loop between participants and the project

The project staff are using the data to provide appropriate assistance to participating teachers, and continuously refine the summer workshops. Furthermore, continuous feedback is solicited from participants to enhance the project, and appropriate actions are taken accordingly.

Excerpt 4 [Rocky Mountain Collaborative]

Interpretations & Conclusions:
Interprets results about implementation and impact from quantitative data

Presents balanced conclusions

Frequencies. In general, responses were quite positive on the Student Course Checklist. Students tended to perceive that proposed course characteristics were implemented in RMTEC courses and that they facilitated their learning. Twenty-six out of the 28 items received responses from 50% of the respondents (or more) indicating that the course characteristics was implemented and was helpful to some degree (see Appendix C). Students reported that the following features were implemented in the class and helped them to learn course concepts and content (items receiving endorsement from 80% or more of the respondents, specifically reporting that the strategy was implemented in the course, and it was helpful to some degree):

• cooperative learning groups
• a supportive atmosphere for learning new ideas
• being asked to take responsibility for their own learning
• in-class activities that encouraged them to evaluate their understanding of course concepts
• in-class discussions with other students
• activities that involved reading, writing, and other methods of communication to increase their learning
• assignments that required them to think about course concepts and critically analyze information
• connecting ideas in this course with other scientific and mathematical areas
• being asked to build on things they already knew
• being given the opportunity to ask questions in class.

The three items that received the highest ratings (20% or more of the respondents checked that the strategy "happened and was extremely helpful") were the following:

• cooperative learning in groups
• in-class discussions with other students
• being given the opportunity to ask questions in class.

Eleven of the items received ratings from 20% of the respondents (or more) that indicated they perceived that the classroom strategy was not implemented:

• solving problems related to everyday life or professional significance use of several different types of teaching methods
• having experienced public school teachers involved in the teaching of this course
• frequent feedback about my learning from other students in the course and from the course instructor(s)
• chances for me to influence what goes on in the course
• connecting ideas in this course with other (non-science/math fields)
• having an opportunity to explore ideas in which I am interested
• having a chance to manipulate concrete objects to help me learn course concepts
• laboratory experiences that enhanced my knowledge of course concepts
• field experiences that enhanced my knowledge of course concepts.

Presents conclusions

Interpretation of Frequencies. Students tended to report that courses associated with the RMTEC Project were implemented effectively and consistently with intended course characteristics. It appears that the cooperative learning groups and other opportunities for students to discuss course concepts with other students during class time were effective in facilitating learning. Students also benefited from invitations to ask questions during class time. Synthesizing across items, it seems that the classes were effective in taking into account students' prior understandings and helping students to perceive the wider utility and meaning of course concepts within scientific and mathematical fields of inquiry.

Notes inconclusive results

It remains unclear whether instructors were able to achieve inquiry-oriented instruction without manipulation of concrete manipulatives (35.8% of the students overall responded that they did not have a chance to manipulate concrete objects).

Describes data limitations

Interpretation of Sub-Group Analyses. Men and women did not seem to differ overall in their perceptions of RMTEC course strategies. However, students preparing to teach and those not preparing to teach appeared to hold different perceptions of the courses. It is possible that the teaching students were more attentive to the nature of instructional strategies used by their college instructors, or that they were simply more aware of educational terms included in the items. Even so, when they did perceive that strategies were implemented, non-teaching students were more likely to rate the strategies as being less helpful than were the pre-service teachers (at a statistically significant level in chemistry but not in mathematics courses). This finding is worth considering further—students' previous exposure to particular instructional strategies and their beliefs about appropriate instructional modes may affect their acceptance of the legitimacy of the techniques as well as their assessment of their contribution to learning. Some of the qualitative comments also revealed concerns with particular teaching methods that were foreign to students. Discrepancies of goals and background knowledge between teaching and non-teaching students may render RMTEC instruction differently effective for the two groups. Finally, the data suggests that students from different ethnic backgrounds may have had different experiences in the courses. However, this conclusion is offered guardedly given the small numbers of students from African American, Hispanic American, Asian/Pacific Islander American, and Native American backgrounds.

Presents conclusions based on qualitative data

Qualitative comments provided support for the contention that project instructors taught in a manner consistent with project objectives. Students occasionally expressed concern with initiatives such as cooperative learning, with teaching methods that made it difficult to understand course concepts (e.g., pace of instruction, lack of depth in explanations, failure to answer questions), and with assessment methods. However, comments overall tended to imply that instructors were making an effort to implement project-endorsed strategies such as cooperative learning, inquiry-oriented approaches, etc.

Presents balanced quantitative data

Highlights from the aggregated student course checklist across institutions:

• 94% (Fall, 1995) and 87% (Spring, 1996) of the students stated that they had engaged in cooperative groups and that they were helpful
• 93% (Fall, 1995) and 87% (Spring, 1996) of the students stated that there was a supportive atmosphere for learning new ideas and that it was helpful
• 89% (Fall, 1995) and 78% (Spring, 1996) of the students stated that there had been use of technology to support learning and that it had been helpful
• 89% (Fall, 1995) and 84% (Spring, 1996) of the students stated that their assignments required them to think about concepts and critically analyze information
• 85% (Fall, 1995) and 77% (Spring, 1996) of the students stated that they were in learning settings which showed respect for diversity
• 20 out of 30 items on the questionnaire were viewed by at least 80% of the students as happened and helpful (Fall, 1995).

Areas that needed improvement were as follows:

• only 49% (Fall, 1995) and 46% (Spring, 1996) of the students experienced public school teachers involved in teaching of the course and found it helpful
• 64% (Fall, 1995) and 54% (Spring, 1996) of the students obtained frequent feedback about their learning from other students
• 68% (Fall, 1995) and 64% (Spring, 1996) of the students obtained frequent feedback about their learning from the course instructor(s).

Results of Faculty Survey. Seven faculty members (70% of those responding) indicated that it would be beneficial for them to have help in developing alternative assessments. Three indicated that they do not need help. One faculty member said that it would be beneficial to have sessions for the whole department, not just RMTEC faculty.

Changes in assessment the faculty are interested in making in RMTEC courses in the future include (1) more attention to application of research in personal rationale and teaching strategies, (2) encouragement of action research in classroom observation, (3) more systematic collection of data, (4) development of a better set of questions for the personal interview, and (5) information about alternative assessments that are available and are being developed.

Describes project impacts

Research is being conducted by the RMTEC faculty at all 3 institutions to assess if the instructional methods have been successful. Faculty also have observed that enrollment in later chemistry classes has risen at UNC, although fewer students took Introduction to Chemistry. As MSCD a faculty member noted that students' interest is higher and class discussions seem more authentic. Another faculty member at UNC noted that student attitudes toward the RMTEC courses are very positive and that scores on tests are significantly higher than they were in the Spring 1995 traditional course.

Compares project with traditional practices

Faculty were asked what changes in RMTEC courses, if any, were made in instructional methods that differed from the traditional format of other courses they have taught. Responses are grouped with respect to Chemistry, Science/Math Education, or Mathematics in Table 2.

Summarizes qualitative data about results in table format

Describes impacts

In the past faculty indicated that they used written assessments (tests) or written and oral presentations of information or both. Assessment methods have changed in all RMTEC courses these faculty teach. The "usual" tests are a thing of the past. Faculty now require more writing, include projects, portfolios, discussion participation, reaction to research, journals, group assessment, exams with more critical thinking questions, proficiency-based assessments; tests include hands-on activities and most use rubrics.

Excerpt 5 [Los Angeles Collaborative]

Recommendations:
Presents recommendations for project improvements

Recommendations

ETI believes that there are specific recommendations that, when implemented, will enhance the various program dimensions of LACTE and strengthen LACTE overall in Years Two through Five. Based on the findings outlined above, the Evaluation and Training Institute offers the suggestions listed below:

[Recommendations were provided for the four general categories of activities. Below are recommendations for the faculty recruitment and development.]

• K-12 teachers should be included as participants in future workshops and other LACTE activities. The inclusion of K-12 teachers would ensure educational continuity and that materials developed are germane to the needs of teachers and prospective educators at that level. Since a primary focus of LACTE is K-12 teacher preparation, it stands to reason that educators at this level could contribute a great deal to the faculty workshops and to curriculum development in terms of both insight and experience.
• LACTE needs to continue efforts to enlist a visible involvement and commitment of senior campus administrators to encourage faculty participation, and particularly junior faculty participation. Administrators at four-year and community college campuses should continue to be made aware of the impact of the collaboration in order to generate high-level support for LACTE on the various campuses.
• LACTE should emphasize the presence of discipline-specific experts presenting information at faculty conferences. Faculty are particularly interested in getting more discipline-specific information on instructional techniques and curriculum revision. At the same time, to further the goals of LACTE, it is also important to continue to have opportunities for faculty from different disciplines to work together.
• LACTE faculty development workshops should provide information on how new technologies can be incorporated into revised or new courses. Faculty are interested in learning about the use of new technologies in math and science classrooms. LACTE may be able to attract new faculty to workshop sessions that look at how new technologies, including Web sites, are being incorporated into courses.

Excerpt 6 [Philadelphia Collaborative]

Recommendations

The Evaluation Committee recommends that course directors examine the specific features of each CETP course (e.g., use of practical examples, small group work, multi-discipline collaboration, student and faculty characteristics) to determine which features are having the most impact and which features may need revision. As the CETP program grows, it will be important to provide new CETP faculty with the skills, documentation, and training needed to teach CETP classes.

Excerpt 7 [Rocky Mountain Collaborative]

Interpretations & Conclusions:
Presents conclusions

Recommendations

The fact that students were unaware of the contribution of the experienced public school teachers (also a finding with the Fall 1995 administration of the survey) raises questions about the roles and responsibilities of the classroom teachers and their day-to-day duties in the classes in which they did participate, there was substantial variation in the amount of time they spent in class sessions. Also, instructors may wish to consider ways in which connections can be strengthened in students' minds between course concepts they teach and aspects of students' everyday lives and professional goals.

Recommendations:
Addresses intended outcomes

Conclusion. Data were limited in many respects, but they do suggest that RMTEC courses were offered in a manner largely consistent with intended course characteristics. Instructors may benefit from additional professional development experiences with a few instructional strategies.

Excerpt 8 [Oklahoma Collaborative]

Interpretations & Conclusions:
Presents conclusions

Across all Summer Academy participants, desired attitudinal changes occurred on nearly every scale. As summarized in Table 2.3, nearly all scales showed change in the desired direction. Furthermore, science-based curricula showed unexpected effects on attitudes towards math. Participants increased in their self-efficacy toward teaching math and decreased in their feelings of math anxiety.

Addresses intended and unintended outcomes

However, several patterns of change contradicted our expectations. Participants did not change in their attitude toward inquiry-based teaching techniques and participants showed an increase on the Pupil Control Ideology scale, indicating increased authoritarian values as a result of their summer academy experience. Overall, the pattern of change was in line with expectations for the Summer Academies.

Summarizes results about statistical significance in table format

Italics denote scales where the direction of change or lack thereof was unexpected. Change was assessed with paired T-test at significance level of p < .05, N = 131.

Presents quantitative results and measurement criteria

To assess the degree of change these measurements were standardized on a common scale. The degree of change was divided by the standard deviation of the scores at Time 1 to create a d statistic (shown in Table 2.4.). D-scores greater than + .10 or - .10 were statistically significant at the .05 level of probability.

The greatest change occurred in attitudes related to Hands-on and Inquiry-based education, Science education, and participants' motivation to continue to learn about teaching methods. The results are displayed graphically in Chart 1. As expected, the scales for science attitudes changed more than attitudes towards math.

Italics denote scales where the direction of change or lack thereof was unexpected. Change was assessed with paired T-test at significance level of p < .05, N = 131.

Presents conclusions that address intended and unintended outcomes

Conclusions

• The summer academies had the desired effect of increasing positive attitudes towards the teaching profession, science, and toward using reform techniques in the classroom.
• The summer academies had the unexpected effect of ameliorating math anxiety, and increasing self-efficacy toward teaching math.
• The summer academies had the unexpected, albeit small, effect of increasing authoritarian attitudes in the classroom.

Summarizes results in table format

2.4. Did Pre-service and In-service Teachers Show Different Patterns of Attitudinal Change?

To evaluate whether pre-service and in-service teachers differed in their experience of the summer academies, we examined differences by service status of the participants (high school student, college student, and in-service teacher). In general, high school students and in-service teachers both changed in desired ways and in line with overall results. College students showed less positive attitudinal change, and showed more change in unexpected directions. In most cases, high school students showed the greatest magnitude of change, and college students showed the least change. These findings are shown in Tables 2.5 and 2.6. Group sizes were similar for all three groups.

Italics denote scales where the direction of change or lack thereof was unexpected.

Italics denote scales where the direction of change or lack thereof was unexpected.

(…)

Interprets results in context of project

Standardized change scores are given for each service status group in Table 2.8, and the scores are plotted in Chart 2. From the chart it is clear that the High School group showed relatively large increases on the majority of attitudinal measures, especially learning motivation, and self-efficacy measures. Most importantly, the High School group showed the largest increase in positive attitudes towards the teaching profession. This finding was comforting in light of the fact that one of the primary goals of the summer academies was to recruit new teachers.

Presents conclusions and speculates on causation

The college students showed some change in the desired direction. They increased on self-efficacy attitudes toward inquiry-based teaching and the field of science. They also decreased on science and math anxiety, and did not increase on the Pupil Control Ideology scale. College students did not show increasing positive attitudes toward the teaching profession, learning motivation, hands-on self-efficacy, or science outcome, and they showed a decrease in attitudes toward inquiry-based teaching. The mixed results for the college students could be the result of several factors. One likely factor is that these students were already exposed to many of the reforms being presented in the summer academies in their current curriculum in college. Thus, these students may already have relatively positive attitudes toward many of O-TEC related goals.

Describes intended and unintended outcomes

The in-service group showed many of the changes that we expected. They increased in their feelings of self-efficacy toward using hands-on and inquiry-based teaching techniques and in their feelings that good teaching will lead students to understand science better (Science Outcome). The in-service group also showed marked decreases in science and math anxiety. Interestingly, the in-service group showed the largest increase in authoritarian attitudes in the classroom. We have no explanation for this finding and will pursue a replication of these results during the next round of summer academies.

Lists conclusions from findings

The analysis of differences across service status suggest:

• The summer academies had the largest influence on changing attitudes toward the teaching profession in the most important group-high school students.
• It appears that the summer academies had the greatest influence on high-school pre-service teachers and in-service teachers, and had mixed effects on college students.
• In-service teachers showed increases on many of the desired outcomes and an unexpected increase in authoritarianism.

(…)

Interpretations & Conclusions:
Lists conclusions

2.9. Overall Conclusions and Recommendations for Summer Academies

• The experience of the summer academies appeared to have strong effects on participants. Most participants showed increases in their confidence to employ reform teaching techniques and decreases in science and math anxiety.
• High school participants showed the greatest increase in attitudes toward teaching. Future summer academies may want to focus more of their efforts on high school participants, as this is where the greatest 'recruitment' effect was found.
• There were no significant differences between the experience of men and women. There were no racial differences on quantitative indices. Follow-up qualitative interviews revealed no racial issues or bias in the delivery of the summer academy topics.
• Although some sites showed less change than others did, the overall pattern of change was consistent with ideal expectations.
• Participants were enthusiastically positive about their experience in the summer academies.

Excerpt 9 [Maryland Collaborative]

Interpretations & Conclusions:
Reiterates evaluation question

Research question 1: Is there a difference between the MCTP teacher candidates' and the non-MCTP teacher candidates' attitudes and beliefs about mathematics and science?

Describes quantitative analysis and data

Describes sample

Interprets results

Sections of our 45 item survey instrument that were verified by factor analysis dealt with beliefs about mathematics and science (=.76); attitudes toward mathematics and science (=.81); beliefs about teaching mathematics and science (=.69); attitudes toward learning to teach mathematics and science (=.79); and attitudes toward teaching mathematics and science (=.60). Data were obtained (total instrument responses, N=931; teacher candidates, n=609; MCTP teacher candidates, n=286) during the 1995/96 academic year from 33 reform-based mathematics, science, or pedagogy undergraduate college classes taught in 7 higher education institutions in Maryland. A key finding is that there are differences between the attitudes and beliefs of MCTP and non-MCTP teacher candidates as they initially encounter reform-based instruction in their teacher preparation classes. The evidence suggests that non-MCTP teacher candidates in MCTP classes become less enthusiastic about taking more college courses in mathematics and science, and less enthusiastic about learning how to use technology to teach these subjects.

Presents generalizable conclusion

The attitudes and beliefs of the MCTP teacher candidates do not experience this negative impact. This finding contributes to a better understanding of the struggles reform minded college-level science and mathematics content and pedagogical specialists face in instituting curricular transformation in teacher education programs.

Reiterates evaluation question

Research question 2: Do MCTP teacher candidates' attitudes toward and beliefs about mathematics and science change over time as they participate in the