Teacher Education Annotated Plan Excerpts

Excerpt 1 [Los Angeles Collaborative]

Evaluation Purposes

Major Evaluation Activities

During Year One of the evaluation, major emphasis will be placed on refining the evaluation design and baseline data against which to evaluate program outcomes during Years Two through Five. In addition, rigorous efforts will be made to provide on-going feedback to LACTE during the course of the evaluation so that program adjustments may be made as needed.

Overall, ETl's evaluation seeks to utilize both qualitative and quantitative data to determine the extent to which LACTE succeeds in meeting its major objectives during the five-year course of the project. Specifically, the evaluation is designed to determine to what extent is the project effective in:

• Student recruitment and retention, including underrepresented students;
• Faculty development, including opportunities and outcomes;
• Student internships and other experiential learning; and
• Support networks for students and faculty, both across and between campuses?

The major activities that will occur in each of the five years of the evaluation are:

• Review and revise evaluation research questions;
• Conduct on-site visits, focus groups and interviews with program staff;
• Review program documents and records;
• Design and distribute written surveys for program participants; and
• Report findings, comparing project outcomes against baseline data.

The evaluation is designed to fully incorporate the collaborative nature of the program, with participation of students, faculty, administrators, master teachers, mentors, employers, and others.

Excerpt 2 [Philadelphia Collaborative]

Stakeholder Involvement

Formative evaluation will be a collaborative effort among representatives from the Advisory Committee, the Joint Operations Committee, and faculty and students from CCP and TU. Summative evaluation will be performed by an outside evaluator.

Evaluation Purposes

The measurable objectives of the Project's goals are as follows:

Relates project goals to evaluation procedures

Goal I: To create a community of scholars dedicated to consistent implementation of recommendations for change in the way mathematics and science are taught.

• The expanded list of participants will show that the community of scholars will have grown beyond the original group, incorporating additional persons from each of the constituencies.
• Surveys and focus groups will show that the community of scholars will perceive by the end of the project both the reality of and the significance of their efforts to effect change.
• The reports of the Joint Operations Committee regarding shared curriculum developments at CCP and TU, Teacher Enhancement Seminars in the PDS's, the Action Research Lab, the Teaching Improvement Center, and the Math and Science Resource Center will show the extent of the inter-agency collaboration critical to the long-term success of the project.

Goals II-IV: To develop, evaluate, institutionalize, and disseminate a teacher preparation model, particularly for elementary and middle school teacher training.

• The teacher graduates of the program will exhibit a better grasp of the nature of mathematics and science and will exhibit stronger teaching competencies than students in existing teacher preparation programs.
• The program will be accepted by the Pennsylvania Department of Education as a model for the preparation of middle school teachers.
• Research findings and descriptions of the program and the new and revised courses will be presented in colloquia and conferences and will be published to disseminate the results of the program and help develop a national model for teaching mathematics and science.

Excerpt 3 [Los Angeles Collaborative]

Stakeholder Involvement

The evaluation research questions will drive the study. ETI proposes to work closely with LACTE in developing the appropriate questions for the evaluation.

Evaluation Questions

Year 1 : ETI anticipates that the research questions will include questions such as:

- To what extent were faculty workshops effective in presenting innovations in science and math courses? Which components of the workshops were found to be most effective? Least effective?

- How were successful teacher recruitment strategies identified? What strategies and procedures were developed to identify and recruit math, science and liberal arts students?

- Which recruiting strategies and procedures worked most effectively for different types of students (based on major, ethnicity, etc.)? What are the barriers to effectively recruiting minority students? What are effective strategies for overcoming these barriers?

(…)

Evaluation Questions:
Specifies provisions for reassessment of evaluation questions

Year 2 : Refine evaluation research questions. ETI anticipates that the research questions will include questions such as:

- How effectively did science and education faculty work together in planning and teaching courses? What contributed to effective working relationships? What were barriers? How were barriers overcome?

- To what extent was the workshop teacher training effective? What indicators exist of shared techniques and goals across campuses? between campuses?

- To what extent were faculty and administrators willing to work together to provide professional recognition and rewards? Were rewards viewed as meaningful by teaching faculty? Were institutional changes implemented that included promotions and merit salary increases? Were these rewards effectively conveyed to the faculty?

(…)

Year 3: ETI anticipates working closely with LACTE to determine the project's evaluation priorities for assessing the pilot courses and faculty development component: ETI anticipates that the research questions will include questions such as:

- Was the Faculty Development component effectively planned, staffed, implemented? What indicators are there of overall program effectiveness? How were faculty recruited to the program? What factors contributed to their retention in the program?

- How were the faculty development leaders identified? Trained? To what degree were they effective in their roles? What evidence exists of their effectiveness?

- To what extent were faculty development workshops effective in reaching the target audience? How were faculty recruited to the workshops? Did faculty members benefit from the workshops? Was this reflected in students' classroom experiences?

- How did the students experience the teaching internships? Were "employers" pleased with the preparation given the students? How much of the students' teacher training was directly employed in their intern experience?

- How effectively were courses transported to the other institutions? What differences, if any, were observed in the implementation of courses to different student populations?

(…)

Year 4: Again, appropriate evaluation research questions will drive all evaluation activities: the Evaluation research questions may include questions such as:

- What opportunities were developed for student internships? How were students recruited to participate in these opportunities? Did the opportunities represent the diverse interests of students? Were appropriate mentors identified at these sites?

- What roles did master teachers play in mentoring students? In what ways did master teachers enrich students' learning? What types of learning experiences did master teachers provide for students? What were the characteristics of the most successful master teacher-student relationships?

- What additional professional development opportunities were identified and utilized? What type of follow-up training was viewed as needed? was viewed as beneficial?

(…)

Year 5: ETI staff anticipate that the evaluation research questions will include questions such as:

- To what extent was a project manual developed that would allow other campuses to duplicate LACTE's activities? To what extent were evaluation findings reflected in the manual so as to assist other projects? How was the manual publicized? disseminated?

- To what extent were teleconference videos developed that would assist other campuses to implement a teacher excellence program? How were the videos publicized? disseminated? How were these videos evaluated by other academics? To what extent was the teleconference series effective? What indicators exist of its effectiveness?

- To what extent did LACTE participating faculty and administrators utilize state, regional and national meetings of professional organizations to provide information on LACTE? To what extent did these meetings lead to increased interest in the project? What indicators are there that state, regional and national meetings are effective dissemination outlets?

- What additional dissemination outlets were identified? How were these used? What impact did they have?

Relates evaluation questions to project goals.

- Overall, to what extent did LACTE succeed in meeting major objectives during the five-year course of the project? Including, to what extent was the project effective in:
   - Student recruitment and retention, including underrepresented students;
   - Faculty development, including opportunities and outcomes;
   - Student internships and other experiential learning; and
   - Support networks for students and faculty, both across and between campuses?

Excerpt 4 [Oregon Collaborative]

Evaluation Purposes

The basic purpose of the OCEPT evaluation is to provide information about the extent to which the five year project has made a positive contribution to the quality of the preparation of math and science teachers in K-12 in Oregon and to document what we have learned in the process. While some significant progress is expected over these next five years, more time will be required to gauge the full impact on K-12 teacher educators; that is, more time will be needed before sufficient numbers of newly prepared teachers will have entered and completed their undergraduate and teacher education experiences and entered our K-12 schools.

Nonetheless, the evaluation will provide important information about the experiences of our students even within this short time-frame. Information will be gathered about changes that can reasonably be expected to occur within the five year project time frame and which themselves individually and collectively may be expected to have a longer term impact on the quality of teacher preparation in math and science (for instance, change in faculty classroom practices, faculty beliefs, curriculum, communication patterns and networks, and student recruitment strategies).

Stakeholder Involvement

Evaluation Purposes

An additional purpose of the evaluation is to provide annual information to the Statewide Advisory Council and the management team about project progress. These formative evaluation activities will focus on how well various project-related activities are being implemented and are operating and will provide feedback to the collaborative participants about how its activities may need to be adjusted.

Finally, as resources will permit, the evaluation will involve a series of special studies designed to address select issues of interest to the collaborative.

Evaluation Questions:
Specifies questions for formative evaluation

The two major formative evaluation questions are:

1. How well is the program being implemented and functioning? As planned? On target? Changes needed?
2. And in particular, how well is it developing as a Statewide collaborative and what changes, if any, are needed, to improve this developmental process?

Evaluation Questions:
Specifies questions for summative evaluation

The three major summative evaluation questions are:

1. Has the project achieved its stated goals and objectives?
2. Has the project contributed to important changes in how we are preparing future teachers of mathematics and science?
3. What have we learned about what works and what does not work in attempting to develop a statewide collaborative that will sustain over time and is aimed at continuing to improve how we prepare future K-12 math and science teachers?

Excerpt 5 [Montana Collaborative]

Evaluation Purposes:
Relates evaluation purposes to project components

The purposes of the evaluation of the STEP project are to (1) document the processes used by the project, (2) provide formative feedback to help the project accomplish its objectives, and (3) support long-term institutionalization of project initiatives. The framework of the evaluation is built on three primary components of the project: Teacher Preparation, Support for Underrepresented Groups, and Project Management.

Evaluation Questions:
Relates evaluation questions to project components

There are three main activities involved in the teacher preparation component: course revision, model schools, and early career support. Course revision work, along with planning and identification of model schools took place in year one. These two activities continue in year two, as well as planning for the Early Career Support component. All three components will be in place during year three and beyond.

I. Teacher Preparation
  A. Course Revision

1. What is the nature of team approach?
2. What are the characteristics of professional development activities?
3. What sorts of courses are emerging?
4. What are the site specific issues?
5. What are the technology needs/accomplishments?
6. How is coordination & sequencing accomplished among/between mathematics and science courses and with other education courses.
7. How many students are enrolled in STEP courses?
8. How many faculty members, teaching assistants, and K-12 teachers are involved in course revision teams?
9. How many teachers and administrators are involved in Model Site programs?
10. How do the courses fit into the program of study in Teacher Education?
11. How do the courses evolve and become institutionalized?
12. How does course development change over the life of the project?
13. Have courses or components of them been adopted at other colleges?

(…)

  C. Early Career Support

1. What are the characteristics of the connection of the mentor program with the universities, model school sites, Montana Science Teachers Association and the Montana Council of Teachers of Mathematics?
2. What processes are used to improve the early career teacher experience?
3. How well does the communication among university personnel, mentor teachers, and early career teachers function?
4. What is the nature or effectiveness of the mentoring process?
5. What is the nature of early career acculturation into teaching?
6. How many mentor teachers, early career teachers and administrators are involved in the early career support program?
7. What multiplier effects have taken place?
8. What factors effect the retention of early career teachers at rural sites?
9. How have telecommunications systems been used in the mentoring process?

Excerpt 6 [Boston/Cambridge Collaborative]

Evaluation Questions

• What characteristics of each institution enhance or impede all those with vested interests in math/science/teacher education reform, i.e., math/science faculties, math/science education faculties, education faculties, and field experience faculties?
• What are the differences in philosophical stances and outcome objectives of the four IHE's which either enable or prevent TEAMS-BC as an entity from effecting reform in math/science teacher education?
• How might factors that impede collaboration be minimized? How can those that enhance it be further supported?

Excerpt 7 [Maryland Collaborative]

Evaluation Questions:
Specifies preliminary set of questions that will be re-examined as data are collected

The following questions serve as the a priori research questions (a posteriori questions will emerge throughout the research period):

1. What is the nature of the faculty and teacher candidates' beliefs and attitudes concerning the nature of mathematics and science, the interdisciplinary teaching and learning of mathematics and science to diverse groups (both on the higher education and upper elementary and middle level), and the use of technology in teaching and learning mathematics and science?
2. Do the faculty and teacher candidates perceive the instruction in the MCTP as responsive to prior knowledge, addressing conceptual change, establishing connections among disciplines, incorporating technology, promoting reflection on changes in thinking, stressing logic and fundamental principles as opposed to memorization of unconnected facts, and modeling the kind of teaching/learning they would like to see on the upper elementary, middle level?

Evaluation Questions:
Specifies general research questions related to the evaluation

Answers to those questions will address the following global research questions driving teacher education research:

1. How do teacher candidates construct the various facets of their knowledge bases?
2. What nature of teacher knowledge is requisite for effective teaching in a variety of contexts?
3. What specific analogies, metaphors, pitfalls, examples, demonstrations, and anecdotes should be taught content/method professors so that teacher candidates have some knowledge to associate with specific content topics?

Excerpt 8 [Oklahoma Collaborative]

Evaluation Purposes

Several O-TEC projects will be evaluated through longitudinal studies on the effects of their initiatives as students/teachers move through the program. Of particular concern are:

Evaluation Questions

Generalizes to other contexts

1. How do talented high school students and undergraduates with interests in science and mathematics respond to experiences with teaching?
2. What learning experiences provide students and teachers with skills of analysis and inquiry while pursuing an understanding of the nature of science?
3. What are the effects of redesigned science and mathematics courses for pre-service training on teacher performance?
4. How do field experiences that integrate theory and practice impact the performance of the pre-service teacher?
5. How do experiences that promote collegiality and professional development among entry-year teachers promote retention and leadership?
6. What are the effects in mathematics and science of linkages among faculty in content areas and pedagogy within the institution?

Relates evaluation purposes to broader needs

These topics have ramifications of possible national significance, and are well-suited for graduate student research projects.

Excerpt 9 [Los Angeles Collaborative]

Evaluator Credibility

The Evaluation and Training Institute (ETI) proposes to undertake the five-year evaluation for LACTE. ETI has had more than 21 years experience working with both secondary and postsecondary institutions conducting evaluations in: underrepresented student recruitment; teacher development; curriculum reform; math and science education programs; longitudinal studies; inter-segmental collaborations; and assessing performance outcomes against baseline data. ETI has conducted numerous studies for both The California State University and the Chancellor's Office of the California Community Colleges, in addition to evaluations for private California universities.

Specifically, ETI is conducting CSU's evaluation of the California efforts between secondary and postsecondary institutions in order to improve the academic preparation of these students for higher education. In addition, ETI previously evaluated four CSU programs whose objectives were the recruitment and retention of underrepresented student in postsecondary education. These included CSU's High School Outreach; Retention Incentive; Faculty Mentoring; and Faculty/Student Mentoring Programs.

Other examples of ETl's extensive evaluation experience in the areas relevant to LACTE include:

• Current work with KCET and the Los Angeles Educational Partnership in assessing the impact of the Math, Science and Technology Professional Development Series on teachers;
• A recently completed evaluation of the California Academy of Physical Sciences program aimed at improving health and physical sciences education in California;
• An award-winning, national study for the National Science Foundation of academic employment patterns and trends of women students and faculty in science and engineering; and
• A current evaluation of Dow Chemical's US and Canadian Hands-On Science Program for K-6 students and teachers, including extensive teacher development and curriculum components.

Excerpt 10 [Philadelphia Collaborative]

Stakeholder Involvement

Formative evaluation will be a collaborative effort among representatives from the Advisory Committee, the Joint Operations Committee, and faculty and students from CCP and TU.

Excerpt 11 [Louisiana Collaborative]

Stakeholder Involvement

The strategic plan for the Collaborative will include a key component devoted to evaluation. The LaSIP (Louisiana Systematic Initiatives Program) Collaborative Assessment and Evaluation Panel, the Associate Director for Testing and Evaluation, along with other staff members and advisors, will assume evaluatory responsibility for the Collaborative. Any evaluation of LaSIP will include the Collaborative; any evaluation of the Collaboration will include LaSIP.

Excerpt 12 [Oklahoma Collaborative]

Stakeholder Involvement

An Advisory Board, composed of distinguished science and mathematics educators from across the country, will complete an annual in-depth review of O-TEC activities and provide guidance on future directions. The members of the Advisory Board will be chosen in consultation with the cognizant NSF program officer. Upon a positive funding decision, O-TEC will constitute the Board with representatives from exemplary programs (like Dean Ioannis Miaoulis of Tufts University), distinguished science and mathematics educators (like Dr. William Bush of the University of Kentucky and Dr. Dorothy Gabel of Indiana University), a national leader in two-year college education (like Dr. David Pierce, Past President of the American Association of Community Colleges), a distinguished classroom teacher, and an individual with expertise in science and mathematics education for Native Americans (like Dr. Stuart Tonomah). The Advisory Board will meet annually to review progress, evaluate the success of individual programs, provide direction in extending programs, and aid in dissemination of information nationally.

Excerpt 13 [Los Angeles Collaborative]

Evaluation Purposes:
Describes intent to document positive and negative outcomes

Based on the review of program records, ETI will develop findings and recommendations as to effective and ineffective recruitment and retention strategies for students, faculty, and master teachers.

Excerpt 14 [Philadelphia Collaborative]

Evaluation Purposes:
Describes intent of analysis

Describes intended utilization of results

After completion of the summative evaluation of the new courses, we will describe them and the evaluation results in publications, and in talks and workshops at national and regional meetings. One focus of our analysis and description of the courses is how changing the learning environment in science and math classrooms leads to students being more interested in learning and, consequently, in teaching science and math. Our program will result in the reform of approximately 74 courses in 5 disciplines and in 8 quite different colleges. The total annual enrollment in those courses will be in the thousands since large introductory courses will be modified. The number of science and math majors exposed to teaching experiences will also be large, 100 or more annually.

Describes intent to generalize from results

Findings about getting students interested in learning and teaching from this extraordinarily rich base will be transferable to a wide range of institutions. The evaluation will also help us understand which kinds of activities with schools and children draw college students into teaching careers. Educators will be interested to know whether any of these exposures is especially effective or if it is the variety of interactions with children that increases the population of college students interested in teaching. We are especially interested in the contributions made by the mentor teachers and the workshops for the participating students.

Describes anticipated results

We expect to see a change in the cultures of the eight participating colleges. By changing science and mathematics courses and programs so that they are more welcoming to prospective teachers, they will be more welcoming to all undergraduates. The result would be that science and mathematics will be seen as worthwhile majors for all undergraduates. This will help us to reach "the second tier" (Tobias, 1990).

The project will have its greatest impact on the pre-service education of elementary teachers and of secondary science and math teachers. The elementary teachers will have a sounder base in these disciplines, and will be prepared to teach them using the pedagogical methods incorporated into the new courses. Science and math majors who become secondary teachers will have profited from experiencing the new pedagogies. The improved pedagogy and the recruiting efforts will produce significant increases in the number of women and minorities entering science and math teaching; CETP scholarships will allow 40 minority students to complete science and math teaching programs. The project will also improve the teaching of science and math in Western Massachusetts, since teachers who participate in the curriculum development committees and as mentors for undergraduates will increase their content and pedagogical knowledge. The use of educational technology in its many forms is a central part of STEMTEC. The infusion of educational technology into the new and revised courses as well as the educational technology courses will be adaptable to other campuses. Intelligent tutoring systems will be developed by STEMTEC and made available for national distribution.

Excerpt 15 [Oregon Collaborative]

Stakeholder Involvement:
Describes the perspectives of project planners about project goals

Describes comparisons to best practice

The project planners hold the view that there are many different and equally effective ways to teach math and science, both at the higher education level and in our schools. The project is not about attempting to have more individuals adopt one set of "best" teaching practices. Planners do believe, however, that the incorporation of particular perspectives and approaches to curriculum and instruction can make a difference in the quality of student learning and teacher preparation. These include finding ways to know your students; using classroom research as a way to understand more about student learning; consulting more with colleagues about teaching and curricular approaches; developing a richer array of instructional strategies and curricular approaches that can be used when they seem most appropriate; increasing the knowledge base about national and State standards; and so forth.