Evaluation Plan for the Oregon Collaborative
for Excellence in the Preparation of Teachers
(OCEPT)
This evaluation plan was prepared by Portland State
University, as a stand-alone proposal.
Table of Contents:
-
Introduction
- Evaluation Overview: Evaluation Purposes
- Design: Methodological Approach
-
Evaluation Approach
- Evaluation Overview: Evaluation Purposes
-
The Evaluative Questions
- Project Description: Project Features
- Evaluation Overview: Evaluation
Questions, Stakeholder Involvement
-
Assumptions
- Design: Methodological Approach
-
Major Evaluation Activities
- Design: Methodological Approach,
Information Sources & Sampling, Instruments,
Data Collection Procedures & Schedule
-
Specific Procedures
- Design: Methodological Approach,
Information Sources & Sampling, Instruments,
Data Collection Procedures & Schedule
-
Populations/Samples
- Design: Information Sources &
Sampling
-
Instrumentation
-
Staffing and Budget
- Evaluation Overview: Stakeholder
Involvement
- Design: Meta-Evaluation
- Reporting
Evaluation Plan for the Oregon Collaborative
for Excellence in the Preparation of Teachers
(OCEPT)
Return to Table of
Contents
The basic purpose of the OCEPT evaluation is to provide
information about the extent to which the five year project
has made a positive contribution to the quality of the preparation
of math and science teachers in K-12 in Oregon and to document
what we have learned in the process. While some significant
progress is expected over these next five years, more time
will be required to gauge the full impact on K-12 teacher
educators; that is, more time will be needed before sufficient
numbers of newly prepared teachers will have entered and completed
their undergraduate and teacher education experiences and
entered our K-12 schools.
Nonetheless, the evaluation will provide important information
about the experiences of our students even within this short
time-frame. Information will be gathered about changes that
can reasonably be expected to occur within the five year project
time frame and which themselves individually and collectively
may be expected to have a longer term impact on the quality
of teacher preparation in math and science (for instance,
change in faculty classroom practices, faculty beliefs, curriculum,
communication patterns and networks, and student recruitment
strategies).
An additional purpose of the evaluation is to provide annual
information to the Statewide Advisory Council and the management
team about project progress. These formative evaluation activities
will focus on how well various project-related activities
are being implemented and are operating and will provide feedback
to the collaborative participants about how its activities
may need to be adjusted.
Finally, as resources will permit, the evaluation will involve
a series of special studies designed to address select issues
of interest to the collaborative.
Return to Table of
Contents
The evaluation approach may best be described as eclectic.
It draws on features of objectives-, developmental-, and responsive-oriented
evaluations. The project has established several major goals
with a series of related objectives. Much of the evaluation
activity will be organized to gauge the extent to which the
objectives of the project are being advanced. At the same
time, however, and fundamental to the success of OCEPT, is
the extent to which it can develop and sustain as a collaborative,
where many individuals become involved in the effort, assume
ownership for the project and seek to involve others in the
process. The development of the project itself as a collaborative
(across institutional and disciplinary boundaries) will be
a major focus of the formative evaluation activities. Finally,
the evaluation will continue to be responsive to the interests
and concerns of the major local stakeholders and in particular,
the faculty involved with developing the collaborative.
Return to Table of
Contents
Beginning in February 1997 a series of meetings have been
held with Co-principal investigators, members of Mentor Teams
and a group of individuals who have expressed special interest
in and have expertise in evaluation and research. This latter
group has emerged into the "Research Team" for the project
and will continue to offer advice and counsel to the evaluation
activities, including instrument development and special studies.
Some may take major responsibility for a special study. These
meetings have helped to generate the key evaluative questions
for both the formative and summative aspects of the evaluation.
The two major formative evaluation questions are:
- "How well is the program being implemented and functioning?
As planned? On target? Changes needed?"
- "And in particular, how well is it developing as a Statewide
collaborative and what changes, if any, are needed, to improve
this developmental process?"
The three major summative evaluation questions are:
- Has the project achieved its stated goals and
objectives?
- Has the the project contributed to important changes
in how we are preparing future teachers of mathematics and
science?
- What have we learned about what works and what does not
work in attempting to develop a statewide collaborative
that will sustain over time and is aimed at continuing to
improve how we prepare future K-12 math and science
teachers?
The project proposes to make a difference in how we prepare
our future teachers by intervening in four major ways: undergraduate
math, science and engineering faculty and teacher education
faculty development; course development; coordinated efforts
to recruit and support greater numbers of underrepresented
students for the teaching profession; and, collaboration building
across educational sectors, institutions and
disciplines.
The more specific evaluation questions are detailed in Attachment
1 that lays out the data collection, analyses and reporting
plan for the project (coming soon!).
Return to Table of
Contents
Several assumptions will influence the evaluation. First,
those involved in the first year Summer Institute are viewed
as a kind of "vanguard" of the math and science educator leadership
and innovation in the State. They have been sought out and
selected precisely because they are such "standouts". They
are expected to play a role in initiating changes and/or continuing
to initiate changes at their own institutions and in Statewide
organizations and to serve as mentors to future Institute
participants. Pre-assessment profiles of those who participate
during this first year are expected to differ somewhat from
those who participate in subsequent years and possibly to
show less change on a number of characteristics over the five
years compared to their colleagues.
Second, we expect to see more change in faculty practices
and attitudes at institutions where Institute faculty continue
to be especially active and visible. At one planning meeting
of the Research Team we considered labeling our evaluation
design an "epidemiology model". Individuals participate in
the Institute, become "carriers" of new perspectives and practices
and transmit their "virus" to those in their local host culture.
An interesting sub-study will be one that seeks to understand
how new perspectives and practices take hold to a greater
extent in certain settings than in others. How is it that
some local environments or cultures are more open or resistant
than others to particular kinds of change?
Third, the project planners hold the view that there are
many different and equally effective ways to teach math and
science, both at the higher education level and in our schools.
The project is not about attempting to have more individuals
adopt one set of "best" teaching practices. Planners do believe,
however, that the incorporation of particular perspectives
and approaches to curriculum and instruction can make a difference
in the quality of student learning and teacher preparation.
These include finding ways to know your students; using classroom
research as a way to understand more about student learning;
consulting more with colleagues about teaching and curricular
approaches; developing a richer array of instructional strategies
and curricular approaches that can be used when they seem
most appropriate; increasing the knowledge base about national
and State standards; and so forth.
Fourth, we have not yet collected baseline information Statewide
from both faculty and students about current curricular and
instructional practices in math, science, engineering and
teacher education. As a result, until these data are analyzed
we are not able to project what kinds of and how much change
may be observable among faculty and students over the course
of the five years.
Fifth, and finally, during the course of the project, many
other potential sources of influence can be expected to affect
faculty practices and beliefs. Other federal and State-funded
projects are underway or will be implemented. Many individual
campuses have initiated faculty development programs through
new teaching and learning centers designed to introduce faculty
to new instructional approaches, uses of technology and classroom
assessment. The evaluators will be hard pressed to separate
out the influence of this particular project from the collective
impact of many other initiatives and developments. We will
make the effort, however, through the use of self-report data
from faculty and students, to identify what they perceive
has contributed most to various changes in their own practices
and perspectives.
Return to Table of
Contents
A multi-method, multi-audience approach will be adopted
that collects both qualitative and quantitative data and information
from students, faculty and administrators as well as from
existing records.
Three clusters of evaluation-related activities are planned.
Cluster 1 will involve the Evaluation Coordinator (EC), working
with the Principal Investigator (PI), the Project Coordinator
and the Co-PIs to provide NSF with the baseline, interim and
final year information they need for their evaluation. Cluster
2 will involve the Evaluation Coordinator, working with the
Research Team, to develop and implement the evaluation activities
needed to address the major formative and summative evaluation
questions of the project. And Cluster 3 will consist of a
series of special studies that are reviewed and approved by
the project leadership and the Research Team.
Key activities related to these three clusters of evaluation
activity are following:
Cluster 1--NSF Data Collection
1. collection of baseline and annual institutional and project
information for NSF, as required (includes profiles of student
participation in math and science teacher education related
programs and degree recipients; special surveys; and making
arrangements for site visits)
Cluster 2--Formative and Summative
Evaluation/Local
2. collection of baseline information from 1997 Summer Institute
participants and during 1997-98 from faculty in as many institutions
as possible who are not participating in the first Institute
(including information about instructional practices, conceptual
view of their field, assessment practices, knowledge and incorporation
of national and Statewide standards, view of their role as
faculty members, communication networks, technology use, beliefs
about teaching and learning, attitudes about teacher education,
professional development activities and perceptions of what
supports and inhibits desired changes in teaching and learning
practices)
3. collection of baseline information from samples of the
following groups of students: those entering a teacher education
program in 1997-98; those exiting a teacher education program
in 1997-98 (to include a review of select student portfolios);
and novice teachers in 1997-98 (in their first or second year
of teaching) whose primary undergraduate and teacher education
took place in Oregon (including information about their experiences
in math, science and teacher education classrooms, attitudes
about teaching math and science, views of selves as teachers,
beliefs about the teaching and learning process); give special
attention to students preparing to be elementary
teachers
4. for each of the above (1-3) collect the same information
during the fifth year of the project
Note: the baseline information collected in 1997-98 will
be used to assess overall changes during the life of the project;
and will also serve to increase awareness among faculty at
OCEPT institutions about current perceptions, practices and
beliefs that relate to project goals
5. an examination of the changes in curriculum over the
course of the project (to involve an examination of "work
samples" that describe the curriculum-in-use and interviews
with the faculty about their curriculum projects)(see NSF
Cooperative Agreement for project
target goals)
6. on-going surveys of students and faculty in OCEPT-developed
courses
7. during 1997-98 a special survey and select interviews
with staff, faculty and students involved with current programs
designed to recruit and support the entry into math and science
teacher education of those from underrepresented groups
8. annual follow-up surveys of and select interviews with
Faculty Fellows, Teachers-in-Residence and Mentor Team faculty
regarding their activities, accomplishments, concerns, observations
of departmental or instititutional developments, and perceptions
regarding the development of a collaborative for excellence
in teacher preparation
9. during the first, third and last year of the project,
interview select institutional administrators about the OCEPT
project and activities on their campuses
10. annual evaluation of the Summer Institute
11. annual collection by institution of a report that summarizes
activities, accomplishments, concerns and future plans related
to OCEPT project goals
12. annual collection of information that describes the
extent of interest in the OCEPT project and collaborative
process (requests to participate in the Summer Institute;
attendance at OCEPT-faculty or program sponsored events on
individual campuses or Statewide; existence of institutionally-based
leadership related to the OCEPT collaborative; etc.)
13. annual collection and monitoring of processes or programs
put in place that appear to hold promise for helping to sustain
the collaborative once the five-year grant ends
Cluster 3--Special Studies (under consideration;
contingent on resources)
14. a series of case studies of a 1997-98 cohort of students
of color and female students who are at various stages of
becoming teachers--those exiting a teacher education program,
those entering a teacher education program and those interested
in becoming teachers who are enrolled as undergraduates in
math and science courses
15. a series of 4 or 5 institutional case studies focused
on identifying what supports and what hinders individual,
departmental and institutional change in the directions supported
by the OCEPT collaborative (a doctoral dissertation)
16. case studies of a sample of individual faculty who participate
in the 2nd and 3rd Summer Institutes
17. a special study of misconceptions and critical barriers
to student learning of concepts and processes in mathematics
and science and the identification of ways of addressing these
(doctoral dissertation)
Return to Table of
Contents
Attachment 1 provides an overview of the evaluation plan,
including evaluation questions, respondents, data collection
approaches, schedule, who is responsible and deadlines. This
section provides a summary of the major formative and summative
evaluation procedures for the project and also proposes a
series of special studies which are contingent on the availability
of resources.
Formative evaluation
- annual evaluation of the Summer Institute
- late Fall surveys of Faculty Fellows & entor Team members;
and focus group of Teacher's in Residence about
implementation
- end of year surveys of Faculty Fellows, Teachers-in Residence
and Mentor Team members about activities, accomplishments,
concerns and progress toward developing a
collaborative
- mid-year interviews with Co-PIs about project activities,
accomplishments, concerns and progress toward developing
a collaborative
- mid-year interviews of those involved with diversity-related
initiatives
- end of year review of institutional reports and summarizing
of these reports during the summer regarding progress on
project goals and includes information on the kinds of project-related
course or program revisions made (new courses, revised courses,
courses dropped)
- annual summary of various indicators of interest levels
in OCEPT-related activities
- first, third and final year interviews with institutional
administrators about project progress
Summative evaluation
- a comparison of survey data collected each summer from
Institute participants with survey data collected from the
same individuals during the Winter Term of the last year
of the project
- a comparison of survey data collected Fall 1997 from
samples of non-Institute faculty from different institutions
with survey data collected from the same individuals during
Winter Term of the last year of the project; and a comparison
of first-year and last-year data collected from Institute
and non-Institute faculty
- a comparison of survey data collected during the first
and last year of the project from a sample of undergraduates
enrolled in math and science courses, samples of students
entering teacher education programs, samples of students
exiting teacher education programs and samples of novice
teachers (those in their first or second year of
teaching)
- a comparison of target project goals (see Cooperative
Agreement) with actual achieved goals, annually and at the
end of the project
Special studies
- 4 or 5 case studies of institutional change--what supports
or hinders change efforts?
- individual case studies of students of color and women
in various stages of interest in and preparation for a future
career in teaching
- individual case studies of a sample of faculty who teach
undergraduate courses in math and science and who participate
in the second and third Summer Institutes
- a study of misconceptions and critical barriers to student
learning of important concepts in math and science and what
approaches hold promise for lowering these barriers
- a closer examination of curricular materials and instructional
approaches that may hold promise for wider
dissemination
Return to Table of
Contents
Five different groups will be asked to provide evaluative
information over the course of the project: faculty, students,
administrators, Co-PIs and those involved with sponsoring
special programs designed to recruit students of color and
women into the fields of math and science as well as teacher
education.
Faculty
- Institute faculty--50 a year; annual baseline survey
data; end of year surveys; select interviews as part of
special studies
- OCEPT-course faculty; on-going surveys of classroom practices
- Non-Institute faculty--first year survey of as many faculty
as possible in math, science and teacher education departments/Schools
in the 34 participating institutions (n=??)
- Case study faculty--a subset of those who participate
in the Year 2 and Year 3 Institute; select 10 each year
Students
- on-going OCEPT course surveys; about course
experiences
- sample of lower division students enrolled in select
math and science courses-- select a representative sample
of 10 OCEPT schools and at each, survey a minimum of 100
students during the 1st and last years of the project
- sample of students entering elementary education programs--select
a representative sample of 10 OCEPT schools, 5 with undergraduate
teacher education programs and 5 with graduate-level programs;
and at each, survey a minimum of 30 students during the
first and last years of the project
- sample of students entering secondary education programs
in math and science-- select a representative sample of
10 OCEPT schools and survey as many students as possible
during the first and last year of the project
- sample of students exiting elementary education programs
during the first and last years of the project--survey from
the same 10 OCEPT institutions as above
- sample of students exiting secondary education programs
during the first and last years of the project--survey from
the same 10 OCEPT institutions as above
- sample of novice teachers (in 1st or 2nd year of teaching)--survey
a minimum of 50 teachers in elementary education and 50
in secondary education whose undergraduate education and/or
teacher education was completed in Oregon;
Co-PIs
- mid-year interviews about project progress
Special Program Faculty and Staff
- first, third and fifth year interviews with key faculty
and staff involved with programs designed to recruit and
retain students of color and women in math and science fields
and teacher education
Administrators
- first, third and fifth year interviews with select institutional
administrators about their expectations for the project,
progress made and continuing support for sustaining the
Collaboratives activities
Return to Table of
Contents
The following data collection instruments will need to be
developed:
- survey for use in collecting baseline and end of project
information from faculty (Institute and
non-Institute)
- surveys for use with OCEPT courses-student and faculty
forms
- survey for use with the 6 student sample groups
- interview protocols for use with students, faculty and
staff involved with diversity-related initiatives
- interview protocol for use with Co-PIs
- survey for use with annual follow-up of Institute
participants
- institutional information reporting forms for use by
Co-PIs annually
- identification of indicators for monitoring the level
of interest and participation in OCEPT-sponosored activities
(a records keeping process)
Return to Table of
Contents
The evaluation coordinator will have overall responsibility
for managing the evaluation activities. Various Research Team
members will assist with survey and interview protocol development
and pilot-testing. Some may take on a special project. Co-PIs
will be involved in some interviewing activity.
During the first year of the project two .15 FTE graduate
assistants will be needed to assist with faculty and student
data collection and analyses activities. One .15 FTE graduate
assistant will be needed during years 2, 3 and 4. During the
last year of the project, one .30 FTE and one .15 FTE graduate
assistant will be needed to assist with final year data collection,
analyses and reporting. Both the Project Coordinator and the
Project Administrator will also provide support to the evaluation
activities. The evaluation plan must now be reviewed in terms
of resource availability and adjustments made where
needed.
Information about the project and the evaluation and research
activity opportunities will be circulated to faculty who work
with Master's and doctoral students in education. Where appropriate,
the project would like to encourage graduate students to take
on various special studies related to the goals of the
project.
Return to Table of
Contents
The reporting schedule and audiences for the various reports
are identified in Appendix 1 (coming soon).
|
