This evaluation plan is embedded in a larger proposal
prepared by The City College of New York, Middle School
Science Leadership Consortium project.
-
Approaches to
evaluation: The evaluation proposed
for the Middle School Science Consortium will combine
monitoring, formative and summative evaluation. Monitoring
will be the strategy when the proposal states an intent
to "develop", "create’ or "maintain"
a particular aspect of the program. For example, for Goal
3 evaluation staff will monitor the establishment of "a
Consortium committee to recommend a plan of action"
or the creation and support of two District Science and
Technology Instructional Centers.
Considering the six specific project goals, monitoring
will be the mode for Goal 2 (on organizing a middle school
curriculum), goal 3 (on establishing the Consortium Committee
and Instructional Centers), Goal 4 (those aspects related
to staff development) and Goal 5 (on resource use and
continuation of project objectives). Evaluation will be
the mode for Goal I (on learning the content knowledge
base and teaching practices), for the leadership development
aspect of Goal 4, and for Goal 6 (the instructional technology
use).
Using annual pre-post data collections as described
below, the evaluation will generate data in the first
and last months of each academic year for the formative
evaluation. The analyses of the "pre" data will
be fed back to program staff immediately after analysis
to provide a basis for program revision and strengthening
for the remainder of the current academic year. Feedback
of the "post" data will also provide the basis
for the summative evaluation at the end of each year,
and the final summative evaluation after the third
year.
Return to Table of
Contents
-
Evaluation
design: The evaluation design will have
four components: 1) a pre-post-data collection for each
annual group of 60 teacher participants; 2) an ongoing post-only
survey of the teachers who visit the STICS; 3) follow-up
surveys of both groups of teachers in years two and three
of the program, and 4). analysis of objective tests data
to identify pupil response to the teachers. The objective
test data on science learning will be collected from children
in classes taught by the participants and from a sample
of comparable classes in the same schools. We will also
increase data collection through selected in-class observation
which will verify the extent to which, and the skill with
which, teachers have implemented the pedagogical
materials.
Return to Table of
Contents
-
Measures
required: To implement the design, seven
instruments will be required. For the teacher participants
evaluation staff will collaborate with program staff to
generate for self-awareness inventories needed. The inventories
involved are knowledge of: 1) the topics and constructs
in the content knowledge base" to be earned (Goal 1);
2) the desired teaching practices emphasizing the practices
program staff consider to reflect "inquiry methodology,
alternative teaching and learning styles reflective practices
and alternative assessment strategies (Goal 1); 3) the specifics
of "instructional technology" (Goal 6) 4) confidence
in ones "ability to teach science using nationally
validated curriculum, alternative teaching and learning
techniques and instructional technology:. (Goal 1) Content
tests and alternative assessment measures for teacher participants
and for students will be developed for the project by instructional
faculty and Consortium committees using available instruments
and assessment criteria. The rating scales to be used for
in-class observers and the external experts will be developed
by evaluation staff. Prototypes of the rating scales exist
from previous evaluations.
The evaluation will also require three surveys: 1)
for teachers who use the Instructional Centers through
which they can evaluate the experience: 2) for those involved
in the use of " electronic classroom exchanges"
to evaluate the extent to which those exchanges have in
fact been "fostered by this project; and 3) the follow-up
survey of use and value of program through review with
program staff and revision as necessary to assure content
validity.
Return to Table of
Contents
-
Sampling: No
sampling will be done of the sixty
direct teacher participants each year, so that data will
be available for all sixty. A time sampling plan will be
used to generate a 25% sample of teachers who visit the
STICS.
Return to Table of
Contents
-
Data collection, data analysis
and criteria for achieving objectives: Data
will be collected by program staff
who will transmit the data to the Evaluator for analysis
change and growth will be evaluated using each teachers
as his or her own control using appropriate tests significance
(i.e., ANOVA for the correlated means from contents tests)
using the 5% level of significance. Note that ANOVA will
also be used to compare learning by program and comparison
classes. The 5% level of significance will be used.
Achievement of gains in content knowledge for teachers
will be evaluated using analysis of variance and the 5%
level of significance. In addition, for teachers the objective
on content learning will only be considered to be achieved
if, in addition to statistically significant gains, at
least 85% reach a score of 80% on the post course
examination.
Changes in self-reports will be evaluated through
chi square, using the 5% level of significance. In addition,
these objectives will be considered to have been achieved
only if 75% of the participants report growth on two-thirds
of the items on the inventory involved.
Return to Table of
Contents
-
Report preparation:
The report provides five milestones
for understanding the impact of this project on teachers.
The first three for teachers come after the Fall semester
of each year, when the impact of the instructional process
on each group of 60 direct participants can be estimated.
The other two come after the second and third year when
the follow-up surveys will indicate the impact of the program
on in class functioning.
Return to Table of
Contents
