Described below is a scenario for using the evaluation
plan resources. For more information on these and other resources,
see the overview for the user scenarios,
as well as the frequently asked questions about
plans.
| Activity |
Description |
|
Goal: Planning the evaluation of a systemic
mathematics course reform project
|
Professor Katherine S. plans to
submit a proposal to EHR for redesign of Drake University's
freshman mathematics courses. She meets with Dr. Craig J.,
the external evaluator, to confer on the details of the evaluation
plan. Dr. J. has reviewed the description of the project's
goals and the strategies for revamping the mathematics course
series and integrating the courses with the community college
curriculum. He has drafted some evaluation questions he has
derived from the project goals.
|
|
Go to the OERL Web site
|
Together, they go to the
OERL website to browse through the evaluation plan
resources that have been used in other EHR curriculum
reform projects. First, they examine the introduction to the
library on the OERL home page to identify which resources
might be most useful.
|
|
Review criteria for sound evaluation plans
|
They click on the Plans
tab to locate the planning resources.
They review the criteria for sound evaluation
plans that overview the major components of a
sound EHR evaluation plan.
|
|
Goal: Determine evaluation purposes
Question: How do evaluation
plans for similar EHR projects specify their purposes
and questions?
Scroll through Evalation Overview plan excerpts for
Curriculum Development projects
|
Because Professor S. and Dr. J.'s
project focuses on the reform of mathematics curriculum, they
click on the Curriculum Development tab on the
left hand side of the page. To get an
idea how plans describe and organize the purposes of
the evaluation, they click on Evaluation Overview
under Annotated Excerpts and scroll through the brief
plan excerpts, looking for evaluation
purposes.
Dr. J. notes that one of the plans specifies three
types of project outcomes to evaluate: effects on
students, quality of materials, and effectiveness of
instructional training. He and Professor
S. agree that these three areas should also be used to organize
their proposed evaluation. In another excerpt, Professor S.
likes the questions specifying five effects of the courses
on students: continuation in mathematics, attitudes, basic
skills, problem solving, and the ability to reason and discuss
a concept in depth. They copy these
questions.
|
|
Question: What evaluation designs have been
used in other Curriculum Development projects?
|
A key issue for the evaluation
is the design that will be used to document project effectiveness.
Can the project provide evidence of value-added in relation
to previous approaches (pre/post) or in contrast to other
ongoing approaches (comparison groups)? Professor
S. and Dr. J. click on the plan excerpts for Design.
They discuss the advantages and feasibility of alternative
designs, while examining the designs used in EHR curriculum
development project evaluations in OERL. They keep
in mind their project goals, institutional contexts, and proposed
budget. They find that most of the designs in the plan excerpts
have contrasted the revamped, experimental courses with other
parallel, but differently designed sections of the courses.
They discuss the feasibility of this approach for their
evaluation.
|
|
Question:What kind of sampling plan should be
used?
|
There are a large number of students
enrolled in the reformed and unreformed mathematics courses
that will be compared in Professor S. and Dr. J.'s evaluation.
They turn to OERL for information on sampling. After reading
several excerpts describing the sampling plans used in different
evaluations, they elect to use random sampling
of students in the reformed and unreformed courses. The use
of a random sample will be cost-effective, since fewer students
will be tested and the generalizability of the evaluation
results will be increased.
|
|
Refer to plan excerpts to find a table that links data
collection procedures to outcomes
|
Professor S. and Dr. J. agree that
a combination of qualitative and quantitative methods will
be used to evaluate their project. To get some different ideas
about how to design the evaluation, they leave the Curriculum
Development section of OERL and examine plans from another
NSF program area, Teacher Education. To locate the
Teacher Education plans, they click on the Teacher Education
tab on the left hand side of the page. They then click on the link to
Teacher Education plans. Finally, they click on
Design under Annotated Excerpts. In the
design excerpts they find a table that efficiently identifies
the data collection procedures to be used for addressing evaluation
outcomes. They copy this table to adapt for their
plan.
|
|
Use a search to look at resources
|
They decide to look specifically
at plan excerpts for mathematics, so they click
on the Search tab in the upper right hand corner
of the page. The search page lists several
types of searches available. They click on
the Plans search. A form appears allowing
them to specify the features of the plans they are
searching for. They click on Plan Excerpts (as
opposed to Complete Plans) and projects in the
mathematics content area, choosing
"everything" for all the other features
available. Then they click the Search
button. A listing of search results appears
organized by Project Type. Each result describes the
content of the plan excerpt and displays information
about the project from which it comes as well as a
link to view the excerpt. They spend some time
reading through the
excerpts in turn and clicking the "Back"
button on the browser to get back to the
search results.
After viewing one excerpt
in particular, they get diverted and leave the page
with the excerpt on it. This time, instead of clicking
"Back" they click on the Search tab
again. Back at the search page, this time they
click on "most recent search"
and the search results they were viewing before
reappear, allowing them to continue perusing the plan
excerpts in the search results.
|
|
Scroll through sample instruments
|
To develop their data collection
procedures and timeline, Professor S. and Dr. J. decide to
look through the sample instruments included
in OERL for evaluating curriculum development
projects. To examine these instruments, they click
on the Instruments tab at the top of the page. After
getting to the section on Instruments, they click
on the link to Curriculum Development instruments.
They tentatively identify some instruments that have useful
sets of questions.
|
|
Skim complete plans
|
Professor S. and Dr. J. look over
the notes and materials they have gathered from the OERL plan
resources and decide to review some complete plans to see
how all the components have been integrated.
|
| Bookmark the OERL Web site |
They bookmark the OERL Web site on their
browsers, anticipating that they may browse
more of the resources once the evaluation plan is
drafted.
|
