Described below is a scenario for using the evaluation
instrument resources. For more information on these and other resources,
see the overview for the user scenarios,
as well as the frequently asked questions about
instruments.
| Activity |
Description |
|
Goal: Preparing data collection instruments for
the evaluation of a Teacher Education project
|
Professor Robert R. directs a three-year
EHR project designed to improve the instructional approaches
of secondary and community college chemistry teachers. Some
of the data are available from the school and college records,
such as years of teaching experience, performance reviews,
awards, student reviews of instructors, and student grades
and course credits. Data about the effectiveness of the professional
development activities and new course materials will need
to be collected by using instruments developed for the project.
Professor R. meets with Dr. Paula Q., the external evaluator
for the project, to confer about the data that they need to
collect for the project.
|
|
Review framework linking data collection instruments to
evaluation questions
|
They refer to the data collection
framework Dr. Q. included in the evaluation plan of the EHR
proposal. The framework proposes types of data collection
for each of the evaluation questions. They plan to use two
types of instruments: a focus group interview for community
college professors and a student survey. They turn to the
OERL instrument resources to search for suitable
instruments.
|
|
Go to the OERL Web site
|
Together, they go to the OERL web
site to access the evaluation
instruments
that have been used in other Teacher Education projects.
They click on the Instruments tab.
|
|
Review criteria for quality of data collection
|
Since
Professor R. is less familiar with instrument selection and
development, they review the criteria for
instruments.
|
|
Question: What kinds of
instruments are used to evaluate similar EHR projects?
|
After reviewing the criteria,
Professor R. and Dr. Q. click on the Teacher
Education tab on the left hand side of the page,
and then on the link to Teacher
Education Instruments.
|
|
Examine the types of sample instruments for Teacher
Education projects
|
They look at the types
of instruments that the collection holds. They find
teacher/faculty instruments, student instruments,
case/study protocol instruments, and observation
instruments.
|
|
Question: What chemistry instruments are in
OERL?
Search for chemistry instruments in OERL
|
Because their project
focuses on chemistry, they are particularly interested in
instruments used in evaluating chemistry projects. They decide
to search the Web site for chemistry instruments, so
they click
on the Search tab in the upper right hand corner
of the page. The search page lists several
types of searches available. They click on
the Instruments search. A form appears allowing
them to specify the features of the instruments they are
searching for. They choose instruments from
projects in the chemistry content area,
choosing "everything" for all the other features
available. Then they click the Search
button. A listing of search results appears
organized by Project Type. Each result describes the
purpose, length, etc. of the instrument and displays
information about the project from which it comes
as well as a link to view the instrument. They
spend some time looking through the instruments
in turn and clicking the "Back"
button on the browser to get back to the
search results.
After viewing one instrument
in particular, they get diverted and leave the page
with the instrument on it. This time, instead of clicking
"Back" they click on the Search tab
again. Back at the search page, this time they
click on "most recent search"
and the search results they were viewing before
reappear, allowing them to continue perusing the
instruments in the search results.
|
|
Examine the OERL table of instruments, which overviews
the content and format of each type of instrument
Look for instruments that can be used or
adapted
|
After reviewing all the relevant
chemistry instruments, Professor R. and Dr. Q. decide to spend
more time examining the Teacher/Faculty instruments within
the Teacher Education area. Because community college professors
will be involved in their project, it is imperative that they
locate an instrument that has been designed to assess standards-based
science reform in a similar group. They click on the
links below the heading for Teacher/Faculty Instruments.
They compare the instruments used in the evaluations of
Teacher Education projects with the types of data specified
in their data collection framework.
|
|
Use particular questions and formats from OERL
instruments as components of new instruments
|
Within the Teacher/Faculty Interviews,
they find an instrument that has been used in focus groups
to capture instructors' changes in beliefs about standards-based
reform. This instrument matches their needs as specified in
the data collection framework, so they make a copy of the
focus group interview instrument.
|
|
Search for instruments that can be used or adapted
|
The pair now read through the OERL
overview matrix for each type of instrument to compare the
topics covered with the topics relevant to the evaluation
questions for their project. They had planned to use student
surveys, but not classroom observations or student assessments.
After examining the types of instruments available, they realize
that observations and student assessments would better address
their evaluation questions. Thus they add the two types of
instruments to their data collection framework and make
copies of the items and formats used in the observation and
assessment instruments that meet their needs. These
items and formats will be incorporated into newly developed
instruments.
|
|
Plan for review and pilot testing of instruments
|
Dr. Q., the external evaluator,
intends to pilot all the instruments to be used. She will
ask community college professors and students to review the
instruments and pilot test them. A brief description of the
pilot testing and the technical quality data for these instruments
will be reported in the evaluation.
|
| Bookmark the OERL Web site |
Professor R. and Dr. Q. bookmark the OERL Web site
on their browsers, anticipating that they may
browse more of the resources once the full set of
instruments is drafted.
|
