Described below is a scenario for using the
evaluation report resources. For more information on
these and other resources, see the overview
for the user scenarios, as well as the frequently
asked questions about reports.
| Activity |
Description |
|
Goal: Evaluation reporting
for a Teacher Education project
|
Professor Alice J. is an external
evaluator for a Teacher Education project funded by EHR.
The project, which provided professional development opportunities
to approximately 200 undergraduate physics faculty, has completed
its final year. Consisting of a series of workshops held during
the academic year and summer institutes, the project provided
physics faculty with new pedagogical and technical skills
to teach introductory physics courses online. Professor J.
is preparing to write the evaluation report and is turning
to OERL to clarify several concerns. Her first concern is
what types of information she should present in describing
the project.
|
|
Go to the OERL Web site
|
To gather information on project
descriptions, she goes to the OERL web
site. |
|
Review criteria
for sound evaluation reports
|
She clicks on the Reports tab at the top of the
page. She begins by examining the criteria
for sound evaluation reports that focus on project
description/documentation. After she reviews the criteria,
she clicks on the report excerpts for Teacher
Education programs.
|
|
Scroll through report excerpts for Teacher Education
Project Description
|
Professor J. scrolls through
all the report excerpts listed in the Program Description
component and notes that many include a list of project goals
or objectives and short descriptions of project components
or activities. Only a few contain context analyses that document
demographic characteristics of participating teachers and
the educational context in which they work. Because the project
she is evaluating has to do with increasing the technological
skills of faculty, she believes it is important to document
the educational and technological resources that were available
to participants.
|
|
Question: How do reports include program
descriptions?
|
Thus, Professor J. decides that
the Program Description section of her evaluation
report will contain a table describing characteristics of
participating teachers and documentation of the technological
resources available to them.
|
|
Question: How do reports specify limitations
in data collected?
|
Professor J.’s second concern
has to do with the attrition of participating teachers during
the second and third years of the evaluation. Although Professor
J. understands that the attrition must be reported, she is
interested in the level of detail that other evaluators report
when describing limitations in the data collected. Professor
J. clicks on the Analysis Process excerpts to better
understand how such information is presented.
Among the Analysis Process excerpts she finds several relevant
examples. The first reports the limitations of data collected
with a survey instrument in which participants provided
only very brief responses and there was no follow-up to
further elaborate their responses. Another excerpt reports
the limitation of not having a random sample of project
participants, which reduced the generalizability of the
findings. Professor J. notes that in both these excerpts
there is a brief description of the limitation, which is
followed by implications for the analysis and interpretation
of the evaluation data. Using these two excerpts as models,
she elects to document the attrition in each year, provide
plausible hypotheses for its occurrence, and provide a cautionary
note concerning the effect of attrition on the generalizability
of the evaluation findings.
|
|
Question: What kinds of graphic displays are
used to report survey data?
|
As part of the Results section
of the evaluation report, Professor J. will present findings
from a survey that was conducted for each year of the three-year
project. She is considering reporting the data graphically
but is unclear about how much complexity can be clearly conveyed
by using simple bar graphs. She needs to report results for
several categories of participant characteristics (e.g., years
of teaching experience, geographic location, gender, age of
participant, etc.) and wants to report the data in a format
that is readily accessible to the evaluation stakeholders.
Professor J. clicks on the Results &
Recommendations
excerpts to see graphic displays of quantitative
results.
She finds several graphs in the Results &
Recommendations excerpts. In addition,
she reads several excerpts that report survey results by
subcategories of participants in prose only. By comparing
the prose and graphic results, she elects to use a combination
of approaches. She will present basic survey findings with
prose explanations, but rely on graphic displays to compare
selected subcategory results by year. Professor J. makes
copies of the relevant excerpts and
graphics.
|
|
Skim complete reports
|
Professor J. decides to skim several
of the complete reports to see how all the basic components
of sound Teacher Education reports have been integrated.
She goes to the top of the page and clicks on the
Reports tab, which returns her to the Teacher Education
Reports page. Then she clicks on the
link to each of the complete reports in turn to view
them.
|
| Bookmark the OERL Web site |
She bookmarks the OERL Web site on her
browser, anticipating that she may browse more
of the resources as she prepares a draft of the report.
|
