Below are answers to frequently asked question about OERL
in general. Also available are answers to specific questions
well as user scenarios for
What is OERL's purpose?
Evaluation is a central component of projects funded
by the National Science Foundation's Directorate for
Education and Human Resources (EHR). Sound project
evaluations provide both project developers and NSF
staff with evidence about how well a project is
accomplishing its goals.
The goal of the Online Evaluation Resource Library
(OERL) is the continuous improvement of evaluations used
to monitor and judge projects' effectiveness. OERL
provides a rich collection of evaluation best
practices, guidelines for their applications to
projects, and a forum for stimulating ongoing dialogue
in the evaluation community. OERL is designed to
support applications of sound evaluation methodologies
to projects, not to replace a full course of study for
those going into the evaluation field.
How is evaluation defined?
Program Evaluation Standards:
2nd Edition (1994) defines evaluation as "the
systematic investigation of the worth or merit of an
object." By using systematic procedures to
appraise the conceptualization, design,
implementation, impacts, and utility of projects,
evaluations can collect valid and reliable evidence
that accurately documents project accomplishments and
Who are the target audiences?
OERL is designed to help current and aspiring
principal investigators and evaluators learn more
about how project evaluations have been planned,
implemented, and reported within the context of EHR
programs. Additional audiences include NSF program
officers, evaluation professionals, and evaluation
What types of EHR projects are represented in the
EHR funds a broad variety of programs aimed at
improving the quality of science, mathematics,
engineering, and technology education. Each project is
funded by these programs is required to conduct an
evaluation. OERL contains materials that have been
used to evaluate large, comprehensive, systemic
projects as well as small, tightly focused
interventions. The projects or project components
target the improvement of teacher education, faculty
development, curriculum development, and laboratory
improvement, as well as under-represented populations and
What types of evaluation resources are in the
The evaluation resources in OERL have been selected
to present a range of qualitative and quantitative
approaches that were employed in evaluating actual EHR
projects. OERL includes three types of resources:
plans, instruments, and reports.
Plans: Typically, evaluation plans
are embedded in project proposal narratives, which are limited to
15 pages, although some evaluators have prepared more
extensive, stand-alone plans. Annotated excerpts from
plans, as well as entire plans—embedded or
Instruments: Evaluations collect
data from existing archival sources and indicators
(e.g., course credits, grades) and develop new
instruments such as surveys, interviews, and tests.
OERL contains complete instruments that represent a
range of formats, accompanied by analyses of their
content and structure.
Reports: Project reports to NSF may
describe progress to date or summarize end-of-project
findings and conclusions. Evaluation documentation and
data may be prepared as separate reports or as
sections embedded within a project report. These
reports can be quite lengthy. OERL has only a few full
reports to illustrate how the components of an
evaluation report are integrated into a complete one.
OERL presents numerous excerpts from EHR evaluation
reports to provide a variety of effective
How were the evaluation materials selected?
The sample evaluation materials have been selected
to represent sound evaluation practice as delineated
by Program Evaluation Standards: 2nd
Edition (1994) and other evaluation sources.
How can the evaluation resources be used?
Evaluation resources in the online library can be
used in a variety of ways.
General Models: The materials can
serve as models for the structure and content of new
plans, instruments, or reports. The examples were
selected to illustrate alternative ways in which a
particular evaluation component has been
Adaptations: The materials might be
adapted in large or small ways for the evaluation of a
similar project. For example, a survey about a teacher
workshop might need only fine-tuning to be applicable
to another workshop. Similarly, a table summarizing the
comparison of data from two groups might need only
minor changes to serve as a way to display results
from a similar project.
Use "as is": An existing
analysis plan or instrument may be perfect for a new
project. For example, a calculus exam or faculty
survey of calculus professors might be usable for an
evaluation of a new calculus project.
How do I search the evaluation resources?
There are three ways to
search the evaluation
resources. For the keyword search, type in
keywords of interest, and documents on the OERL site
that contain content matching keywords will be
displayed. The evaluation resources search
allows the user to search for resources within Plans,
Instruments, Reports, or Contributing Projects.
Each of these searches will present a list of
evaluation features that the user can select to
generate more relevant results. Finally, the
component resources search allows the user to
search for resources across plans and reports.
How do I navigate around the site?
Resources inside OERL are cross-referenced between
resource type and project type. So, for example, if
you are on the home page and you want to find
instruments for teacher education, you could click on
the Instruments tab, which will take you to
the Instruments overview page. Then click on the
teacher education tab. Conversely, you could click
first on Teacher Education, taking you to the teacher
education overview page. From there, either click
the Instruments tab or click the Instruments link.
What are alignment tables?
These tables describe the
major components and
characteristics of evaluation plans, instruments, and
reports. The table defines each component or
characteristic (the glossary); the features that
make it sound (the quality criteria); and which
standards (based on the Program Evaluation Standards:
2nd Edition) should be met that are aligned to
Where did the evaluation resources come from?
OERL evaluation resources have been selected from
projects funded by programs administered by divisions
of the National Science Foundation's Directorate of
Evaluation and Human Resources. A small group of
technology projects funded by the U.S. Department of
Education also are represented in OERLl.
What are user scenarios?
Each user scenario describes
one way to use
the evaluation resources in the library.
Each scenario takes the user step by step through an example illustrating how evaluators might use OERL to
develop plans, instruments, or reports.
How do I contact OERL?
If you have questions or comments, or need some help,
you can email us at firstname.lastname@example.org.
Your message will be sent to OERL staff, including
Edys Quellmalz (project director) and Patti
Schank (technology director).
You can also reach us via phone at 650-859-5504, or
send postal mail to
333 Ravenswood Ave.
Menlo Park, CA 94025