Frequently Asked Questions About OERL

What is OERL's purpose?

Evaluation is a central component of projects funded by the National Science Foundation's Directorate for Education and Human Resources (EHR). Sound project evaluations provide both project developers and NSF staff with evidence about how well a project is accomplishing its goals.

The goal of the Online Evaluation Resource Library (OERL) is the continuous improvement of evaluations used to monitor and judge projects' effectiveness. OERL provides a rich collection of evaluation best practices, guidelines for their applications to projects, and a forum for stimulating ongoing dialogue in the evaluation community. OERL is designed to support applications of sound evaluation methodologies to projects, not to replace a full course of study for those going into the evaluation field.

How is evaluation defined?

Program Evaluation Standards: 2nd Edition (1994) defines evaluation as "the systematic investigation of the worth or merit of an object." By using systematic procedures to appraise the conceptualization, design, implementation, impacts, and utility of projects, evaluations can collect valid and reliable evidence that accurately documents project accomplishments and challenges.

Who are the target audiences?

OERL is designed to help current and aspiring principal investigators and evaluators learn more about how project evaluations have been planned, implemented, and reported within the context of EHR programs. Additional audiences include NSF program officers, evaluation professionals, and evaluation training programs.

What types of EHR projects are represented in the library?

EHR funds a broad variety of programs aimed at improving the quality of science, mathematics, engineering, and technology education. Each project is funded by these programs is required to conduct an evaluation. OERL contains materials that have been used to evaluate large, comprehensive, systemic projects as well as small, tightly focused interventions. The projects or project components target the improvement of teacher education, faculty development, curriculum development, and laboratory improvement, as well as under-represented populations and technology.

What types of evaluation resources are in the library?

The evaluation resources in OERL have been selected to present a range of qualitative and quantitative approaches that were employed in evaluating actual EHR projects. OERL includes three types of resources: plans, instruments, and reports.

Plans: Typically, evaluation plans are embedded in project proposal narratives, which are limited to 15 pages, although some evaluators have prepared more extensive, stand-alone plans. Annotated excerpts from plans, as well as entire plans—embedded or stand-alone—are presented.

Instruments: Evaluations collect data from existing archival sources and indicators (e.g., course credits, grades) and develop new instruments such as surveys, interviews, and tests. OERL contains complete instruments that represent a range of formats, accompanied by analyses of their content and structure.

Reports: Project reports to NSF may describe progress to date or summarize end-of-project findings and conclusions. Evaluation documentation and data may be prepared as separate reports or as sections embedded within a project report. These reports can be quite lengthy. OERL has only a few full reports to illustrate how the components of an evaluation report are integrated into a complete one. OERL presents numerous excerpts from EHR evaluation reports to provide a variety of effective examples.

How were the evaluation materials selected?

The sample evaluation materials have been selected to represent sound evaluation practice as delineated by Program Evaluation Standards: 2nd Edition (1994) and other evaluation sources.

How can the evaluation resources be used?

Evaluation resources in the online library can be used in a variety of ways.

General Models: The materials can serve as models for the structure and content of new plans, instruments, or reports. The examples were selected to illustrate alternative ways in which a particular evaluation component has been developed.

Adaptations: The materials might be adapted in large or small ways for the evaluation of a similar project. For example, a survey about a teacher workshop might need only fine-tuning to be applicable to another workshop. Similarly, a table summarizing the comparison of data from two groups might need only minor changes to serve as a way to display results from a similar project.

Use "as is": An existing analysis plan or instrument may be perfect for a new project. For example, a calculus exam or faculty survey of calculus professors might be usable for an evaluation of a new calculus project.

How do I search the evaluation resources?

There are three ways to search the evaluation resources. For the keyword search, type in keywords of interest, and documents on the OERL site that contain content matching keywords will be displayed. The evaluation resources search allows the user to search for resources within Plans, Instruments, Reports, or Contributing Projects. Each of these searches will present a list of evaluation features that the user can select to generate more relevant results. Finally, the component resources search allows the user to search for resources across plans and reports.

How do I navigate around the site?

Resources inside OERL are cross-referenced between resource type and project type. So, for example, if you are on the home page and you want to find instruments for teacher education, you could click on the Instruments tab, which will take you to the Instruments overview page. Then click on the teacher education tab. Conversely, you could click first on Teacher Education, taking you to the teacher education overview page. From there, either click the Instruments tab or click the Instruments link.

What are alignment tables?

These tables describe the major components and characteristics of evaluation plans, instruments, and reports. The table defines each component or characteristic (the glossary); the features that make it sound (the quality criteria); and which standards (based on the Program Evaluation Standards: 2nd Edition) should be met that are aligned to it.

Where did the evaluation resources come from?

OERL evaluation resources have been selected from projects funded by programs administered by divisions of the National Science Foundation's Directorate of Evaluation and Human Resources. A small group of technology projects funded by the U.S. Department of Education also are represented in OERLl.

What are user scenarios?

Each user scenario describes one way to use the evaluation resources in the library. Each scenario takes the user step by step through an example illustrating how evaluators might use OERL to develop plans, instruments, or reports.

How do I contact OERL?

If you have questions or comments, or need some help, you can email us at oerl-contact@ctl.sri.com. Your message will be sent to OERL staff, including Edys Quellmalz (project director) and Patti Schank (technology director).

You can also reach us via phone at 650-859-5504, or send postal mail to

Geneva Haertel
SRI International
333 Ravenswood Ave.
Menlo Park, CA 94025