Quality Criteria for Plans

The following features of the targeted project should be overviewed:

• Project goals (both explicit and implicit) and objectives
• Principal project activities designed to achieve the goals
• Expected short-term and long-term outcomes

The following, additional overview information should be provided, if available:

• Project location and implementation sites
• Project duration
• Resources used to implement the project

If more than one site is implementing a project, the plan should, if possible, describe the sites and the anticipated variation that may be expected across the sites.

The different stakeholder groups should be identified and their relationships to the project summarized, as well as whatever is already known about their perspectives that has impacted decision making on the evaluation design being proposed in the plan.

An understanding of contextual factors is necessary if an evaluation is to be realistic and responsive to the conditions within which the project operates.

The purposes of the evaluation should be stated in terms of goals and intended uses of results by stakeholders.

The evaluation should focus on whether or not promised project components are delivered and it should compare project outcomes against the assessed needs of the targeted participants or other beneficiaries. The evaluation should also be directed at finding unanticipated outcomes, both positive and negative.

Evaluation questions that address context, implementation, and outcome variables provide the perspective not only for the eventual interpreting of results, but also for understanding the conditions under which the results were obtained.

The questions should be justified against the following criteria:

• To which stakeholders will answers to the questions be useful, and how?
• How will answers to the questions provide new information?

The plan can also state questions that are worth answering but that will not be addressed in the evaluation, due to constraints (e.g., limited time or resources, insufficiency of available data-gathering techniques).

The professional qualifications of the evaluator should be specified in order to build trust in the evaluation as it unfolds.

The plan should describe how the positions and perspectives of the stakeholders will be taken into account throughout the evaluation, from planning to data collection, analysis, and interpretation. Stakeholder involvement in the evaluation can be beneficial because stakeholders can help the evaluator better understand project goals and objectives, shape evaluation questions, recommend data sources, and review findings. As a consequence of being involved, stakeholders are more likely to find the results credible, useful, and relevant, and less likely to curtail evaluation operations or hinder accurate and appropriate uses of the results.

The plan should describe the proposed methodological approaches and how, within the constraints of time and cost, they will yield data that help answer the evaluation questions. The data gathered will need to be aligned with the goals that the project is intended to achieve. The data can vary, however, in how directly they indicate the attainment of project goals. Most projects are more likely to show effects on proximal outcomes than on distal outcomes that are either logically or temporally remote. (For example, a project has been designed to improve high school students' motivation to learn science. A proximal measure of the project's success would be student self-reports of interest in science content, gathered immediately before and after the project. A distal measure would be whether the students decide to study science in college.)

Furthermore, the approaches should be grounded in respected methodological frameworks and best-practice literature. This increases the chance that project features and context that are likely to make a difference in project operations and outcomes will be identified.

Methodological approaches that look narrowly at project inputs and solely examine the results of quantitative outcome measures may not capture all the noteworthy influences, impacts, and outcomes of a complex project. Qualitative and mixed method approaches present alternative ways of detecting impacts, especially unanticipated ones. To corroborate evaluation findings and to provide multiple perspectives, it is highly desirable that evaluators measure multiple outcomes and gather data from multiple sources (triangulation).

Important constraints on the evaluation design (e.g., lack of random assignment of respondents to treatment and comparison groups, or lack of data on long-term effects) should also be stated at this point in the report.

The sources of information that will be used in the evaluation should be described in enough detail to build confidence that the information will be sufficient to meet the evaluation's purposes.

The groups selected to provide information (e.g., administrators, teachers, students, parents) should be identified and briefly described. If a sample is to be drawn, the description should contain

• the sample selection criteria (e.g., the lowest achievers, the best instructors),
• the process by which the sample is to be selected (e.g., random, purposive),
• the proposed sample size,
• whether or not any comparison or control groups will be included, and
• whether and how participants will be assigned to treatment and comparison groups.

The extent to which the sample will be representative of the entire population should be indicated. Information about the sample will help reviewers determine the extent to which the information provided about the sample is of sufficient depth to help users of the report judge its representativeness and appropriateness, given the scope, context, and resources of the evaluation.

The plan should describe the nature of the various instruments and how they will be used to gather the needed information. Instruments should be used as intended in order for the data produced to be reliable and valid.

The plan should describe how and when data will be obtained from the various sources and how the sources will provide corroboration and multiple perspectives.

A description of the data collection and its intent will provide a context for the eventual judging and interpreting of evaluation findings and recommendations.

The timing of data collection is important because the project's maturity is likely to have an impact on outcomes.

Hence, this section should describe

• how and when an appropriately broad range of data will be collected,
• what steps will be taken to get essential data from the sample and other targeted sources (these steps might include a human subjects review),
• what steps will be taken to ensure that the data meet the criteria of validity (e.g., piloting, field testing, stakeholder review), and
• what steps will be taken to ensure that reliability is achieved (e.g., systematic training of data collectors, and consistent data collection and scoring procedures).

Different models of evaluation present different data collection needs. For example, a formative evaluation requires that ongoing project activities be assessed at points in time that enable project developers to refine the project's components.

Evaluation purposes and procedures should be reviewed periodically, particularly during longitudinal evaluations, to determine whether the evaluation design, instruments, and procedures are adequately capturing the project's implementation, impacts, and outcomes.

The proposed quantitative analysis procedures should be appropriate to the evaluation questions being addressed and the characteristics of the information being analyzed. Evaluators should consider the practical significance (e.g., effect sizes) and replicability, as well as statistical significance, when drawing inferences and formulating conclusions from quantitative analyses. Analyses of effects for identifiable subgroups should be planned, as appropriate, because a program may have differential effects for those subgroups.

Potential weaknesses in the quantitative data analysis, along with their possible influence on interpretations and conclusions, should be explained.

The proposed qualitative analysis procedures should be appropriate to the evaluation questions being addressed and the characteristics of the information being analyzed. As the evaluation progresses, evaluators will need to confirm the accuracy of findings from qualitative data by gathering evidence from more than one source and by subjecting inferences to independent verification.

Potential weaknesses in the qualitative data analysis, along with their possible influence on interpretations and conclusions, should be described.