Alignment Table for Plan Components

Describes what strategies and procedures will be used to gather and analyze data; as well as which ones will be used to periodically review the course of the evaluation.

Specifies:

• formative or summative approaches that will be taken
• types of data that will be needed (e.g., quantitative, qualitative, pre-post, longitudinal)
• sources of the data (e.g., participants, documents)

The plan should describe the proposed methodological approaches and how, within the constraints of time and cost, they will yield data that help answer the evaluation questions. The data gathered will need to be aligned with the goals that the project is intended to achieve. The data can vary, however, in how directly they indicate the attainment of project goals. Most projects are more likely to show effects on proximal outcomes than on distal outcomes that are either logically or temporally remote. (For example, a project has been designed to improve high school students' motivation to learn science. A proximal measure of the project's success would be student self-reports of interest in science content gathered immediately before and after the project. A distal measure would be whether the students decide to study science in college.)

Furthermore, the approaches should be grounded in respected methodological frameworks and best-practice literature. This increases the chance that project features and context that are likely to make a difference in project operations and outcomes will be identified.

Methodological approaches that look narrowly at project inputs and solely examine the results of quantitative outcome measures may not capture all the noteworthy influences, impacts, and outcomes of a complex project. Qualitative and mixed method approaches present alternative ways of detecting impacts, especially unanticipated ones. To corroborate evaluation findings and to provide multiple perspectives, it is highly desirable that evaluators measure multiple outcomes and gather data from multiple sources (triangulation).

Important constraints on the evaluation design (e.g., lack of random assignment of respondents to treatment and comparison groups, or lack of data on long-term effects) should also be stated at this point in the report.

U3 Information Scope and Selection
Information collected should be broadly selected to address pertinent questions about the project and be responsive to the needs and interests of clients and other specified stakeholders.

F3 Cost Effectiveness
The evaluation should be efficient and produce information of sufficient value, so that the resources expended can be justified.

Describes the sources of information that will be used in the evaluation, which may include:

• records and archival documents that contain relevant information
• the entire population of participants in the project, if data were collected on all of them
• the sample or samples of participants or other informants that will be observed or solicited for information, in order to maximize the generalizability of the findings to the population from which the sample or samples are to be drawn

The sources of information that will be used in the evaluation should be described in enough detail to build confidence that the information will be sufficient to meet the evaluation's purposes.

The groups selected to provide information (e.g., administrators, teachers, students, parents) should be identified and briefly described. If a sample is to be drawn, the description should contain:

• the sample selection criteria (e.g., the lowest achievers, the best instructors)
• the process by which the sample is to be selected (e.g., random, purposive)
• the proposed sample size
• whether or not any comparison or control groups will be included
• whether and how participants will be assigned to treatment and comparison groups

The extent to which the sample will be representative of the entire population should be indicated. Information about the sample will help reviewers determine the extent to which the information provided about the sample is of sufficient depth to help users of the report judge its representativeness and appropriateness given the scope, context, and resources of the evaluation.

A3 Described Purposes and Procedures
The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed.

A4 Defensible Information Sources
The sources of information used in a program evaluation should be described in enough detail, so that the adequacy of the information can be assessed.

Describes the design and content of the instruments that will be used to collect and analyze data (e.g., survey questionnaires, interview protocols, observation forms, learning assessments).

The plan should describe the nature of the various instruments and how they will be used to gather the needed information. Instruments should be used as intended in order for the data produced to be reliable and valid.

A3 Described Purposes and Procedures
The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed.

Describes how the data and other information will be gathered to meet the criteria of validity and reliability. Also describes the intended frequency, order, and duration of the various data collection activities.

The plan should describe how and when data will be obtained from the various sources and how the sources will provide corroboration and multiple perspectives.

A description of the data collection and its intent will provide a context for the eventual judging and interpreting of evaluation findings and recommendations.

The timing of data collection is important because the project's maturity is likely to have an impact on outcomes.

Hence, this section should describe:

• how and when an appropriately broad range of data will be collected
• what steps will be taken to get essential data from the sample and other targeted sources (this might include a human subjects review)
• what steps will be taken to ensure that the data meet the criteria of validity (e.g., piloting, field testing, stakeholder review)
• what steps will be taken to ensure that reliability is achieved (e.g., systematic training of data collectors and consistent data collection and scoring procedures)

Different models of evaluation present different data collection needs. For example, a formative evaluation requires that ongoing project activities be assessed at points in time that enable project developers to refine the project's components.

F1 Practical Procedures
The evaluation procedures should be practical, to keep disruption to a minimum while needed information is obtained.

A3 Described Purposes and Procedures
The purposes and procedures of the evaluation should be monitored and described in enough detail, so that they can be identified and assessed.

A5 Valid Information
The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the interpretation arrived at is valid for the intended use.

A6 Reliable Information
The information-gathering procedures should be chosen or developed and then implemented so that they will assure that the information obtained is sufficiently reliable for the intended use.

Describes procedures that will be undertaken to review the quality of the evaluation being conducted.

Evaluation purposes and procedures should be reviewed periodically, particularly during longitudinal evaluations, to determine whether the evaluation design, instruments, and procedures are adequately capturing the project's implementation, impacts, and outcomes.

A12 Meta-Evaluation
The evaluation itself should be formatively and summatively evaluated against … standards, so that its conduct is appropriately guided and, on completion, stakeholders can closely examine its strengths and weaknesses.