Step 1: Identify assumptions about change (R).

(R) = report example

Your first step is to identify the assumptions about change that provide the rationale for the project. After all, the very word "intervention" reminds us that the project is intervening to change something. Defining what the intervention is supposed to change, and how, is a necessary preliminary to deciding how to measure its effectiveness. Ideally, the assumptions are grounded in theory, which means that they have research to back them up.

Different assumptions about the value of the project carry different implications for determining its worth. Sometimes, a project is theorized as providing a new and better solution to a commonly accepted problem. Such projects need to be evaluated on the basis of whether they solve previously defined problems better than alternatives do. Other times, the project is viewed as an opportunity to pursue new goals not previously articulated. An example would be a project that uses new learning technologies in an instructional context that are hypothesized to open up new possibilities for learning that were previously beyond reach of the learners in that context. Such projects need to be evaluated on whether they succeed in being the agents of the outcomes they are hypothesized to cause.

Therefore, you need to decide whether the intent or intents of the evaluation should be to:

• Determine effects of the intervention (that is, causal relationships between the intervention and the outcome measures).


• Allow you to evaluate the strength and uniformity of the intervention across the different intervention participants.


• Help you explain the sources of variability in the results, provided that:
  • such an analysis is of interest to the stakeholder, and
  
  
  • additional conditions and factors have been identified that could be exerting an influence on outcomes.

Example:

The administrators of a school district are concerned that not enough of their students are enrolling in science elective courses. They have read research literature promoting the notion that people become more intrinsically motivated about a subject if they are exposed to it in a low-pressure, nonacademic context. Taking their cue from this research, they decide to implement an extracurricular field trip program. They hypothesize that two outcomes will happen:

1. The field trips will increase student interest in science, which will then increase their selection of the electives.


2. Interest will increase for all students, but high-achieving students will select more science electives than low-achieving students.

Note that these are assumptions about change: that, in this case, interest will increase, and that interest among particular students will increase the most. Having made these assumptions explicit makes it easier to define the evaluation's goals. In this case, the evaluation should certainly track the number of electives taken and the achievement level of the students signing up for electives.