Step 3: Determine what types of data to collect, and whom to collect data from (P, R).

(P) = plan example (R) = report example

After the assumptions about change are identified (Step 1), you need to consider the kinds of data you will need to address the evaluation questions and from whom to collect it.

What data to collect

Begin by considering which kinds of measurements, observations, or analysis you will need to do to yield the evidence you need. There are many kinds of data that can be collected—counts of the number of times participants take books out of the library, science achievement scores, amount of time spent by teachers in instructional preparation, degree of district innovativeness, students’ self-reports on academic self-concept, or parent attendance at after school meetings. The objectives of the evaluation, the stakeholders’ interests, and your own skills are the most important sources of influence in deciding what to measure or observe. There are five broad kinds of data that can be collected as part of an evaluation, and each is briefly described below.

Types of data to be collected:

• Context Characteristics: Projects operate within a framework of resources and constraints. These include: sociological factors, such as leadership, and communication patterns; project specific factors, such as size of participating schools and classes, amount of curricular material to be addressed, project budget, and use of incentives; demographic factors, such as percent minority, percent free lunch, number of teachers with credentials, and reform history of the school in which the project is set. These data are often collected from project records and interviews with the project director and staff.


• Participant Characteristics: Describing the project participants is critical in order to understand the populations and subpopulations being studied. Examples of data collected on individual participants may include: age, gender, socioeconomic status, language status, attendance in the intervention activities, experience in related activities, and prior knowledge of relevant content and skills. Participant questionnaires and, if resources permit, individual interviews are used to collect such information. Institutional records can provide evidence of prior performance of participants on important variables.


• Project Implementation Information: Implementation data provide evidence about the success with which the project’s components and features are established at the participating sites. Implementation measures might include: fidelity to the developers' goals; delivery of materials to project sites; usage of project-related practices and materials by staff; and participation of key personnel in training sessions. Another type of data that can be used to characterize project implementation involves descriptions of the project itself—the number and kinds of services provided, processes employed to ensure that the project activities occur on time and on budget, inventories of materials, and numbers and types of staffing and administrative structures used to support project activities. Implementation data can be collected through interviews with project directors and other staff. Focus groups can be conducted with staff as well as project participants. Observations of project activities and participant logs can document the occurrence, quality, and number of activities that are being implemented.


• Project Outcomes: Several kinds of outcomes are of interest to the evaluator. These include measures of the attainment of both intended and unintended outcomes; long- and short-term outcomes; outcomes for explicitly stated goals drawn up by key stakeholders as well as outcomes of interest to the larger community. Very frequently, outcome data are collected from project participants before and after engagement in the intervention or project. Attitudinal data, performance or behavioral data, and cognitive data (e.g., facts, application of knowledge and skills, explanations of phenomena, self-perceptions of performance) are often used in this way and serve as a key source of evidence about the effectiveness of the intervention or project. There are many standardized instruments available to measure these outcomes, including those with established technical quality data (i.e., reliability and validity). Outcome measures can include Instruments such as observation protocols of staff or participant behaviors, as well as instruments that scaffold the counting of events or tangible resources in the intervention or project setting.


• Project Costs: Evaluators are frequently required to provide data that can support decisions to continue, cancel, or modify an intervention or project. Such decisions require examination of project effectiveness as demonstrated in outcome data as well as the project’s cost effectiveness when compared to other projects with similar intents. Thus, evaluators often collect data about a project’s required resources and costs. These data might include costs of project labor, materials, and facilities, as well as indirect and opportunity costs. Project and institutional records, especially budgets and expenditures, become the key sources of such data.

These five broad categories of data provide general guidelines for the kinds of information that can be collected, but each evaluation is unique and final decisions about what kinds of information should be collected have to be based on the purpose and features of the specific evaluation.

Data Sources

The following are three different scenarios of an implementation evaluation. The project is a new calculus curriculum being taught in a large school district by a population of 100 teachers. (Here, we use the word "population" to refer to everyone who is participating in the intervention.) The first two scenarios illustrate a poorly balanced evaluation. The third illustrates a balanced approach that is achieved by mixing methods of data collection.

Depth at the expense of breadth

The evaluators spend many hours observing two of the teachers. The observations yield rich data, but no time is left over to gather data from the other 98 teachers. Furthermore, the two teachers are atypical of the larger group in terms of the amount of prior training they have had and the types of students they serve.

Breadth at the expense of depth

The evaluators ask all the teachers to fill out a questionnaire about the curriculum. Because they have only 5 minutes at the end of a workshop to administer it, they keep it very brief, with only three items about implementation:

1. Rate how well the implementation is proceeding.
   
2. Select (from a list provided) what barriers to implementation you are encountering.
   
3. Select (from a different list provided) what successes you are having.

The results will yield generalizable but superficial information. The evaluators will not be able to detect important details or unanticipated factors, nor will they be able to corroborate how reliable the responses are through the sort of firsthand information they would be able to glean from observations.

Balance between breadth and depth

The evaluators administer a questionnaire to a randomly drawn sample of 25 of the teachers and observe 10 of them teaching their classes. This approach enables them to have breadth (gathering survey data from 35% of the population) while also obtaining in-depth observational information about 10% of the teachers. This kind of evaluation is likely to be considered credible.

If your resources are too limited to achieve both breadth and depth, take stock of your evaluation needs and make your goals more modest. Ask yourself:

• What types of data do I need to answer my evaluation questions?
  
• Do I have enough resources to collect as much data as I would need from each type to provide conclusive answers to my evaluation questions?
  
• If not, how can I revise my goals in ways that permit me to reduce the types of data I need to collect, and thereby free up enough resources to collect enough data in the reduced set of types?