Step 1: Determine when, where, and how often to collect the data (P), (R).

(P) = plan example (R) = report example

Putting together a sound data collection plan requires determining (1) how the data will be collected; (2) if, when, and how often data should be collected from various people and settings; and (3) if and how data on the same people should be collected in multiple settings.

Determine how the data will be collected.

There are different ways to collect different types of data: (1) questionnaires can be administered by email, on interactive Web pages, and by regular mail; (2) observations can be conducted at the project setting or instead by observers watching videotapes of the setting; (3) interviews can be conducted face to face or on the phone; (4) learning assessments can be administered.

Choosing an option is initially predicated on the information needs for a particular evaluation question. For example, if the evaluation in part seeks to measure achievement, a learning assessment is the common-sense approach to take.

Each of these options carries advantages and disadvantages that should be weighed on a case-by-case basis. Issues that could impact decisions about data collection strategies and therefore should be given serious consideration include cost, availability of staff for carrying out the data collection, credibility of results, and logistical requirements. (See other modules about individual instruments for more details about the advantages and disadvantages of these different data collection methods.)

Determine if, when, and how often data should be collected from various people and settings.

To detect change, you will usually want to revisit the people from whom you need data multiple times. For example, if you are administering questionnaires to instructors taking workshops on a new teaching method to see how they think their skills have improved as a result of their learning the method, you may want to collect data.

1. just before the workshop (to get baseline data),


2. just after the workshop (to gauge the immediate sense of impact), and


3. at some point or points during their teaching of a class when they put the method into practice.

To take another example, if you are trying to measure a gain in learning, you may want to administer to a sample of people the same learning assessment before and after the intervention (e.g., a pre- and post-assessment) that is theorized to cause the gain.

You may also want to revisit the settings from which you need data. The value of observing a setting multiple times is that it allows you to detect patterns that you can draw conclusions from. The fewer the number of times you observe, the greater the chance that what you observed is the "exception," rather than the "rule." For example, if you have information that students in a particular classroom tend to behave differently on Fridays than on Mondays, you may want to visit the classroom on both days.

To administer the same instruments to the same people multiple times, interview them multiple times, or make repeated visits to the same settings, you should determine what interval of time should transpire between the data collections. To make that determination, answer this sequence of questions:

1. When should the first and last data sets be collected?


2. How many data sets should be collected in the interim


3. Will the interim data sets be most informative if they are collected at equal intervals or should their collection be timed to correspond to specific milestones or events?

Related to the issue of when and how many times to collect the data is concern about how far into the future you can project an effect. Some project stakeholders theorize that their project will result in an effect that is sustained over a long time. An example would be a project that aims to build inquiry skills in students that they can apply in many contexts in the future. Conversely, an example of a project that does not project a sustained effect would be one that only aims to better prepare students to improve their achievement in the content of a certain course. If the project you are evaluating is meant to result in a sustained effect, you should do what you can to detect that effect.

Determine if and how data on the same people should be collected in multiple settings.

If your design requires collecting data about a particular group of people, you may want to collect data about the group in different settings to see if a common pattern emerges. For example, if you are trying to evaluate the interaction between groups of students taking a course that meets in a variety of settings such as the computer lab, field trips, and the classroom, you may want to observe them in each setting.