Step 5: Pilot-test the questionnaire (P, R, U) and make needed revisions. Once your questionnaire is designed, pilot-test it with people not participating in the evaluation.

(P) = plan example (R) = report example (U) = user scenario

Pilot testers can provide valuable feedback about how to improve the wording and appearance of the questionnaire. In the case of an electronic questionnaire, additional pilot testing is necessary to make sure the questionnaire works on different computer systems and with respondents who have varying degrees of computer experience. Although there are no set rules for how many people should pilot a survey, a reasonable pilot-test protocol for an average-size evaluation is as follows:

A) Editing: Have 3-6 individuals who are familiar with the evaluation (but not involved in the questionnaire development) read through the questionnaire and make suggestions. Make revisions.

B) Early Pilot: Give the questionnaire to 4-8 individuals who are similar to your target respondents. Test each individual separately and have the testers "think out loud" (i.e., express their thoughts orally) as they work their way through the questionnaire. Make revisions.

C) Full Pilot: Give the questionnaire again to 10-15 individuals who are similar to your target respondents. In this case, try to approximate the conditions you plan to use for the questionnaire's full implementation. Collect feedback in the manner most convenient to the situation (e.g., conduct follow-up interviews and have the testers write down suggestions and comments). Make revisions.

For larger, higher-stakes evaluations, larger test groups and more iterations are advised.