Step 3: Edit and pilot instruments (R).

(R) = report example

All instruments to be used in an evaluation should first be edited by subject matter and assessment experts. These individuals will ensure that the content is appropriate and that attributes of good item writing are used.

The instruments should then be piloted. Piloting means trying the instrument out in order to get constructive feedback about its quality and usefulness for meeting your needs. Piloting is a critical step for detecting problems that could adversely affect the validity and reliability of results (e.g., confusing directions, misleading graphics, poorly designed response formats, insufficient or excessive length, awkward wording, confusing vocabulary). To pilot assessments and questionnaires, groups of respondents are administered a draft version of the instrument. To pilot interview and observation protocols, evaluators use a draft version of the protocols on a trial basis.

Piloting can be done in phases. An early pilot test is usually carried out with several representative subjects. This will result in one round of changes. Then a "full pilot" (otherwise known as a "field test") can be conducted with a larger representative group of subjects, the size of which depends on the characteristics to be represented in the targeted population. Both newly developed and highly adapted instruments should be subjected to a full pilot.

A good technique to use in assessment, questionnaire, and interview piloting is a "think-aloud." A think-aloud is a process in which respondents express what they are thinking as they answer the questions on an instrument. For example, in a learning assessment, they can state that a particular question is too confusing or otherwise difficult to answer.

Whether piloting an observation protocol or an instrument through a think-aloud, it is a good idea to have multiple observers record what is taking place. Then they should review their data together and modify the instruments as indicated by the results of the review. If the review reveals problems with the items, the instrument should be revised and piloted again.