Step 3. Edit and pilot instruments.

You want to conduct an "early pilot" of each questionnaire with three to six respondents. The facilitators, teachers, and students you want to select for the early pilot are not in your participant samples (for more on sampling, see the Methodological Approaches and Sampling module), yet they are qualified because they participated in the earth science in-service workshop in previous years. They will help you to identify problems with questionnaire wording and format. In the process, you ask them to say whatever comes to their minds as they answer the questions. Unfortunately, you cannot get enough previous facilitators and teacher participants for the pilot, so you seek participation from some facilitators of other science education workshops, and high school teachers of other science courses. All have enough familiarity with earth science education, cooperative learning, and field data gathering to provide useful feedback on your questionnaires.

After the pilot, you look for errors in the way they answer the questions. These errors could indicate problems in the content or format of the questionnaire. Below are two illustrations:

1. The presence of "Skip" questions on the teacher questionnaires could contribute to difficulty following directions. See Step 3 of the Questionnaire Design module for more on Skip questions. You ask the teachers who have students with limited English proficiency in their earth science classes to answer a special set of questions that teachers who do not have such students can skip. You want to see if the appropriate teachers are answering the special questions.
2. Certain patterns of responses would be contradictory if selected by the same teachers. You want to determine if any of your pilot teachers answers the questions in a contradictory manner. For example, a contradiction would exist where a teacher indicates that he or she has five years of teaching experience and, in response to another question, claims six years of experience in teaching earth science.

If these or other problems arise, you will ask the teachers about their responses. Their responses may yield suggestions for revising the questionnaires.

To pilot the teacher participant interview protocols, you go through the entire interview with the teachers with whom you piloted the questionnaires. After each interview, you ask the teachers to give feedback about how comfortable they felt with the interview setting, the clarity of the questions, and the length of the interview. On the basis of this feedback, you revise the interview protocol to shorten it and clarify some ambiguous questions.