Step One: Decide how highly specified your observation approach needs to be in order to generate data that can answer your evaluation questions.

Observing student groups and teachers during the class periods when Science Search is in use seems the most straightforward way to generate answers to the evaluation questions. Thus, the context of the observations is quite specific (e.g., you would want to visit a class only on a day that Science Search was being used).

Within the context of interest, you reason that your observation approach should be moderately specified. You want to be open to capturing the dynamics of student use of the tool, which has not been well documented in prior data-gathering efforts. However, too much open-endedness might produce an overwhelming amount of information that could not be interpreted easily or cost-effectively. Also, there is no guarantee that observers would view things consistently across different classrooms.

Because you want to be able to compare data across all 10 classrooms, you realize there will be distinct advantages if you design some items that focus on clearly defined aspects of student group behavior and teacher support behavior that can be coded and quantified efficiently and reliably. You also realize that the function of this pilot study is to document complexities of implementation (e.g., technical problems, availability of resources) and any important unanticipated issues. Thus, you also intend to include items that would prompt observers to record these kinds of data.