Developing Observation Instruments: Selecting an Observation Approach
By Alexis Mitman Colker, Ph.D.

Observation instruments permit direct measurement of the settings in which participants work and their behavior in those settings. Whereas questionnaires and interviews capture participants' self-reported judgments and beliefs about evaluation-related topics, observation instruments typically involve the recording of selected aspects of the setting and of behavior exhibited by and among individuals. These recordings usually are interpreted in terms of the degree to which the setting and participants' actual behavior support or have been influenced by the activities of the project being evaluated.

The most salient advantage of using an observation instrument is that it creates the opportunity to witness firsthand the complexities of the target setting. What the observer sees is likely to be not only more intricate than what could be captured from an interview or questionnaire but also more grounded in the events of the situation. A participant who self-reports in an interview or questionnaire is vulnerable to conditions such as memory unreliability, social desirability, and the intention-behavior gap, all of which decrease the accuracy of the reporting. When a participant is observed, it is far less likely that her or she can radically alter the setting or established patterns of behavior involving other people, especially when the observation is unobtrusive and samples enough representative events.

The difficulties of selecting and using an observation instrument are multiple and significant. First, a large variety of measurement techniques are available for an observation protocol, and selecting the appropriate one(s) can involve numerous decisions. Second, observation instruments are inherently complex and pose distinct challenges in designing them and training observers to understand and record on them properly. Third, the logistics of using an observation instrument are especially difficult. Observations involve greater barriers to acceptance by participants than either questionnaires or interviews, careful selection of repeated observation occasions to allow for meaningful and reliable data, and a high level of expense. This module focuses on helping you deal with the first set of difficulties: understanding and selecting the most appropriate observation techniques for your evaluation. The remaining issues will be dealt with in forthcoming observation modules.

Although this module assumes that an evaluator is interested in designing some sort of observation instrument, it should be noted that evaluators often may be tempted to turn to recording technology first to accomplish their observations. It certainly is true that audio or video recordings of the target setting can yield a good and unbiased permanent record of events that can be analyzed repeatedly after it is collected. Such recordings, however, do not necessarily diminish the challenges of generating usable and meaningful observation data. An audiotape or videotape by itself will not provide an "answer" to an evaluation question. Instead, decisions must be made about what to look for on the tape(s) and how the recorded behaviors and events of interest can be interpreted and characterized reliably. Thus, audio or video recordings are not shortcuts to conducting observations. The same steps detailed in this module will be required to make use of the recordings.

The objectives of this module are for you to understand:

1. The importance of determining what features to observe
2. The underlying dimensions of observation techniques
3. The advantages and disadvantages of different kinds of observation techniques