go home

Select a Professional Development Module:
Key Topics Strategy Scenario Case Study References

Introduction  |  Step 1  |  Step 2  |  Step 3

Step Three: Weigh the advantages and disadvantages associated with each observation technique before selecting which one or combination to use as the basis for your observation instrument.

The preceding discussion about the five dimensions underlying observation items indicates the complexity of choices facing the designer of an observation instrument. Some or many of these choices may be heavily constrained by the evaluation questions (e.g., a very narrow focus) and logistical realities (e.g., a limited budget, the number of available observers and their level of experience). Nonetheless, the evaluator usually has considerable latitude in designing the instrument, and many of the choices will be a matter of preference. In Table 2, we summarize the advantages and disadvantages associated with the contrasting sides of each dimension.

Table 2. Advantages and Disadvantages of Five Observation Dimensions

  Lower Inference
Higher Inference
Advantages
 Easier to develop, define and code because deals with surface features/events. Easier to address complex features/events or comprehensive judgments.
Disadvantages
 Unable to capture more complex features/events; coding may become repetitious and insensitive. Difficult to construct and define, and to achieve observer agreement.
  Event Sampling
Time Sampling
Advantages
 Allows focus on particular context and elimination of other information. Coding at intervals will sample well over the time span and capture broad range of events; multiple data points generated.
Disadvantages
 May involve intense coding in short intervals; little data may be collected; observer may not always recognize target context. Coding categories must capture all that can occur in an interval; coding may involve quick decisions and become repetitious.
  All-Inclusive Subjects
Targeted Subjects
Advantages
 More likely to yield data representative of the context; all subjects contribute to database. Easier for observer to focus on selected subjects; possible for data to focus on/contrast particular kinds of subjects.
Disadvantages
 Difficult to see/hear all subjects: difficult to track large numbers of subjects and code separate IDs if desired. Targeted subjects may not be representative; nontargeted subjects who strongly influence situation may be missed.
  Real Time
Post Hoc
Advantages
 Allows recording of many sequential details; minimizes memory burden. Allows for more global judgments that take into account multiple features and developments over time; observer can be more relaxed and attentive to entire environment.
Disadvantages
 Often requires extreme focus and quick decisions; recording may not keep up with pace of events. Unsuited to capture great detail; observer may have difficulty remembering events on which recording needs to be based.
  Quantitative
Qualitative
Advantages
 Allows for quick coding; possible to cover a large range of topics/concepts; data quickly ready for statistical analyses. Allows for descriptive recording that can capture great detail and atmosphere; possible to note unanticipated features/events.
Disadvantages
 Coding categories may fail to capture breadth of possibilities; rich description lost and unexpected events lost; observers need extensive training for coding agreement. Requires skilled note taking and writing; observers' biases more apt to distort recording; data not readily amenable to analysis.

As mentioned earlier, any given observation item can be characterized in terms of its placement on each of the five dimensions. For example, a particular rating of a teaching assistant's overall management skills could be designed to be higher inference, time sampled, trageted subjects, real time, and quantitative. An in-depth description of three students doing a science presentation could be designed to be lower inference, event sampled, targeted subjects, post hoc, and qualitative. Although only some of the dimensions may be crucial determining factors for your item's design (e.g., wanting lower-inference judgments in real time), it behooves you to think of where your item falls on the other dimensions because all dimensions have implications for ease of development, use, coding, and reliability of the measures.

Typically, an evaluator will choose several diffferent item styles to go in a single observation instrument. For example, a given instrument could call for a combination of checklists, ratings, and open-ended descriptions, thereby representing different parts of a dimension or hybridizing within and across dimensions. Such combinations may be appropriate for covering a range of evaluation questions or traingulating different types of data with regard to answering a single evaluation question. Figure 6 shows an example of a portion of an observation instrument containing different item styles.

Section A: Classroom Overview

Students are seated in (check all that apply):

Rows
Small groups
U-Shape
Next to each other in a line (e.g., lab benches)
Isolated from other students
There are not enough seats for all

Does the seating arrangement seem fair? _________________ If no, explain briefly:

 

 

Curriculum-Related Materials Accessible or on Display (circle one number)

  1. The classroom has virtually no curriculum-related information on display. No engaging topic-related materials are apparent or available for student exploration.
  1. The classroom has minimal curriculum-related materials on display, e.g., one poster on famous scientists and one plant experiment on a back shelf.
  1. The classroom has several examples of curriculum materials on display that represent different topics likely to be covered. Some of the available materials might be useful to students.
  1. The classroom has many examples of curriculum materials on display that represent a good range of topics likely to be covered. Many of the materials should be of interest to students.
  1. The classroom has many curriculum materials on display that are strikingly attractive and should reinforce and stimulate students' understanding of many topics likely to be covered.

Student Work Products on Display (circle one number)

  1. The classroom has virtually no displays of student work.
  1. The classroom has only one or two displays of student work. The displays are haphazard and may only represent the work of some students.
  1. The classroom has several displays of student work. The displays are moderately organized and probably represent the work of most students
  1. The classroom has many displays of student work in different parts of the room. The displays are attractive and probably represent the work of all students.
  1. The classroom has many displays of student work in different parts of the room. The displays are exceptionally attractive and creative and appear to represent the work of all students.

If anything stands out especially about the appearance of this classroom, note it here:

 

 

Figure 6. Portion of an observation instrument illustrating the use of different types of items (Adapted from Gallagher et al., 2003).