Step 10: Maximize the efficiency of your sample.

Evaluations often classify their subjects by dividing them into hierarchical units of analysis, such as districts, schools, classes, and students. Here, the district would be the broadest unit and the student would be the smallest unit. In your sampling procedure, you can select quantities of units in a way that maximizes the sample's efficiency for detecting power. This is done by maximizing your sample size at the broadest ("primary") unit of analysis and minimizing the size at more granular ("secondary") units of analysis.

Why maximize the broadest units? Because they influence the units within them. For example, school cultures influence the behavior of teachers, who in turn influence the behavior of students. The primary.unit of analysis always has attributes that could influence outcomes among its members (the secondary units of analysis).

Statistically speaking, you lose power when you have too few cases in your primary unit of analysis, and this power is not compensated for by large samples of the secondary units. In other words, if you have too few primary units, you are not going to be able to compensate by adding more secondary units. For example, if you are doing an evaluation in a district with 40 schools, and your primary unit of analysis is the school and the secondary unit is the teacher, your sample will be more efficient if you have more schools and fewer teachers per school than if you have fewer schools and more teachers per school.

Consider an evaluation of a new curriculum that is being piloted in classes at a university. The classes have been randomly assigned to intervention and control groups. Table 13 shows the different consequences of sampling when the primary unit of analysis is the class and the secondary unit of analysis is the student. Whereas alternative #1 is the worst because it has low power, alternative #4 is the best because it has both high power and high efficiency.

Getting large sample sizes at the primary unit of analysis can be difficult because large quantities of them may not be available. If this is the situation you face, you need to make trade-offs. The following is an example of a population in which there do not exist enough primary units of analysis to generate much power. This forces a reconceptualization of what the primary unit needs to be and the formulation of a strategy for minimizing the resultant risks of bias.

Example of maximizing efficiency of a sample

A district is conducting an evaluation of a curriculum intervention. To avoid contamination from interactions between intervention and control teachers, which would be likely to happen if they were within the same school, the evaluators make the school the primary unit of analysis. In other words, they want schools to be randomly assigned to intervention and control groups, so that all the participating teachers in each school are all one or the other. Getting sufficient power requires selecting a large number of schools. Unfortunately, there are only six schools in the district.

Hence, the evaluators decide that the risk from contamination is not as important to them as the risk from low power. They make the teacher the primary unit of analysis and decide to redirect their evaluation goals away from trying to differentiate effects by school. They try to minimize contamination by asking the intervention teachers to refrain from talking about the intervention with the control teachers.