Step 9: Determine the optimal sample size.

You learn from the superintendent and the math curriculum committee that they do not expect to see a big intervention effect when comparing the intervention and control groups on the standardized test scores. Therefore, they want to be asssured that the statistical tests used in data analysis will detect any true difference, even if it is small. Otherwise, they will not be able to properly weigh the pros and cons of a full adoption of the intervention.

There are no previous studies of the effect of Math World on student performance. Hence, there are no easy answers to the question of how large the sample should be to convince the stakeholders that a detected effect is real. You discuss with the key stakeholders what effect size would satisfy them. Knowing what previous experience others have had with the intervention in similar schools would help in determining this.

Then, you discuss with them how important it is for them to have high power and high confidence in the results. You learn that they are more worried about missing an effect (Type II error) than finding an effect that does not really exist (Type I error). In other words, they are more interested in having high power than high confidence. They are not as worried about the latter because most of the money has already been spent on the intervention, hence there are few risks to continued use of Math World by interested teachers, even if the detected effect is erroneous.

Finally, with the help of confidence intervals, you explore what sample sizes would be required to declare the targeted amount of effect true at various levels of confidence. Also, with the help of statistics books about power, you explore what sample sizes would be required to meet certain expectations of power. Once you help them weigh the trade-offs between sample size on one hand and power and confidence on the other, they can reach a decision.