Step 3: Determine what types of data to collect, and whom to collect it from.

The two kinds of evaluation data that are most important to this evaluation are project implementation information and project outcomes. To evaluate how well implementation is proceeding, you will observe and interview the teachers regularly. The interview and observation protocols will be organized by categories of interest implied by the nature of Math World, such as time on task, logistics, and training on using the materials. The data from the protocols will be either quantitative or qualitative and thus require tabulation or coding, respectively.

Your evaluation of the outcomes will focus on whether the students' use of Math World leads to the two anticipated effects. To measure whether Effect A (that is, whether the intervention leads to increased student understanding of important content in the math curriculum) is being met, you consider comparing standardized math test scores given at the end of 8th grade (the intervention year) with the same students' performance at earlier grade levels. To measure whether Effect B is being met (that is, whether the intervention leads to increased student skill at solving complex math problems and communicating well-reasoned, evidence-based solutions), you consider administering pre- and posttests that are designed to assess the skills that Math World is hypothesized to enhance. Both of these pretest-posttest designs could be strengthened with the collection of data from a comparison group. Step 4 addresses the use of comparison group data in evaluation design.