Technology Annotated Report Excerpts

Excerpt 1 [Agents]

Information Sources & Sampling:
Lists multiple instruments and studies that will be used to gather information about implementation

Also underway are a series of interrelated studies designed to look more closely at the project's implementation and gauge its progress towards the stated goals and outcomes. These include:

• surveys and interviews with participating teachers,
• documentation of students' multimedia presentations and other events,
• a telephone survey of principals and administrators,
• a study of writing development and achievement among sixth-grade students and writing instruction among teachers,
• a study of self-efficacy and technology competency among students, and
• an assessment of the technology infrastructure.

Information about the rationale, design, and sample for each study, along with preliminary findings, is provided below. The final report will include further data analysis and full discussions of findings.

Excerpt 2 [Agents]

Data Collection Procedures & Schedule:
Describes data collection activities and their various purposes

At the beginning of this project, we conducted in-person interviews with the 18 teachers attending the 1997 Summer Tech Camp. We also administered written surveys to all new participating teachers in the Fall of 1997. The purpose of both activities was to gather a baseline of information on teachers' technology skills, familiarity with the African and African-American Infusion curriculum, and expectations for this project. This year, we sent surveys to all 60 current participants to continue to track changes in teachers':

• technology skills and attitudes,
• personal use of technology,
• use of technology for instruction,
• observations on student use of technology,
• curriculum and teaching strategies,
• use of professional development,
• project expectations, and
• views on parental participation and community involvement.

Excerpt 3 [Delaware]

Methodological Approach:
Describes the underyling role of theory in the evaluation

THEORY-BASED EVALUATION

All projects are based on theories, although often unstated, of how and why they should "work" (Weiss, 1995). Theory-based evaluation provides a useful framework for formalizing the logic of the theories underlying a project and in guiding the determination of measurement points during the evaluation (Aronson, Mutchler, & an, 1998). Examining the theories on which a project is based aids in determining what evaluation data should be collected as well as when during the project lifecycle the data should be collected. However, evaluative data cannot be interpreted in isolation without also examining how the project was implemented. For example, if an evaluation reveals that student outcomes did not improve, it would be incorrect to automatically assume that the theories underlying the project should be rejected. Rather, the project's implementation should be examined to determine if the implementation was congruent with the hypothesized theories underlying the project. On the other hand, if student outcomes did improve, it is equally as important to postpone acceptance of the underlying theories until sufficient implementation has been verified. The overarching goal of Delaware's Challenge Grant is to increase student learning. Theory-based evaluation methods were used to document why project staff believe this intervention will result in an increase in learning and to specify what data must be collected during the evaluation lifecycle to determine if intervention results support these theories. The critical theories behind this project are that through extending the learning day as well as through increased parent involvement in education, student learning will improve. Although, there are other theories project staff believe may aid in reaching their ultimate goal, such as improving teaching strategies and making learning fun through technology. With these theories in mind, data elements were identified that will aid in determining if the theories are acceptable. Based on these theories, classroom usage and student home usage of the software were collected (the hypothesis supposes that student learning time will increase prior to seeing an increase in achievement). Also collected was the amount of time the parent spends with the student at home on the software (the hypothesis presumes parent involvement will increase prior to seeing an increase in achievement). While it was not measured this year, next year's evaluation (1999-2000) plans to measure changes in student attitudes towards learning (the hypothesis being that student attitudes towards learning will improve prior to seeing an increase in achievement). And, of course, student achievement is and will continue to be measured to determine if the ultimate goal of the project has been achieved. Figure 2 shows an abbreviated theory-based outcome grid for the Delaware Challenge Project.

Excerpt 4 [Anonymous 2 ]

This report summarizes the student responses to a test designed to measure attainment of the objectives in the Microbe Detectives (MD) unit portion of the larger project. Results are presented for both an experimental group (the MD) and a comparison group, who discussed similar content, but did not use the new curriculum. The results are given for the total test score as well as for individual items.

Instruments:
Presents evidence for validity and reliability

The Measure

There were twenty-two items in the test, though many had more than one part, producing thirty-six separate responses that could be scored zero through three depending on the completeness of the answer. Content validity was assured by constructing items to match the educational objectives of both the MD curriculum guide and the state frameworks where the MD unit was piloted.

There were two forms of reliability estimated for the test. The first, Cronbach's Alpha, is a measure of internal consistency, demonstrating the value of using a single score, the total, to represent the children's level of competence. For objectively scored achievement tests the alpha is expected to exceed 0.80; the pretest alpha was 0.871 while the posttest alpha was 0.894. The second form of reliability calculated for this data was test-retest, a measure of stability over time. This measure allows for a mean change, but tests whether respondents remain in essentially the same order. For the MD group, the test-retest reliability was 0.699 and for the comparison group it was 0.756, both at acceptable levels.

The actual pre and post test is included in Appendix A at the end of this report.

Methodological Approach:
Describes use of a control group

Experimental Design

The independent variable in this study is group, either students who experienced the Microbe Detectives unit or students who were in comparable science classes. Students were not randomly assigned to these two groups, so equivalence was achieved using statistical methodology. The dependent variables were the total test score and individual item scores. The total test score could range from zero to 108 since there were 36 actual items, and each item could be scored from 0 to 3. There were 138 students in nine teachers' classes in the MD group and 119 students in five teachers' classes in the comparison group.