Curriculum Development Annotated Report Excerpts

Excerpt 1 [Oregon State University]

Interpretations & Conclusions:
Presents unanticipated findings

One preliminary finding that has been surprising is the strength of the effect an individual instructor evidently has on students' preferred choice of representation. This suggests to us that the instructor's role in mediating students' use of technology may be the single most powerful catalyst for change in calculus instruction.

(…)

Interprets results

Results suggest that students in the experimental classes at some institutions are at very low levels of calculus readiness. Care must be taken in analyzing data to take this into account. At a few of the test sites, measures have been administered to parallel classes taking a traditional approach to calculus.

Presents project weaknesses

Most of the criticisms regarding the text materials themselves were related to their draft form (misprints and typos, lack of index and answer key). These problems should not be underestimated, for the lack of an answer key turned out to generate much more extra work in the eyes of many of the pilot instructors.

(…)

Several of the pilot test sites reported that attrition from the experimental classes was far lower than the usual attrition from a traditional class (some as low as 0%). Ricks College reported a 38 to 33 attrition in the experimental class versus a 35 to 21 attrition on a traditional class (same instructor for both).

Excerpt 2 [Oregon State University]

Interpretations & Conclusions

Recommendations

High school teachers follow the textbook very closely, much more than college faculty. For calculus reform to impact at the high school level, teachers must be exposed to and prepared to use appropriate curriculum materials. Such training must begin with pre-service teachers through their high school and college calculus experiences.

Excerpt 3 [Oregon State University]

Interpretations & Conclusions:
Describes data limitations

Recommendations:
Presents future evaluation plans to address the data limitations

Students have done at least as well or better than national (high school) norms on almost every question. While this type of comparative student evaluation (traditional vs. experimental) is necessary, we feel it will be incomplete if we do not devise instruments which are sensitive to some of the new skills which we believe the experimental students have developed. In particular, students' use of numerical and graphical representations of functions seems to be dramatically increased. We will need to develop a non-calculator based instrument which would be appropriate to administer to both experimental and control groups.

Excerpt 4 [Oregon State University]

Interpretations & Conclusions:
Describes project improvement based on original evaluation

The first inservice workshop was held in summer 1990 and was less than satisfactory in many respects. As a result, our workshop format was thoroughly revised on the basis of feedback of our first group of pilot testers. We have found the subsequent workshops to be extremely successful.

Stakeholder Review & Utilization:
Discusses dissemination of findings

Included as part of this report is a copy of the Spring 1992 Calculus Currents Newsletter. This special issue is dedicated to the Oregon State University Calculus Project and will be disseminated to approximately 2500 people across the country by PWS-KENT publishers.

Excerpt 5 [Inter-American University of Puerto Rico, San Juan]

Interpretations & Conclusions:
Presents conclusions and supporting evidence

The results of the evaluation demonstrate the favorable impact of the project for the students. In general, it can be concluded that the goal of the project was completely attained. The overall findings show that students demonstrated a formative progress towards a more positive disposition, attitudes and beliefs about chemistry. Moreover, students’ entries in the reflective dairies throughout the year showed that they gained a gradual and formative understanding of the academic content (balancing chemical equations, mathematical computations, handling of chemical formulas and solution of problems) of the course which gave them self confidence and security in the laboratory. In addition, it evidenced students’ progress in their interpretations of laboratory experiments.

Generalizes from findings

The opinion of the effectiveness of the teaching techniques used by the professor indicated, from the students’ perspective, that the Discovery Approach has a great potential for stimulating student’s reflection about their learning process and allows for setting high expectations of improvement in those areas of the learning process that require further improvements. The evaluators strongly believe that all the data gathered points out to the fact that participating in a course taught with the Discovery Approach helps lessen negative feelings and frustrations most commonly found in introductory chemistry courses taught with the traditional method.

Excerpt 6 [Lehman College]

Interpretations & Conclusions:
Presents quantitative findings in table form

It has now been possible to follow up the first group who were in the program for four academic years, and the second group for three. As of the fall of 1997, the fates of the 44 in the experiment group and the 69 in the control group were as shown in the table below:

Presents balanced conclusions about findings

The first notable thing is that there is no striking difference between the two groups. A miracle did not happen here. We were hoping that exposure to our course might give such a general improvement in the students' problem-solving ability that their performance in every aspect of college work would show a measurable improvement, but there is no evidence for any such effect. The only difference that looks significant is, however, an important one, the number of students retained in the nursing major. If one takes as the null hypothesis that the average number of students surviving in the nursing program is just the average of the two groups combined, and these two percentages are just two that happened by chance to come out that different or more different for samples of n=44 and n=69, the null hypothesis is rejected with 93% confidence, not conclusive but not bad. In both groups, the sum of nursing and nursing-related Health Services majors is nearly the same, but the proportions are reversed, most of the experiment group being in nursing and most of the control group in Health Services. This strengthens the impression that the FYI program helped students who wanted a career in health professions to stay in nursing.

Excerpt 7 [Oregon State University]

Interpretations & Conclusions:
Relates conclusions to other studies

Generalizes

All of the trends indicated that the classrooms were less teacher-centered and students were taking more responsibility for their own learning as well as working together with their peers and helping each other learn. All of the teachers interviewed mentioned this pattern to some extent. These results are consistent with those reported in previous research (Dick, 1990; Heid et al., 1990; Steen, 1988; Swandener & Blubaugh, 1990; Wiske et al., 1988). Associated with a less teacher-centered classroom comes an increase in open-ended questions. Although research indicates that the increase in questioning by students may pose a threat to teachers’ predictability in teaching (Dick, 1990, Farrell, 1990; Rich 1991; Schofield & Verban, 1988), the tone of the responses from teachers in this study was that of excitement and enthusiasm for a less teacher-centered environment. These trends support the findings of Berenson (1990) and Farrell (1990), and suggest that the dynamics of classroom tends to shift to more discussion, inquiry, and cooperative learning.

Recommendations:
Draws on related literature to recommend further research

Past literature has indicated that inservice programs have often focused on altering teacher behaviors by familiarizing teachers with new technology and new instructional methods, but that insufficient attention has been given to teachers’ concerns on the actual use in their environment (Schultz, 1980; Wiske et al., 1988). Providing a chance for teachers to share their knowledge, experiences, and insights can help guide the desiging of curriculum materials, inservice programs, and support systems that will best equip teachers to handle the demands and opportunities that technology presents. Finally, in future investigations it is essential that we listen to all stages of preparation and integration of new technological tools, thus, enabling continuous fostering of the transition that is necessary for successful implementation.

Excerpt 8 [Lehman College]

Interpretations & Conclusions:
Specifies group comparisons

The difference between the grades in CHE 114 obtained by our students and the controls is striking. Below is a table comparing the grade distributions in both groups.

Describes quantitative analysis

The fraction of students getting grades of D or F among students taking the FYI program with our course was 18%, as compared with 45% for the controls. The mean grade of our students was more than half a grade point higher than that of the controls. This is statistically significant. Taking as the null hypothesis that the average distribution of grades was the sum of the two distributions, and these two averages for groups of size n=44 and n=69 was at least that different by chance, the null hypothesis is rejected with 98% confidence. Since it is necessary to get a grade of C or better in this course to stay in the nursing program, this directly influenced retention in nursing.

Excerpt 9 [Rensselaer Polytechnic Institute]

Interpretations & Conclusions:

Students were given a pre-test and post-test to determine their facility with cognitive robotics before and after the experiment. Unfortunately, the first experimental group did not improve to the same degree that the control group did. The instructor was able to identify several problem areas with the setup (detailed momentarily), improved the setup, and ran the experiment again with a new group of students. . . .

Describes control and logistical problems in first pilot of treatment

There were a number of factors which apparently affected the performance of the first experimental group. First, no effort other than random assignment was made to match the groups for programming experience — as it turned out, all of the control group's students reported some experience with C (albeit not Interactive C), whereas only one of the experimental students did. Since this background is vital for some portions of the task, this difference represented a significant confound. Also, mechanical problems with the setup led to poor communication with the students. Though the instructor was able to see and hear them reasonably well, and so assumed that they could also see and hear him, this was not the case. . . . Particularly problematic were the lighting, which was both insufficient and poorly placed, such that glare obscured important material, and the volume and placement of the speakers (which could not at the time be changed; because of feedback we now have the ability to control them), and the poor quality of our scan converter, which displayed the computer video to the students. Because of these factors, students had great difficulty reading the computer screen, and all diagrams and handwritten notes were well nigh unintelligible. Also, the instructor discovered that there is a degree of skill required to use the technology most effectively in teaching, particularly in choosing the angles at which to present demonstrations such that the students could actually see the relevant material. Though obvious when dealing with students in person, choosing the correct angles is challenging when dealing with cameras. With these lessons learned, we engaged in the task of improving the setup to run the experiment a second time. . . .

Describes modifications for second pilot

In the second experiment, two of the three students had prior experience programming in C. Before the experiment was run, the students were tested for logical aptitude using the Raven Progressive Matrices test, which was later found to show a remarkable correlation between test scores and aptitude for the experimental task. For example, the student who scored a 61st percentile score on Raven improved 6 points on the experimental post-test, while the student who scored a 95st percentile on the Raven test improved 9 points from pre-test to post-test.

For the experiment setup, more care was taken in the placement of lighting equipment to avoid the cameras picking up excessive glare. A higher quality scan converter was purchased to make sure the text on the computer screens was transmitted with a resolution high enough to be easily visible. A digital camera was used to transmit document images, which allowed for easier reading by the students. This, plus lessons that the instructor learned from the first experiment in teaching technique, allowed for a teaching experience that very closely resembled the instructor actually being physically present.

Excerpt 10 [Anonymous 9]

To thoroughly compare the response of the experimental and control groups (hereafter referred to as the treatment groups) to the corresponding learning approaches, the following questions were investigated and are answered herein:

Interpretations & Conclusions:
Presents guiding questions for analyses

1. Were the treatment groups originally equivalent in ability (entering GPA) before any treatment was administered?
2. Is there a difference in achievement (final examination scores and course grades) between the experimental group and the control group overall (i.e., regardless of GPA level)?
3. To what extent is achievement (without regard to treatment group) a function of ability (entering GPA)?
4. Is the difference in achievement between the experimental group vs. control group the same for students in different entering GPA groups or levels? In other words, does the treatment have a more significant effect on certain ability levels?

Addresses equivalence of treatment and control groups

In Table 3 and Figure 4, the results of a one-way analysis of variance of entering GPA's are presented. The difference in GPA's between the experimental and control groups had a probability of .59 of occurring by chance. Therefore, the difference is not statistically significant. In fact, Figure 4 shows clearly that the GPA means for the experimental and control groups were very similar. Clearly, the experimental and control groups were equivalent at the beginning of the study. This answers question number one above and strengthens the remainder of the analysis considerably because it establishes that the groups, while not randomly assigned, were very similar before they received the statics course and homework treatments. Therefore, if they are different after a treatment is administered, it is safe to say the homework treatment caused the difference.

Figure 4. Least Squares Means for Treatment vs. GPA

Table 4 shows the results of a two-factor (2 x 5) analysis of variance for final exam scores in the engineering mechanics classes. The results of the analysis for the course grades were almost identical to the results for final examination scores; thus, only the results for the comprehensive final examination is presented in this paper. The two factors were homework treatment (experimental or control) and entering ability (GPAGROUPS) from low to high. In addition to presenting the main effects of the treatment and ability, Table 4 presents the interaction effect of the combinations of treatment with ability that cannot be explained by either factor alone. There was not a significant main effect due to the treatment (P = .543), thus the answer to question number two for the study. A very significant main effect (P = .000) due to ability or entering grade point average (GPAGROUP) was found. This answers question number three for the study. Finally, Table 4 shows a significant (P = .043) interaction effect on final exam scores for homework method with ability and answers question number four.

Presents further analyses to clarify interaction effect

To adequately understand these results, especially the interaction effect of method and ability, the means for the analysis have been computed and plotted and are presented in Figure 5. Figure 5a shows that the experimental group did better on the final exam than the control group, but not significantly. Figure 5b demonstrates that the final exam performance is highly related to the previous GPA of the students. Figure 5c compares the final exam performance of the experimental group to the control group students by each ability (entering GPA) group. This plotted interaction is significant and shows that for those 34% of students whose GPA is either below 2.24 or above 3.50, the conventional method of instruction produces better final examination scores. For those 66% of students whose GPA is between 2.25 and 3.49 (the three middle GPA groups) the treatment produces better final exam scores. In other words Fig. 5c shows that the exam scores are significantly better for the 2.25-3.49 group when they receive the homework treatment. This illustrates what was presented in a previous paragraph where the interaction effect for homework method and entering ability was described as having a significant probability of P = .043, smaller than the conventional standard of =.05.

In summary, the groups were equivalent at the beginning of the study in average GPA, therefore, one can be reasonably sure that differences observed at the end of the course were due to the experimental variables. There is no significant difference in achievement as measured by final examination scores and course grade averages between students receiving the homework treatment and those receiving the conventional homework treatment when the overall population is considered. However, the non-significant difference favors the homework treatment.

Additionally, achievement differences due to the treatment vs. control were not the same for all ability levels. The large middle range of grade point averages (2.25 to 3.49, or 66% of the students) perform significantly better on the final exam and in the course grade if they receive the treatment. Clearly, the outer GPA groups responded differently than the middle ranges, and definitive explanations for this occurrence are difficult to formulate. It may be hypothesized that the lower GPA students are overburdened by the additional learning curve required by the software and that the higher GPA students have developed successful study methods and are encumbered by the formality of the treatment approach. At any rate, a large middle section was shown to respond well to this pedagogical instrument as compared to the traditional homework approach.

(A) Treatment vs. Final	(B) GPA Groups vs. Final

(C) Treatment vs. Final for GPA Levels

Figure 5. Least Squares Analysis

Excerpt 11 [University of Minnesota-Twin Cities]

Interpretations & Conclusions:
Concludes that high participant satisfaction is not matched by improved outcomes (e.g., pass rates)

Summary

After two years of offering computer-mediated instruction the General College has been able to effectively integrate computer-mediated instruction into the developmental mathematics program. For many students, the opportunity to study mathematics through computer-mediated instruction, as opposed to lecture, has been greatly appreciated and has contributed to an improvement in their attitudes towards mathematics and increased confidence to succeed in mathematics. Challenges remain; such as working to improve pass rates in both the computer-mediated and lecture developmental mathematics courses and in subsequent credit-bearing courses.

Excerpt 12 [Kettering University]

Objective 1: Provide tools for material and process selection

Evaluator's Assessment: This objective has been fully met.

Interpretations & Conclusions:
Presents future evaluation plans

Recommendations for Improvement: There are no recommendations for improving Objective 1. However, Kettering University faculty will continue to evaluate the quality and condition of lab equipment, and will solicit feedback from students in the process.

Presents conclusions and supporting evidence

Evidence: Evidence for assessing Objective 1 was collected via the MFGE 375 Student Survey that was administered to students who had completed the class. In addition to asking questions about the quality of the equipment purchased as a result of the NSF grant, comparison data was collected from MFGE 375 students who used the lab in an earlier class when it contained older equipment.

Data from the MFGE 375 Student Survey indicate agreement about the superiority of the new equipment (CES material and process selection software, optical metallurgical microscope and software, metallurgical grinding wheels, and atmosphere controlled furnace). On a scale of 1.0 to 5.0 (1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree), students gave the following responses:

• Compared to when I took MFGG 370, there has been a significant improvement in the quality of the equipment available in the metallurgy laboratory.
  
  Average: 3.75
  Standard Deviation: 0.71
  
  
• The lab equipment used in MFGE 375 is more up-to-date than the equipment I used when I took MFGG 370.
  
  Average: 3.88
  Standard Deviation: 0.83
  
  
• The lab equipment in the metallurgy lab is generally better than that in other labs at Kettering University.
  
  Average: 3.13
  Standard Deviation: 0.35

In contrast to the generally positive responses above, comparison data from the MFGG 370 baseline data prior to the purchase of new equipment suggests that students did not believe the old lab equipment was adequate. For example, students gave the statement "I would describe the equipment used in the metallurgy lab during MFGG 370 as up-to-date" an average rating of only 2.58 (Standard Deviation: 1.12) compared to 3.88 (Standard Deviation: 0.83) in statement 2 above. For the statement "The lab equipment in the metallurgy lab when I took MFGG 370 was generally better than that in other labs at Kettering University at that time," students response was only 2.69 (Standard Deviation: 1.04) compared to 3.13 (Standard Deviation: 0.35) in Statement 3 above.

Open-ended responses from the MFGE 375 Student Survey (using new equipment) and the MFGG 370 Baseline Survey (using old equipment) supported these data. While no students indicated their dissatisfaction with the new equipment by providing comments, below are statements about the lab equipment that existed prior to the project:

"The lab equipment was good enough to perform the lab experiments but it was antiquated and in ill repair."

"The lab needs desperate help."

"Most of the equipment other than some of the polishers and cut-off saws is very old. I think that there really needs to be some newer equipment in the lab. We especially need a new tensile test machine."

.

Objective 3: Increase teamwork and communication skills of students

Evaluator's Assessment: This objective has been fully met.

Recommendations for Improvement: There are two recommendations for improving Objective 3: 1) Place more emphasis on oral communications; and 2) consider regrouping students at mid-term instead of keeping students in the same group for the entire semester, to allow for greater interaction among students in the class.

Evidence: Objective 3 was measured by several means: independent evaluator review of students' work (project reports, journals, team assignments), observation of students and teams by course instructors, peer and student self-evaluations, MFGE 375 Student Survey, and the MFGE 375 Small Group Instructional Diagnosis conducted by Kettering University Center for Excellence in Teaching & Learning. Data from the MFGE 375 Student Survey indicate agreement about the effectiveness of MFGE 375 in enhancing the communications and teamwork of students enrolled in the class. On a scale of 1.0 to 5.0 (1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree), students gave the following responses:

• MFGG 375 has made a significant improvement in my communication skills.
  
  Average: 3.88
  Standard Deviation: 0.35
  
  
• MFGG 375 has made a significant improvement in my teamwork skills.
  
  Average: 4.25
  Standard Deviation: 0.71

An analysis of students' journals and projects indicate significant improvements in their ability to communicate and to work in teams. The MFGE 375 Small Group Instructional Diagnosis conducted with students enrolled in the class also found that MFGE 375 does a very effective job of emphasizing teamwork and increasing the capacity of students to work in teams. Several students commented that this course was "good" or "incredible" at meeting this goal, and that the grading scheme—with the entire grade dependent on team-oriented project work—reflected this. Students were also very supportive of team-oriented project work that includes a teacher evaluation of the team, peer evaluations, and self-evaluations. Several students commented that providing timely feedback, and helping students recognize their strengths and weaknesses, better prepared them for their coop placements.

The MFGE 375 Small Group Instructional Diagnosis also found that MFGE enhanced their writing skills. However, several students believed that more could be done to increase their oral communication skills:

Presents balanced conclusions about findings

"The class agreed that the journal writing and the format of project reports allowed them to focus on and improve their writing skills. There was a mixed reaction to your six-page limit on reports - some groups felt that this detracted from the quality of writing, while others felt that perhaps it required a more thought-out report, thus enhancing their writing quality. Also, since the groups had to reach consensus about what to include and how to write the reports, they felt their writing skills improved. The class agreed that this course has had little impact on their oral communication skills since there were no class presentations. Some students did note, though, that the in-class discussions have improved their public speaking skills to some extent."