Curriculum Development Annotated Report Excerpts

Excerpt 1 [American Institute of Physics]

Qualitative Analysis:
Describes methods used to identify trends and patterns in the data

By its very nature, most of the information we collected (journals, questionnaires, site visits) was qualitative and subjective, reflecting the attitudes and interpretations of students and faculty or of site visitors. Because many of the faculty and students expressed themselves well and described their experiences in interesting ways, it would be tempting to use our material to simply select anecdotes. Although much of our public summation of the evaluation will include testimony from the students in their own words, we wanted to be sure that the valuation did not simply consist of stringing together a sequence of quotes selected to put forward a particular spin. We therefore used our first readings of the journals to identify trends or patterns, then followed up whenever possible during site visits, and formulated hypotheses that could be systematically tested, often through computer-assisted analysis of the journals.

Excerpt 2 [Anonymous 1]

Quantitative Analysis:
Describes analysis of data about comparison groups

Responses were compared for the combination group and the rest, and in general, the combination group showed larger proportions of responses in the desired direction. That is, combination students tended to report more enjoyment of math, more confidence in their abilities, and more transfer of concepts into courses and situations outside the mathematics classroom than students who did not take the course. Combination students were also more likely to report using the Internet for mathematics and physics reference than regular students. Students did not differ on many items tapping mathematics processes.

Describes factor analysis to establish the validity of the instrument

In order to examine the integrity of the survey itself, an exploratory factor analysis was performed, resulting in a three-factor Varimax rotated solution. Prior to the factoring procedure, the valences of the negatively worded items were reversed (this was supported by negative item total correlations for these items) and all items were put on a standard Z score scale (mean=0 and standard deviation=1). Standardizing the scores allowed for use of the three frequency items, which were on a three-point scale. The instrument yielded an overall internal consistency (alpha) coefficient of .87, which is well within the accepted range. The first factor corresponded to transfer of mathematics into other situations. Factor II concerned attitudes toward technology, and Factor III corresponded to confidence in mathematics ability. (Physics items were excluded from the factor analysis.) This is an indication that the survey is tapping at least these three domains.

Excerpt 3 [Inter-American University of Puerto Rico, San Juan]

Quantitative Analysis:
Describes use of chi-square procedure

The grades obtained by the students in the discovery sections were compared with that of students taking the course in the traditional sections using the Chi-square statistical test. Students’ perception of the Reflective Diary as an assessment technique was also evaluated using the Evaluation of the Reflective Diary Questionnaire, which included quantitative and qualitative data.

Excerpt 4 [Purdue University]

Quantitative Analysis:
Describes results of F-tests

The 59-question survey we used in the evaluation was distributed to all students in Freshman Engineering as part of the ENGR 100 course. The surveys produced six categories comprised of a variety of questions. Category four was significant based on an F-test at a level <0.0001. No other categories were found to be significant, although there were individual items which were significant.

Addresses reliability of the data

Describes results of chi-square analysis

The reliability of these measures is seen in the Cronbach’s Alpha level for all size categories. Alpha levels range from 0.81 to 0.91, showing a high level of reliability for this data. The surveys for students enrolled in the orientation class or assigned to the control group were also analyzed using a Chi-square analysis for each survey item. Seventeen survey items were significant at <0.10 when comparing the control group to the orientation class.

Excerpt 5 [University of Michigan]

Quantitative Analysis:
Describes use of multivariate procedures to detect patterns in the data

Preliminary Multivariate Analyses. We have begun to formulate exploratory multivariate regression models that allow us to untangle the complicated causal threads that link student background, attitudes, behavior, and performance. Already certain promising patterns are emerging. Not only did new wave students receive higher average course grades than their traditional peers, this improvement was similar for all students. The new wave curriculum improved the grades of males, females, minority students, and engineering students alike.

Excerpt 6 [Oregon State University]

Qualitative Analysis

The goal of the data analysis was to identify recurring themes in teachers’ perceptions regarding graphing calculator use. A qualitative approach was determined to be most appropriate due to the open-ended nature of the interview. Formal analysis of the interview data was undertaken upon completion of the data collection. As discussed in Bogdan and Biklin (1982), this analysis began with developing preliminary coding categories for each of the sections of the interview protocol. These coding categories were formed by looking for patterns both between and within the individual interview transcripts. Between 20 to 30 codes were generated for each of the sections of the interview protocol in the preliminary analysis. Each unit of data (sentence or paragraph) was then marked with the appropriate coding categories. The marked data was then sorted using a word processor. A major trend was determined if the coding category represented the perceptions of more than 50% of the teachers. A minor trend was determined if the coding category represented the perceptions of 25% to 50% of the teachers. None of the trends were found to represent the perceptions of all of the teachers.

Describes methodological limitations

Although the qualitative approach used in this investigation allowed for a more detailed analysis of teachers’ perceptions of the impact of graphic calculators, the generalizability of these findings are limited by the methodology used. First, the use of volunteers directly limits generalizability of results and conclusions. Secondly, though the results appeared to reveal internal consistency of teachers’ responses, the use of interviews without direct observations of their classrooms hinders the credibility of the conclusions. The reader should keep in mind that the trends reported in this study represent only the self-reported perceptions of the teachers interviewed.

Excerpt 7 [Oregon State University]

Quantitative Analysis:
Presents quantitative findings

Workshop leaders scored averages between 3.57 to 3.91 on a 4.0 scale on the following characteristics; enthusiasm, presentation, preparation, rapport, encouraging problem solving, knowledge of calculus, calculators, and the teaching with calculators.

(…)

When asked if there were any changes in student learning, 77% of teachers reported an increased in student understanding. When asked about changes in student attitudes, 85% of the teachers indicated positive changes. When asked about students' experiences at the college level, 74% of the teachers reported positive responses of easy transitions and students being successful and prepared when entering college. About 27% reported that their students were allowed or had limited use of their calculators in college calculus courses. College courses which required the use of paper-and-pencil computations did not pose a problem for most students.

Excerpt 8 [University of Arizona]

Quantitative Analysis:
Describes effort to correlate student background variables to learning outcomes

Correlation with pre-course data

We searched for factors in the pre-course data that correlated well with success in the course. We calculated the correlation coefficients of the grades in the course with SAT/ACT scores, prior grades, and MRT placement. The highest correlation was with Math 110 grades (r = .481), for those who took 110. For those who placed out of 110, we only had the MRT placement code and not the test score, so we couldn't check if algebra skills correlated this well with success in the course.

Correlations with MIS 111 and with the SAT Math score (with ACT for those who didn't take the SAT) were much smaller. The corresponding values of r² are small, and together these factors account for only about one-third of the variation in course grades.

Grades in the course tended to be somewhat higher for students who had taken more mathematics in the past, but these differences were not statistically significant.

Compares and contrasts aggregate background data of course completers and dropouts

Students who dropped the course

We also tried to identify characteristics of those who did not complete the course. These students had similar profiles for the SAT/ACT and MIS 111. But students who dropped were less likely to have taken Math 110, and those who took it had a much lower grade. Of the 28 students who didn't finish, 13 took 110 with mean grade of 1.92 compared to 2.23 for those who finished and also took 110.

Table 15 - Correlation of Course Grade with Performance in Other Mathematics
Courses, the SAT and ACT
Math 128A Spring 2000

		Course Grade	ACT Math Score	SAT Math Score	Grade in MIS 111	Grade in Math 110
Course Grade	r	1.000	.301**	.118	.218**	.481**
	N	219	83	157	211	139
ACT Math Score	r	.301**	1.000	.684**	.067	.436**
	N	83	83	50	81	46
SAT Math Score	r	.118	.684**	1.000	.116	.153
	N	157	50	157	154	96
Grade in MIS 111	r	.218**	.067	.116	1.000	.393**
	N	211	81	154	211	136
Grade in Math 110	r	.481**	.436**	.153	.393**	1.000
	N	139	46	96	136	139

Table 16 - Comparison of Students Who Withdrew with Students Who Completed
the Course on Course Grades and Placement Scores
Math 128A Spring 2000

The completion rate (89%) was largely unaffected by the previous courses that students had taken. For each of the previous courses we asked about, between 85% and 95% of the students who had taken the course completed 128A.

Those who dropped reported a much lower comfort level with math (but a similar success rate) and, surprisingly, a higher comfort level with technology. Caveat: the number of students who dropped the course is small, so we must be careful about drawing conclusions from the percentages given.

Excerpt 9 [University of Minnesota-Twin Cities]

Quantitative Analysis:
Explains and interprets comparison data gathered from available student records

Withdrawal rates

The withdrawal rates during the two years of this study, the academic years from 1999-2001, are shown in Table 1. A student is classified as having withdrawn from a course if they withdraw between the end of the second week of class and the end of the tenth week of class. If a student withdraws prior to the end of the second week of class they do not appear on the university's official class list. After the tenth week of class, the student must be assigned a grade or be given an incomplete.

Pass Rates

A student is classified as having passed a course if he or she earns a final percentage of 65% or higher. A percentage of 65% is assigned a final grade of D- while a final percentage of 70% is assigned a final grade of C-. Students who withdrew from a course or were assigned incompletes were not included in the results.

The data in Table 2 includes results from students who stopped actively working in the course after the tenth week of the semester, which is the last week to withdraw from a course. Because some students stop actively working in the course but do not withdraw, the pass rates are lower than they otherwise might be. This also means that the withdrawal rates reported in Table 1 would be higher if every student who did not actively complete the course were to be considered to have withdrawn.

Pass Rates of Students who took the Final Exam

The pass rates for students who took the final exam are shown in Table 3. Students who took the final exam, in most cases, remained actively involved in the course until the end of the semester. By examining the pass rates of only the students who took the final exam, we gain some insight into how students perform if they remain active in the course until the very end. The differences in the Ns in Table 2 and Table 3 indicate the number of students who did not take the final exam for each course.

Excerpt 10 [Capital University]

Quantitative Analysis:
Describes and presents data from analysis of changes in students' attitudes.

Statistical Summaries Of Student Attitudes for AY 2001-2002

Evaluation of CSAC 245

Measures were taken regarding students' attitudes about Computational Science, Science, Computing, and Mathematics. Students responded to survey items using a 5-point Likert-type scale where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree; responses of NA or "don't know" were omitted from analyses. The survey was developed in collaboration with professional evaluators who were hired to evaluate the NSF CCLI Computational Science Across the Curriculum project. Thirty-one students completed the pretest and the posttest. A one-tailed dependent sample t-test followed by a measure of effect size (Cohen's d) was calculated for each pre-post item. A negative t-value indicates that the students' ratings' increased from the pretest to the posttest. All items that reflect a significant (p < .05) change in ratings appear in bold.

Data reported includes t values, p values, and effect sizes.

. . .

Limits the reporting of data to statistics that are appropriate for the very small sample size

Evaluation of Computational Environmental Science

Measures were taken regarding students' attitudes about Computational Science, Science, Computing, and Mathematics. Students responded to survey items using a 5-point Likert-type scale where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree; responses of NA or "don't know" were omitted from analyses. The survey was developed in collaboration with professional evaluators who were hired to evaluate the NSF CCLI Computational Science Across the Curriculum project. Four students completed the pretest and the posttest — this represents the entire class. Because the sample size is so small, only the mean pretest score and mean posttest score for each item will be reported here. All items in which there is an increase in rating from the pretest to the posttest appear in bold.

Survey Item	Pretest	Posttest
Generally, I feel secure about attempting computer science.	4.00	4.25
I study computer science because I know how useful it is.	4.25	4.50
Knowing computer science will help me earn a living.	4.50	4.50
I am sure I can do advanced work in computer science.	3.50	3.50
Generally, I feel secure about attempting mathematics.	2.75	3.00
I study mathematics because I know how useful it is.	4.00	4.00
Knowing mathematics will help me earn a living.	3.75	4.50
I am sure I can do advanced work in mathematics.	2.75	3.00
Generally, I feel secure about attempting science.	4.50	4.50
I study science because I know how useful it is.	5.00	4.75
Knowing science will help me earn a living.	4.25	4.75
I am sure I can do advanced work in science.	4.25	4.75
Generally, I feel secure about attempting computational science.	3.50	3.75
I study computational science because I know how useful it is.	4.25	4.25
Knowing computational science will help me earn a living.	4.25	4.25
I am sure I can do advanced work in computational science.	3.50	4.00
I have a good understanding of what computational scientists do.	2.75	4.25
It is clear to me how computational science is connected to other disciplines like math, sciences and computer science.	4.50	4.50
Computational science is relevant to real world issues.	5.00	4.25
I understand the methods of computational science.	3.25	4.00
I enjoy working in groups.	4.00	4.00
When I am working in a group, I am comfortable in a leadership role.	3.25	3.50
When I am working in a group, I usually participate actively.	4.00	4.00
When I am working in a group, I feel that I have important things to say.	3.75	4.25
I feel that my contribution to group work is valued by the other members of the group.	4.00	4.33