NSF CCLI-A&I Pilot Project Evaluation

Return to Table of Contents

Executive Summary

This report presents an evaluation of the implementation and student learning impacts of four web-based instructional modules developed by the San Francisco State University's PSIRUS team. This curriculum enhancement pilot project, funded in 1999 by a National Science Foundation Adaptation and Implementation grant (NSF CCLI-A&I), was fully implemented by Spring 2001 and independently evaluated by researchers at the Public Research Institute from November 2000 to May 2001.

The project's goal was to produce four computer-mediated instructional modules to enhance the curricula of Political Science (PS), International Relations (IR) and Urban Studies (US) undergraduate courses offered at SFSU. The modules were intended to require students to interact with web-based material and utilize new information technology in order to:

• teach fundamental social science concepts and skills, such as accessing, analyzing and communicating information, and to empower students to use these skills in subsequent coursework;
• provide engaging and challenging computer-based exercises to develop students' ability to critically assess empirical evidence and to explore their own values in relation to this evidence; and
• help students gain confidence and overcome fears of computers and scientific methodology through guided exposure to emerging technology.

To assess the success of the project's implementation and determine its impact on students, the evaluators conducted a four-part study.

1. Student Evaluations of Teaching Effectiveness (Close-ended SETs) from pre- and post-module course offerings were compared to measure students' satisfaction with module-enhanced classes and to provide an evaluative context.
2. Student Evaluations of Teaching Effectiveness (Open-ended SETs) from module-enhanced courses were analyzed to provide qualitative support for student satisfaction with both the courses and their modules.
3. Focus groups were conducted with a subset of module users in each course to provide in-depth understanding of students' response to the modules and to obtain specific feedback regarding their effectiveness.
4. Module Effectiveness Surveys, administered anonymously to all module users, captured students' perceptions of module strengths, weaknesses, overall effectiveness and impacts on learning. Survey data was also used to gather student characteristics and to assess how often students had used the modules during the semester.

Because results indicated the degree of implementation, students' satisfaction and perceived impacts on learning varied considerably by course, major findings of the evaluation are presented separately by module.

Sustainable Urban Development: This module was a series of mandatory web-based exercises implemented in Fall 2000 in Urban Studies Research Methods (URBS 492).

• Students reported high module usage: 50% used it at least eleven times in the 15-week period and 33% used it 15 or more times during the semester. The designated lab time and PSIRUS Center location of this course enhanced module usage and overall integration.
• Students in this module-enhanced class felt the course had motivated them to learn about and contributed to their knowledge of the subject somewhat more than students in a pre-module offering of the course.
• Although the class as a whole reported only moderate overall satisfaction with the module, over 75% believed the exercises were applicable to the course material, were well integrated with course goals, were not too difficult to perform and had increased their confidence in accessing new technology. Students who reported increased confidence in accessing new technology were among the most satisfied.
• Two thirds of students felt career-relevant skills and interest in social research had been enhanced by module use. In particular, students said that exposure to a wide variety of research tools had improved their communication and information access skills and had provided immediate and long-term academic and professional benefits. Frequent users were more likely to report these benefits.
• Self-reports of before- and after-class competence on five course-related dimensions increased markedly, and over 95% of students rated their after-class competence on 4 of 5 dimensions as medium to high. Frequent users reported the largest increases in competency ratings.
• Students generally felt the exercises were challenging enough to appeal to all academic levels, and neither satisfaction nor perceived impact on learning varied by sex, age, class level or ethnicity. However, the step-by-step instructional design had more appeal for students with "computer anxiety," while technically adept students felt exercises needed to present more visual and contextual information to increase clarity and relevance.
• Students overall felt the module facilitated the link from abstract concepts to practical application by reinforcing lecture and reading material through hands-on learning. Many also specifically cited its potential to provide an alternative mode of learning for students of different learning styles and abilities.

Media and Politics: This module, implemented in Fall 2000 in Political Science American Politics (PLSI 200), was a series of voluntary web-based exercises for which students received extra-credit.

• Students in this course reported relatively low exposure to the module, as users represented no more than 10% of the class and 50% had used the module less than 5 times during the 15-week period. One third indicated the technical problems they encountered represented a significant barrier to use. Other barriers cited by students included insufficient module-specific instruction or technical assistance, students' lack of regular computer access, and the voluntary nature of the exercises.
• Students disagreed over how the module should be integrated with the overall class structure. Though 33% of students felt module use should remain voluntary, 50% suggested the exercises be included as part of regular class assignments.
• Students in this module-enhanced class rated 8 of 9 course effectiveness dimensions higher than students in a pre-module offering of the course. Students in the current semester more strongly agreed the course had significantly contributed to their knowledge of the subject.
• Among users, 75% felt the module's purpose was well integrated with course goals, felt the exercises made coursework more engaging and were easy to do, and reported being satisfied with the overall quality of the module's content. Nearly all students found the content highly applicable to the course material.
• Students who felt the module made coursework more engaging, enhanced their interest in social research, or improved skills relevant to them were among the most satisfied, while those discouraged by its technical problems were least satisfied. Satisfaction did not vary by student sex, age, class level or ethnicity and was unrelated to frequency of use.
• Although students only moderately agreed the exercises had directly impacted learning or improved skills, self-reports of before- and after-class interest on five course-related dimensions all showed increases. Students generally felt exercises had enhanced their comprehension of and provoked thought about the concepts presented in the text, and 33% believed the module had improved their overall class performance. Impacts on learning ratings were unrelated to frequency of use.
• Despite the technical barriers encountered, students felt their computer skills were more than sufficient to use the module effectively, and the range of challenge presented in the exercises made the tool appealing to students of all academic levels. The module demonstrated a potential to engage those who may come to an introductory course with higher academic interest and expectations.

Global Conflict: This module was implemented in International Relations South and Southeast Asia Foreign Relations (IR 326) as a semester-long data retrieval and analysis exercise.

• Students reported moderate module use during the 15-week semester. Of those reporting usage, 60% had performed the module exercise 7 or more times in the period.
• Although students felt the module was not structurally integrated with other class components, all agreed the purpose of the comprehensive research and analysis exercise supported the course goals.
• The class as a whole reported only moderate satisfaction with the overall quality of the exercise, but all students agreed it was applicable to the course material and its purpose was clearly integrated with course goals. Students' satisfaction did not vary by sex, age, class level or ethnicity, but students who worked more per week tended to report higher satisfaction.
• Seventy-eight percent of students agreed they had received adequate instruction to do the exercise effectively and had been motivated to learn more about computers. All students felt their prior computer experience was more than sufficient to perform the exercise.
• Two thirds of students felt the exercise had enhanced their interest in social science research and increased their confidence in accessing information, and those who most strongly believed this were more likely to report it had improved skills relevant to their careers.
• Self-reports of before- and after-class competence on 4 of 5 course-related dimensions showed moderate increases and 75% felt the primary academic and professional benefit of the exercise was to help students conduct basic research, assimilate large quantities of data and analyze complex topics in-depth. Perceived impact on learning did not vary by age or ethnicity, but juniors and women tended to rate these benefits somewhat higher.
• Although 50% of students indicated the exercise highly appealed to them conceptually, 63% felt they had had enough prior exposure to the web-based tools introduced to perform the exercise without formal instruction and would have preferred exposure to a wider variety of tools and online resources.

Immigration and Politics: One component of this module, implemented in Spring 2001, demonstrated spatial analysis techniques through a comprehensive guided exercise using ArcView GIS software. Students enrolled in Scientific Inquiry in Political Science (PLSI 300), San Francisco Political Issues (PLSI 475) and Urban Politics and Community Power (PLSI 512) took part in the exercise.

• Although students performed this exercise only once late in the semester, 88% reported high satisfaction with the module's overall content. Over 85% believed the exercise was applicable to the course material, well integrated with course goals, technically easy to perform, and had increased their confidence in accessing new technology. Students who felt the exercise made course work more engaging, motivated them to learn about computers, or increased their confidence in accessing new technology, were among the most satisfied.
• Two thirds of participants felt the exercise had enhanced career-relevant skills, and 58% reported it had increased their interest in political science. Older students were somewhat more likely to report these professional and academic benefits.
• Self-reports of before- and after-class competence on five course-related dimensions showed moderate increases, with the largest gains reported for data analysis and ArcView experience. A majority of students (82%) felt they had enough prior computer experience to perform the exercise effectively, and over 85% of students rated their after-class competence on all 5 dimensions as medium to high.
• While students' overall satisfaction did not vary by sex, age, class level or ethnicity, older students were more likely to report the module exercise delivered higher overall improvements in learning.
• Although students overall believed the module exercise supported other components of the course, 38% cited limited access to the module as a primary weakness, and 65% felt increased exposure to the module would have enhanced its relevance.

Although findings indicated the extent to which each module had been implemented, operationally tested and integrated into the curriculum varied considerably by course, most students perceived the activities had delivered some academic or professional benefits. The modules appealed equally to students of all ages, class levels, ethnicities and prior computer experience, although students who felt the activities improved relevant skills or made coursework more engaging were among the most satisfied.

Return to Table of Contents

Next Page