NSF CCLI-A&I Pilot Project Evaluation

Return to Table of Contents

Previous Page

Conclusions

Although the extent to which each module had been implemented, operationally tested and integrated into the curriculum varied considerably by course, students overall responded favorably to the modules and most perceived the activities had delivered some academic or professional benefits. In general, modules appealed equally to students of all ages, class levels, ethnicities and prior computer experience, and those who felt the activities improved relevant skills or made coursework more engaging were among the most satisfied. As findings indicated the degree of implementation, students' satisfaction and perceived impacts on learning varied considerably by course, each module's results are discussed separately in the following section.

Summary of Findings

Sustainable Urban Development — URBS 492

Students in this course reported high module usage as 50% had used it at least 11 times in the 15-week period and 33% had used it 15 or more times. The designated lab time and PSIRUS Center location of this course enhanced module usage and overall integration.

Although the class as a whole reported only moderate overall satisfaction with the module, over 75% believed the exercises were applicable to the course material, were well integrated with course goals, were not too difficult to perform and had increased their confidence in accessing new technology. Students who reported increased confidence in accessing new technology were among the most satisfied.

Two thirds of students felt career-relevant skills and interest in social research had been enhanced by module use, and frequent users were more likely to report these benefits. Self-reports of before- and after-class competence on five course-related dimensions increased markedly, and over 95% of students rated their after-class competence on 4 of 5 dimensions as medium to high. Frequent users also reported the largest increases in these competency ratings.

Students generally felt the exercises were challenging enough to appeal to all academic levels, and neither satisfaction nor perceived impact on learning varied by sex, age, class level or ethnicity. Its step-by-step instructional design had more appeal for those with "computer anxiety," while technically adept students felt the exercises needed to present more visual and contextual information to increase clarity and make the concepts applicable elsewhere.

Media and Politics — PLSI 200

Students in this course reported relatively low exposure to the module, as users represented no more than 10% of the class and 50% had used the module less than 5 times during the 15-week period. One third indicated the technical problems they encountered represented a significant barrier to use, and others felt lack of module-specific instruction or technical support, lack of regular computer access, and the voluntary nature of the exercises deterred usage.

Although students found the module's content highly applicable to course material, they disagreed somewhat over how the module should be integrated with the overall class structure. One third of students felt module use should remain voluntary, but 50% suggested the exercises be included as part of regular class assignments. Among users, 75% felt the module's purpose was well integrated with course goals, believed the exercises made coursework more engaging and were easy to perform and reported being satisfied with the overall quality of the module's content.

Students who felt the module made coursework more engaging, enhanced their interest in social research, or improved skills relevant to them were among the most satisfied, while those discouraged by its technical problems were least satisfied. Satisfaction did not vary by student sex, age, class level or ethnicity and was unrelated to frequency of use.

Although students only moderately agreed the exercises had directly impacted learning or improved skills, self-reports of before- and after-class interest on five course-related dimensions all showed increases. Students generally felt exercises had enhanced their comprehension of and provoked thought about the concepts presented in the text, and 33% believed the module had specifically improved their overall class performance.

Global Conflict — IR 326

Students reported moderate module use during the 15-week semester. Of those reporting usage, 60% had performed the module exercise 7 or more times in the period. Although most felt the module was not structurally integrated with other class components, all agreed the purpose of the comprehensive research and analysis exercise supported the course goals.

Students reported only moderate satisfaction with the overall quality of the exercise, but all agreed it was applicable to the course material and its purpose was clearly integrated with course goals. Student satisfaction did not vary by sex, age, class level or ethnicity.

Two thirds of students felt the exercise had impacted learning by enhancing their interest in social science research and increasing their confidence in accessing information. Those who most strongly believed this were also more likely to report the exercise had improved skills relevant to their careers. Self-reports of before- and after-class competence on 4 of 5 course-related dimensions showed only moderate increases, and 75% felt the primary academic and professional benefit of the exercise was to help students conduct basic research, assimilate large quantities of data and analyze complex topics in-depth. Perceived impact on learning did not vary by age or ethnicity, but juniors and women tended to rate these benefits somewhat higher.

Seventy-eight percent of students agreed the exercise had motivated them to learn more about computers, and all felt their prior computer experience was more than sufficient to perform the exercise, but 63% felt they had had enough prior exposure to the web-based tools introduced to perform the exercise without formal instruction. These students felt the exercise could have exposed them to a wider variety of tools and more novel online experiences.

Immigration and Politics — PLSI 300, PLSI 475, PLSI 512

Despite their relatively low exposure to the module during the 15-week period, students in all three courses reported generally high overall satisfaction with the module's content. Over 85% believed the exercise was applicable to the course material, well integrated with course goals, not too technically challenging, and had increased their confidence in accessing new technology. Among the most satisfied were those who felt the exercise made course work more engaging, motivated them to learn more about computers, or increased their confidence in accessing new technology.

Two thirds of students felt career-relevant skills had been enhanced by exposure to the module, and older students were more likely to report these professional benefits. Self-reports of before- and after-class competence on five course-related dimensions improved moderately, with the highest gains in data analysis and ArcView experience. Most students felt they had enough prior computer experience to perform the exercise effectively, and over 85% of students rated their after-class competence on all 5 dimensions as medium to high.

Although most students felt the module exercise supported other components of the course, 38% cited limited access to the module as its primary weakness, and 65% felt increased exposure to the module would have enhanced its relevance.

Recommendations

The results of this pilot project evaluation reveal a number of opportunities for potential improvements in module development, project implementation, and evaluative activities. Three primary areas of concern for project administrators, module developers, and evaluators include overcoming barriers to module use, meeting challenges posed by diverse student expectations, and enhancing evaluation activities. The following recommendations address these three general areas.

Recommendation 1: Reduce or eliminate barriers to module use.

Evaluators identified several conditions that, to varying degrees, limited student access to or use of the modules. Some of these conditions, such as technical performance problems, might be reduced or eliminated by more thorough testing prior to implementation. Students' module access might also be facilitated by scheduling some portion of all module-enriched courses to take place in on-campus computer labs.

Additionally, instructors and teaching assistants should be prepared to introduce modules early in the semester and to foster students' ability to apply these new resources to their own class projects. Instructors might develop group or in-class activities to promote module use, encourage dialogue, and stimulate interest among students. Activities designed to increase interaction among module users would also help advance a primary objective of teaching students to present and communicate their research findings to others.

Recommendation 2: Design modules for a diverse student body with rising expectations.

Evaluators recognize that, to some degree, module usage, satisfaction and impact on learning ratings reflect a wide range of student expectations, academic level and previous experience. Thus, one major challenge for module developers includes the need to design modules that will appeal to students of highly diverse backgrounds and varying degrees of computer literacy.

To optimize the appeal and relevance of modules to all students, developers might focus on exposing students to the broadest range of novel resources, software and self-paced activities. Instructors might incorporate a combination of mandatory verses optional exercises, facilitated verses autonomous activities, or step-by-step verses exploratory tasks to increase students' confidence while holding their interest. Additionally, to keep pace with emerging technologies and to meet the rising expectations students, developers may need to continually maintain and upgrade their modules.

Recommendation 3: Improve evaluation activities.

Although the evaluation activities conducted to date have provided essential feedback regarding the development and implementation of four web-based module prototypes, evaluators caution that the results discussed in this report largely reflect student perceptions and should not be interpreted as objective measures. Future evaluation activities might include other methods of assessing whether module-enhanced courses deliver the academic or professional benefits intended, such as implementing a before- and after-class survey design or using test scores or other outcome data to assess impact on student learning.

Finally, this evaluation has been conducted with the cooperation and support of the PSIRUS team members, who devoted valuable instructional time to our data collection activities. In order to best coordinate evaluation activities to minimize classroom disruption, we recommend instructors consider evaluation activities as they develop their curricula. Scheduling evaluation activities into the curricula will make optimal use of limited instructional time, facilitate access to all module users, provide students with the opportunity to offer thoughtful feedback, and help ensure the validity and reliability of evaluation results.

Return to Table of Contents

Next Page