
	: Reports : Curriculum Development

Curriculum Development Stand-Alone Report 2 (Final)

Return to Curriculum Development Reports

NSF CCLI-A&I Pilot Project Evaluation

Return to Table of Contents

Immigration and Politics — PLSI 300, PLSI 475, PLSI 512

Module Effectiveness Surveys

Characteristics of Participants

In May 2001, a total of 50 students enrolled in PLSI 300, PLSI 475, and PLSI 512 completed the Immigration and Politics Module Effectiveness Survey. Thirteen students in PLSI 300, eleven in PLSI 475, and twenty-six in PLSI 512 performed the same module exercise and participated in this evaluation. As students' ratings of module satisfaction, impact on learning, and self-ratings of prior experience on five course-related dimensions did not vary by course, the following analysis was conducted on all 50 participants as a group.

Forty-six (92%) participants provided complete demographic data. Students ranged in age from 19 to 50 (M=26.2, SD=7.6), and 24 (51%) were female. Of 46 who provided ethnicity data, 29 (63%) were white, 10 (22%) Hispanic, 5 (11%) Asian, and 2 (4%) African American. A majority (77%) were employed at least part-time and worked from 8 to 70 hours per week (M=27.3, SD=12.6). Juniors comprised 46% of the class, 40% were seniors and 10% were graduate students. Political Science majors made up 50% of the class, another 33% were Urban Studies majors, and 70% of participants said the primary reason they enrolled in the course was to fulfill their major or minor degree requirements.

Although students' self-rated competence on each of five course-related dimensions varied considerably, 82% of participants felt they had enough prior computer experience to use the module effectively. Most students (93%) characterized their prior Internet experience as medium to high, while 55% rated their before-class data analysis experience as low. Additionally, more than 90% of participants reported low pre-module exposure to ArcView.

Frequency of Use

All participants of the Immigration and Politics module performed the exercise once during an hour-long in-class presentation and demonstration. The step-by-step exercise was facilitated by the instructor, but students were encouraged to pursue self-paced learning through optional exercises and additional electronic resources. No analysis was conducted on frequency of use.

Student Satisfaction

Students reported generally high satisfaction with the module's overall quality (M=4.24 SD=.77) as measured by response to Q11. Students more strongly agreed they had received sufficient instruction to perform the module exercise effectively (M=4.41 SD=.73), which they felt was highly applicable to the course material (M=4.36 SD=.63) and technically easy to perform (M=4.32 SD=.65). Most students (86%) also believed the module's purpose was clearly integrated with the goals of the course (M=4.20 SD=.67). Overall, student satisfaction did not vary by sex, age, class level or ethnicity.

Students who found the module exercise technically easy to perform felt they received sufficient instruction to use it effectively (r=.67 p<.01), and were highly satisfied with its overall quality (r=.45 p<.01). These students were also among those who felt the exercises were highly applicable to course material (r=.51 p<.01) and well-integrated with the goals of the course (r=.55 p<.01).

Impact on Learning

Although students overall agreed the module increased their confidence in accessing new technology (M=4.10 SD=.74) and improved skills relevant to career goals (M=3.74 SD=1.10), those who reported the exercise enhanced their interest in political science or made the required course work more engaging were most likely to report it had a positive impact on learning (r>.81 p<.01). Students only moderately agreed the module motivated them to learn more about computers (M=3.65 SD=1.00). Total impact on learning scores did not vary by student sex, but older students tended to rate the module's learning impacts somewhat higher than younger students (r=.34 p<.05).

Students' self-ratings of their pre- and post-class competence on five dimensions as measured by Q13 is illustrated in Figure 4. Before and after ratings of competence with data analysis and ArcView increased from low to medium on average. Students indicated less dramatic increases in Internet experience, as most students (93%) reported medium to high Internet experience before the module-enhanced course. An overall before and after competency rating, derived by summing Q13 scores on all five dimensions, revealed students' mean self-appraisals increased from 9.28 (before this class) to 12.14 (after this class) on a scale from 5 (lowest competence) to 15 (highest competence).

Students most likely to report the module increased their confidence in accessing new technology were those who found the module technically easy to use (r=.44 p<.01) or who felt they received enough instruction to use it effectively (r=.44 p<.01). Perhaps not surprisingly, students who reported these higher gains in confidence also believed the exercise made course work more engaging (r=.58 p<.01) and motivated them to learn more about computers (r=.53 p<.01). Students who strongly agreed the module made course work more engaging were also more likely to report the exercise enhanced their interest in political science (r=.65 p<.01) or improved skills relevant to their career (r=.45 p<.01).

Strengths and Weaknesses

Only 17 of 50 participants (34%) provided open-ended comments regarding the module's major strengths. Of these, 7 (41%) found the exercise easy to learn or understand, "clear and simple," and "very well put together." Five students (29%) mentioned the tool's usefulness and its variety of applications, while another 4 (24%) focused on the visual features and speed of ArcView software itself. One student felt the module handout was especially "well prepared and effective."

Thirteen (26%) students provided feedback regarding module weaknesses. Of these, 5 (38%) students cited conditions they felt limited their use of or access to the module, including insufficient lab time or space, inability to access the tool from home, and the prohibitive cost of the technology. Another 4 students (31%) felt the number or steps, commands or toolbars made the exercise seem rushed, hard to absorb, or "a little intimidating." Three students (23%) found the resource difficult to apply to their own project or topic, and one student felt the module ran too slowly in the network.

Module Relevance

One close-ended and two open-ended questions added to the Immigration and Politics survey provided additional feedback regarding the module's relevance to the course. All 39 students (78%) who responded to Q23 believed the exercise supported other components of the class, and 23 (59%) provided open-ended comments to support their views. Among reasons offered as to how the module supported other class components, over half of participants (52%) felt the module strengthened or enhanced their understanding of fundamental course concepts, such as political outcomes, data analysis, and data presentation. Again, many of these respondents (75%) focused primarily on the visual aspects of the tool. Five students (22%) felt the exercise supported class components by reinforcing and demonstrating material read or presented in lectures. Finally, 4 students (17%) reported the exercise had given them skills or needed direction for completing class projects.

Students were also asked to indicate what would make the module exercise more relevant to them, and 17 of 50 students (34%) provided responses. Of these, a majority (65%) felt they needed more training, time or exposure to the tool, and 4 of these students expressed interest in applying the new technology to their own data or class projects. Four students (24%) said the content or specific subject matter of the exercise was not relevant or interesting to them personally, and one felt the module should be made available for home use.

Content and Function

One close-ended and two open-ended questions added to the survey helped assess student response to the module's content and function. While 6 of 45 (13%) students reported they had encountered problems that prevented them from doing the exercise, none of these problems was related to the module itself.

Thirty-three students (66%) indicated which aspects of the exercise most interested them. While students mentioned a range of features, a majority (67%) found the visual and mapping capabilities of ArcView most novel and satisfying. Eight others (24%) indicated the hands on application of new technology, such as the opportunity to incorporate GIS results into reports, had the most appeal.

Seventeen students (34%) indicated which aspects least interested them. Of these, 6 (35%) disliked the step-by-step instructional method used to present the material. Others characterized the module's least interesting aspects as "precinct voting maps," "Stataquest," "technical stuff," and "very complicated toolbars." Overall, students' least favored aspects were highly specific and individual.