NSF CCLI-A&I Pilot Project Evaluation

Return to Table of Contents

Previous Page

Findings (cont'd)

Media and Politics — PLSI 200

Student Evaluations of Teaching Effectiveness (Close-ended SETs)

Of the nine dimensions compared from student SETs, eight dimensions were rated somewhat higher by students in the module-enhanced semester and one showed a slight decline in mean rating from Spring to Fall 2000. In both semesters, students strongly agreed the instructor showed an interest in the subject matter and encouraged critical thinking about the course topics. Students in the module-enhanced class agreed considerably more that the instructor was able to express ideas clearly and the course had contributed to their knowledge of the subject. In both semesters, students also generally believed the class was well organized and effectively taught; however, in Fall 2000, students agreed slightly less that syllabus clearly stated the course objectives, requirements and grading policies.

Table 2 provides mean scores and computed differences between semesters on all nine dimensions considered in this analysis. Scores range from 1 (strongly disagree) to 5 (strongly agree) and dimensions are ranked from highest to lowest score received in Fall 2000.

Table 2 Comparison of Student Evaluations of Teaching Effectiveness (Close-ended SETs)

Student Evaluations of Teaching Effectiveness (Open-ended SETs)

Of the sixty-one student sample of Fall 2000 open-ended SETs selected for review, 41 (67%) offered highly positive comments such as "my knowledge… and interest [in politics] has increased," and "[the class was] interesting and informative." Students' positive appraisals often centered on the instructor's enthusiasm, humor, knowledge and organizational skills. However, the instructor's unstructured lecture style was as often cited as a source of confusion for students who relied on the readings, study guides or discussion sessions to prepare them for exams. Fourteen students (23%) felt regular homework or more assignments designed to reinforce the text material would have helped their test performance.

Overall, comments from students who had used the module during the semester, although brief, were as positive as those who hadn't. Of the 61 comments reviewed, 13 (21%) referred directly to the Web-based extra credit assignments in their comments.³ Nine (69%) of these provided overall positive feedback, such as "the Website is very effective," "an excellent idea for an assignment medium," and "[it] helped a lot and should be kept as part of the class." However, students expressed disagreement over how integrated the module should be with the overall class format, as some felt they would have made better use of the exercises had they been mandatory, while computer access posed a barrier to use for others. Some students felt the exercises were informative and educational, and should be required assignments, while others seemed to value them more as an opportunity to earn extra credit.

Focus Groups

The majority (75%) of students who took part in the PLSI 200 discussion group had enrolled in the course to fulfill their General Education or Political Science major requirements. Two international students expressed an interest in learning more about American government. All participants were unaware of the module component of the course prior to enrollment.

Module Relevance

As the module was offered as an extra credit activity in this course, students expressed considerable disagreement as to whether or how the module supported other components of the class. Students did agree the module generally enhanced their comprehension of concepts presented in the text, but half the students found exercises varied considerably in their relevance. One student stressed the need for more structural integration, suggesting students may be tempted to quickly complete the exercises for the extra credit, rather than for developing concepts. Another felt the module would be more useful if exercises were introduced as weekly assignments by the instructor or the teaching assistants. Several others concurred the teaching assistants should be more involved with and informed about the module to help motivate students to use it. Most agreed the module was an interesting and engaging resource which was underutilized.

Content and Function

Most discussion in this session centered on the technical difficulties students experienced accessing the module, which overshadowed questions regarding specific content, applicability and usefulness of the exercises. Initially, several students had difficulty with passwords, frozen screens and improperly recorded extra credit points, although one student experienced no technical problems. Despite these barriers, students persisted in using the module, as all generally agreed the exercises were highly stimulating and motivating. Two students specifically asserted they felt the module's content had improved their test performance and grades, and although the difficulty of the exercises varied considerably, most found the content generally fun, engaging, and thought-provoking. One stated, "It definitely reinforced what was introduced in the book."

All students felt their computer skills were more than sufficient to use the module effectively, technical problems aside, and the wide range of academic difficulty in the exercises made the tool potentially useful to all student levels. This range of activities was cited as one of its primary appeals. Students said they spent an average of 45 minutes per exercise, which for many was time well spent.

However, students hesitated to assess the tool's effectiveness as a whole, stating they had been unable to use several components that performed poorly. Technical problems also prohibited some students from completing exercises in the order intended. As a result, although about half the group felt the activities were somewhat disjointed, most attributed this sense of disorder to the module's technical difficulties rather than to its design.

Module Effectiveness Surveys

Characteristics of Participants

Surveys were administered to 12 students enrolled in PLSI 200 in Fall 2000 in attendance on December 13, 2000 who had reported using the module at least once during the semester. Participants ranged in age from 18 to 42 with a median age of 20.5 (M=23.3, SD=6.57), and 10 participants (83%) were female. Seven students (58%) identified themselves as exclusively white, 3 (25%) were Hispanic, and 2 (17%) were Asian.

Half of the students surveyed were currently employed at least part-time and worked from 20 to 60 hours per week (M=28.8, SD=15.8). All undergraduate class levels were represented, but juniors and seniors comprised 83% of the sample. Only one respondent was a political science major, as the SFSU General Education (GE) requirement this course satisfies attracts a broad range of students. Majors represented in this sample included biology, business, broadcasting, journalism, nursing and psychology, among others. Additionally, 2 (17%) students, both sophomores, had yet to declare a major.

Students' self-assessment of their prior interest in five course-related subject areas did not vary by class level or ethnicity, but older students indicated somewhat higher interest in research (r=.62 p<.05). While all respondents characterized their prior Internet interest as medium to high, students typically reported low to medium interest in US History (67%), Political Science (75%), Research (92%), and Data Analysis (100%).

Frequency of Use

Students in PLSI 200 reported relatively low exposure to the module during the semester, averaging 5-6 times in the 15-week period. One of twelve students (8%) reported using the module fifteen or more times. Frequency of use did not vary as a function of sex, age, ethnicity, but students who worked more hours per week tended to use the module somewhat less. In support of focus group data in which some students attributed their lack of use to the technical difficulties encountered, those who used the module less frequently agreed more that technical problems had discouraged them from doing the exercises (r=-.63 p<.05).

Student Satisfaction

Generally, students reported moderate satisfaction with the overall quality of the module's content (M=3.75 SD=.75) and felt the module made course work somewhat more engaging (M=3.75 SD=.75). Students more strongly believed its content was applicable to the course (M=4.33 SD=.65), its purpose clearly integrated with course goals (M=3.92 SD=.90), and the exercises were not too difficult (M=3.92 SD=.51). However, students rated satisfaction with the module's technical aspects considerably lower (M=2.83 SD=1.27), which may have presented a barrier to overall satisfaction. Overall satisfaction did not vary by sex, age, class level or ethnicity.

Students who felt the module was applicable to course material tended to be most satisfied with the overall quality of its content (r=.74 p<.01). Those most satisfied with the overall quality of content also felt the module made required course work more engaging (r=.68 p<.05) and enhanced their interest in social science inquiry (r=.60 p<.05). Students who found the module very easy to use reported the most satisfaction with its technical quality (r=.69 p<.05), but one third of respondents felt they needed more instruction to use the module effectively.

Impact on Learning

Respondents only somewhat agreed the module motivated them to learn more about politics (M=3.50 SD=.80), enhanced their interest in social science inquiry (M=3.25 SD=.75), or improved skills they deemed relevant (M=3.25 SD=.87). Overall, these ratings did not vary by frequency of use, sex, age, class level, or ethnicity.

Student self-ratings of their pre- and post-class interest on five course-related dimensions as measured by Q13 is illustrated in Figure 2. Although students' before and after interest level on all dimensions rose only slightly, students reported relatively high pre-class interest on all dimensions except data analysis. An overall interest measure derived from summing responses to Q13 revealed students' mean self-appraisals increased marginally from 9.83 (before this class) to 10.64 (after this class) on a scale from 5 (low interest) to 15 (high interest). These interest level increases were unrelated to students' overall satisfaction with the module's content or to appraisals of the module's impact on learning and did not vary by participant characteristics.

Students who did believe the module enhanced their interest in social science also felt the module motivated them to learn more about politics (r=.83 p<.01) and helped to improve skills relevant to them (r=.87 p<.01). Those who felt the module improved relevant skills were also highly satisfied with the overall content of the module (r=.66 p<.05).

Strengths and Weaknesses

Ten respondents provided open-ended comments regarding the module's major strengths. Of these, students most often mentioned the module's easy, fun and interesting activities, and 2 of 10 students (20%) specifically mentioned the tool's ability to "appeal to all learning styles" as its primary advantage. One student felt the module successfully reinforced course material, one especially valued the immediate feedback provided by the simulations, and another suggested the tool has the potential to engage students who may come to the introductory undergraduate course with somewhat higher academic interest and expectations.

The module's most common weakness, as cited by 6 of 8 respondents (75%), was its "technical problems." Specifically, students felt the module's links too often performed improperly, images did not correspond with instructions, and sign-in procedures were unnecessarily "long and confusing."

Figure 2

³Based on estimates from teaching assistants, about 10% (n=20) of students enrolled in PLSI 200 used the module at least once during the semester.

Return to Table of Contents

Next Page