NSF CCLI-A&I Pilot Project Evaluation
Return to Table of
Contents
Previous Page
Methodology
Data collection for the evaluation of three NSF CCLI-EMD
modules implemented in Fall 2000 and one module implemented
in Spring 2001 took place from November 2000 to May 2001.
Four data collection strategies were proposed by the project's
Principal Investigator Professor Richard LeGates, modified
for each module by PSIRUS team faculty, and developed and
implemented by PRI. Four sources of data were used. They included:
- Student Evaluations of Teaching Effectiveness (Close-ended
SETs)
- Student Evaluations of Teaching Effectiveness (Open-ended
SETs)
- Focus Groups
- Module Effectiveness Surveys
Student Evaluations of Teaching Effectiveness (Close-ended
SETs)
To compare students' general satisfaction with a module-enhanced
course verses a traditional offering of the course, evaluators
obtained close-ended responses to the Student Evaluation of
Teaching Effectiveness (SETs) from the semester in which modules
were implemented and from a previous (non-enhanced) offering
of the same course. These 14-18 item anonymous surveys, administered
to students at the end of a semester, vary by department but
are designed to promote high teaching standards among faculty
by measuring several dimensions of student satisfaction on
a 5-point Likert type scale. Complete instruments are included
in Appendix A, and the dimensions used for comparison are
marked with an asterisk. For this evaluation, mean ratings
of nine dimensions deemed relevant to the module enhancement
have been compared.
For Sustainable Urban Development (URBS 492), student evaluations
from Fall 2000 have been compared to those from Spring 1999.
For Media and Politics (PLSI 200), Fall 2000 student evaluations
have been compared to Spring 2000 evaluations. Mean comparisons
of student satisfaction on nine dimensions are presented here
for context only.1 Student evaluations were not used in the
Global Conflict (IR 326) or Immigration and Politics (PLSI
300/475/512) module analysis.
Student Evaluations of Teaching Effectiveness (Open-ended
SETs)
To gather qualitative data regarding the overall effectiveness
of the module-enhanced courses, evaluators obtained all responses
to three open-ended questions administered to students as
part of their SETs. These questions are typically presented
by a departmental staff member other than the instructor at
the end of each semester. The complete text of the questions
presented is included in Appendix A.
Only those responses from URBS 492 and PLSI 200 students
that informed the evaluation or provided a meaningful context
for assessing general satisfaction with the Fall 2000 module-enhanced
courses were analyzed. All module-specific comments or recommendations
are presented in detail.
Focus Groups
To obtain in-depth and specific feedback on the modules'
effectiveness, three semi-structured focus groups, facilitated
by PRI Project Coordinator Diane Godard, were conducted with
a sample of students from three of the four module-enhanced
courses during regularly scheduled class time. No instructors
were present at the sessions, no student identification data
was gathered, and participants were assured their comments
would remain confidential. Sessions were not audio taped.
PRI Research Assistant Pernille Hansen took notes for all
three sessions. A guideline for the discussion groups was
prepared in advance by evaluators and modified for each module
based on PSIRUS faculty input. Appendix B includes the complete
Focus Group Guideline.
On Wednesday evening, November 29, 2000 a 40-minute focus
group of twelve randomly selected URBS 492 students was held
in a private classroom on campus. No students declined participation.
A 40-minute focus group of six PLSI 200 students was held
Friday, December 8, 2000 in a private classroom. Due to low
attendance, all students meeting for their regularly scheduled
session that morning who had used the module during the semester
were asked to participate in the PLSI 200 session.2 No students
declined. A brief 25-minute in-class focus group of eight
of the nine students enrolled in IR 326 was conducted Thursday
morning, December 14, 2000. One student declined participation.
Focus groups were not held with students who participated
in the Immigration and Politics module. Detailed accounts
of the focus group sessions were transcribed into a database
shortly after they were held.
Module Effectiveness Surveys
A two-page 21-item self-administered questionnaire was developed
by PRI and modified for each PSIRUS course. The survey included
12 close-ended questions to capture students' perception of
module effectiveness on various factors, one open-ended item
regarding module strengths and weaknesses, one close-ended
item capturing frequency of use, and 7 demographic items.
Students in PLSI 300, PLSI 475 and PLSI 512, who did not participate
in focus groups, received a survey modified to include five
additional qualitative questions drawn from the focus group
guidelines. Appendix A includes all four versions of the Module
Effectiveness Survey administered.
Module Effectiveness Surveys were administered to all twenty-seven
URBS 492 students in class Wednesday evening, December 6,
2000; twelve PLSI 200 students who indicated they had used
the module during the semester and who were in class Wednesday
morning, December 13, 2000; and to all nine IR 326 students
in class Thursday, December 14, 2000. These surveys were distributed
to students along with Fall 2000 SETs by PRI staff Diane Godard
and Pernille Hansen. On May 8 through May 10, 2001, thirteen
PLSI 300, eleven PLSI 475, and twenty-six PLSI 512 students
completed module effectiveness surveys immediately after an
hour-long presentation of the Immigration and Politics exercise.
Surveys took approximately 12 to 18 minutes to complete.
1As students enrolled in pre-module courses may differ from
current semester students on a number of uncontrolled variables,
they do not represent a valid control group. For instance,
the Fall 2000 presidential elections may have contributed
to differences in students' response to course material. Additionally,
students were not matched on characteristics, such as age
or GPA, which may impact their overall satisfaction scores.
For these reasons, significance tests were not computed for
pre- and post-module comparisons.
2PLSI 200 module exercises were presented as a voluntary,
extra-credit activity this semester. Teaching assistants estimated
this sample represented about one third of actual module users.
Return to Table of
Contents
Next Page
|
