Under-Represented Populations Embedded Plan 1

Teaching Traineeships for Undergraduate Women Evaluation Plan

This evaluation plan was prepared by the University of Rochester for the Teaching Traineeships for Undergradute Women project.

Table of Contents:

5. Evaluation
   • Evaluation Overview: Evaluation Purposes
   • Design: Methodological Approach, Information Sources & Sampling, Instruments, Data Collection Procedures & Schedule

V.  Evaluation

At the outset of this narrative, we stated five principal objectives of the proposed project: (1) to recognize and reward the demonstrated achievement of selected women sophomores and juniors in physics classes; (2) to assist them in developing teaching competency, leadership, and communication skills; (3) to make them participating and financially-compensated members of the teaching staff of the Department; (4) to increase the fraction of female instructors in physics labs to 50% overall with at least one per laboratory section; and (5) to improve the quality of instruction and learning in physics labs, through the presence of a second, trained teaching assistant in each section. As noted here, these objectives serve three important goals: increasing the long-term retention of undergraduate women in science, creating a gender-equitable learning environment in physics laboratories, and improving the quality of instruction in physics laboratories.

Evaluation of the project will focus in two primary directions. First, pertinent to objectives (1), (2), and (3) above, we will evaluate the impact of the project on the participant Trainees. Second, pertinent to objectives (4) and (5), we will evaluate the impact of the project on students in the introductory laboratory sections, co-taught by the Trainees. As a matter of secondary (but still significant) interest, we will evaluate the project from the perspective of the graduate TA's who are co-instructors with the Trainees and the faculty teaching the courses.

Critical questions in assessing impact on the Trainees concern that quality and usefulness of the various aspects of the training process, including summer orientation and academic year follow-up, and attitudinal changes brought about by participation in the project. Answers to questions in these areas will be sought through written surveys and personal interviews conducted by an external evaluator in both formative and summative stages of the project evaluation. Another critical area concerns the quality and improvement in the Trainees' teaching skills, which will be assessed partly through subjective measures but primarily through feedback from each Trainee's students and graduate co-instructor, as outlined below. [See Table V-1.]

To assess the project's impact on students in the introductory physics lab we plan to adopt the Small Group Instructional Diagnosis (SGID) methodology, which has been used at the Center for Instructional Development and Research at the University of Washington and is considered one of the most effective means of both assessing and improving instruction (Nyquist, 1994). In our implementation of this method, a team of advanced and highly-rated TA's in physics will function as evaluators. The team members will simultaneously conduct half-hour interviews with the students in each lab section, each evaluator meeting with a small group of students. The team will aim to elicit responses stating specific things the instructor is doing that help and hinder learning, as well as specific suggestions for the instructor. Assembling both written and verbal responses to these questions, the team will write a short evaluation and meet with the instructor. As the final step of the process, the instructor will thank the class for their input at the next class meeting.

Using the SGID format, the evaluation team will interview each Trainee's students once during the year. Each semester, however, all students will also have the opportunity to provide feedback through their written TA evaluations, which will be gathered, processed, and made available to the Trainees (as to all TA's) at mid-semester. The processing of these written evaluations will be done by clerical staff in the Department, as is current practice, with the help of the graduate evaluators.

The impact of the project on gender equity, particularly for women students in the laboratories, is considered a distinct component of the impact on all students taking the laboratories. We plan to design a short questionnaire focussing on the effect of having a woman instructor and distribute this questionnaire to women students only. While the number of women students per lab section is small, we will gather these questionnaires in all sections and over the two year period of the project, thereby increasing the statistical significance of the responses. Female members of the graduate evaluation team will also facilitate two or more focus groups comprised of women undergraduates taking the labs, in order to investigate this question.

Assessment of the impact of quality of instruction will therefore comprise the most extensive and costly aspect of the evaluation plan, due to the number of Trainees and the number of students in the lab sections. We believe that this is justified for several reasons. Undergraduate student satisfaction (and improvement of satisfaction levels) with instruction in the labs is critical to determining whether this program should be institutionalized, in the Department of Physics and Astronomy or elsewhere. Furthermore, changes which improve science instruction stand to benefit women and under-represented minorities disproportionately, according to many proponents of science education reform (Tobias, 1990) and gender equity in science.

The external evaluator will conduct group interviews with the graduate co-instructors of the Trainees and with the course professors. These interviews will make up an additional part of the evaluation of the Trainee's teaching skill and improvement, and they will also contribute to the implementation evaluation (the other major portion of which will come from the evaluator's interviews with Trainees, as mentioned above). The external evaluator will also have access to the summer orientation program and lab sections in order to make observations of the Trainees as the project proceeds.

The evaluation plan will make use of University-based computerized databases in tracking objective data relating to the two principal cohorts of interest to this project, the participant Trainees and the students in introductory physics labs. Such data include information on course enrollments, grade distributions, and changes in academic major. While it is doubtful that such information will be relevant to the immediate objectives of the project, in combination with the other evaluative information we will gather, it will serve as baseline research with which to monitor trends over the long term. For instance, this project has bearing on the question of whether undergraduate teaching experience positively impacts women's retention in science, in the way that undergraduate research experience has been shown to do (Rayman, 1993). For similar reasons, we will aim to contact the Trainees every twelve months to keep track of their post-bachelor's career developments.

Question 1: Did the project positively impact Trainee's confidence, teaching competence, and sense of belonging in the Departmental community?

Question 2: Did the project improve the quality of instruction in the laboratories?

Return to Table of Contents