Under-Represented Populations Annotated Report Excerpts

Excerpt 1 [Education Development Center]

Methodological Approach:
Describes multiple methods of inquiry

Research data on project implementation and impact was collected in several ways. To characterize Telementoring student participants and compare them with the general population of students enrolled in the AiS program, we designed a pre-test attitudinal questionnaire that was administered over the World Wide Web. This questionnaire probed a multitude of issues related to students' attitudes and perceptions about science, math, and computer science prior to implementation of the Telementoring program. Nearly 1,000 students, including 845 high school students and 115 junior high school students, completed the questionnaire. We also collected descriptive data on participants through program applications.

To gather data about how the Projects were implemented in a range of different classrooms and regional settings, we conducted preliminary site visits to each of the ten participating schools. Final site visits are now in progress.

Following an observation/interview protocol developed for these visits, researchers investigated topics such as expectations and understandings of the Telecomentoring Project, logistics of implementation, and perceptions of the student-mentor and student-student relationships.

To investigate the quality and perceptions of participants' mentoring experiences, we collected and analyzed two types of data—self-report data contained in the form of Mid-Project evaluations and on-line conversational data. Mid-Project evaluations for both students and mentors were distributed via e-mail to participants who had been active in the project for at least six weeks. Data from these self-reports were later examined in light of on-line conversational data which enabled us to document the frequency of communication, the topics discussed between students and their mentors, as well as communication among student participants.

Research data from attitudinal questionnaires, preliminary and final site visits to data, Mid-Project evaluations, and on-line conversational data is summarized in this section. Findings are reported in terms of how they relate to the various players in this year's program implementation—students, mentors, teacher liaisons and parents. In addition, a discussion of general project findings follows.

Instruments:
Describes survey questionnaire

A. Students

1.  Profile of Student Attitudes

Over 100 young women (n=115) in grades 9-12 applied to participate in the Telementoring project this year, and 109 were matched with mentors. Sixty six (66) of these young women completed the pretest attitudinal questionnaire prior to their participation in the Telementoring project. This questionnaire probed many issues related to student attitudes and perceptions of science and computer science. For this report, five sets of issues were analyzed: (1) confidence with technological applications and methodologies; (2) post-secondary educational aspirations; (3) interest in and perceptions of various college and career fields; (4) extracurricular science and technical activities; and (5) perceptions of belonging (having a voice and being listened to in science and technical courses). Attitudinal data discussed in this report represents high school respondents only.

Confidence with technological applications and methodologies.
Students were asked to rate how confident they felt with applications and techniques such as manipulating data with spreadsheets, graphing and visualizing data, communicating over the Internet, conducting research over the Internet, solving math problems related to their AiS projects, and working in teams.

Information Sources & Sampling:
Describes selection of small group of subjects

A total of 7 female junior-year mechanical engineering students from Brooklyn Technical High School and their respective parents were recruited to participate in formative research for the entire academic year.¹ Five of the girls were African-American and two were white. Brooklyn Tech's mechanical engineering program was specifically selected because the program's coordinator was interested in sustaining the interest of young women in engineering. In addition, it is an exemplary project-based program and resembles the types of classes we anticipate working with next year in our pilot implementation phase with the Department of Energy's Adventures in Supercomputing (AiS) program.

¹ The number of student participants was smaller than projected due to the low enrollment of females in the mechanical engineering program and the transfer of some female students out of the school at the time of our recruitment. While the project attempted to solicit additional female students in other traditional engineering programs at the school, these students failed to respond to repeated recruitment efforts.

Excerpt 2 [Education Equity Concepts]

Methodological Approach:
Describes multiple methods of inquiry for formative and summative evaluation

The formative evaluation centered on the collection of data to inform the ongoing project, including group leader evaluations of training, interviews with group leaders about activities and the training and group leader evaluations for each PS+ activity they did with students.

The summative evaluation focused on the collection of data about the impact of the project on the quality and quantity of science activities done by group leaders as well as the impact of the activities on students' attitudes towards science. Pre interviews were done with a sample of 28 participants from eight sites including six administrators and 23 leaders. Post interviews were done with 25 participants, including three administrators, one parent and 21 group leaders from these sites. However, because of staff turnover issues, only 14 pre/posts were received from the same participants, three of whom were administrators. The interviews focused on the science, career and literacy activities group leaders did with students as well as their own attitudes towards science. In addition group leaders filled out evaluation forms for the activities they did. Twenty-eight forms were received.

Pre, post, and one year follow-up interviews were conducted with 26 students from the two centers who participated in PS+ for two years. These interviews focused on the science students did, their definitions of science, science careers and their attitudes towards science. Students included sixteen girls and eight boys.

Excerpt 3 [Montana State University-Bozeman]

Methodological Approach:
Relates design to evaluation purposes

Evaluation objectives were designed to: (1) provide feedback to the institute organizers, (2) assess short term effects of the Institute, and (3) collect baseline data to gauge long-term effects of the institute on SEA initiatives for the following year. Several data collection strategies and instruments were developed in order to meet these objectives.

Methodological Approach:
Describes multiple instruments and data collection procedures

1. Pre-Institute Survey. This questionnaire assessed participants' opinions prior to the course as an awareness activity.
2. Mid-Institute Survey. This instrument collected session-by-session feedback for specific workshop components or features. Participants were asked to rate aspects of the institute from 5 (outstanding) to 1 (unsatisfactory), plus write comments on highlights and suggestions for improvement for each session.
3. July 17 Final Assessment. This survey included demographic information, Likert rating of various aspects of the institute, and three questions regarding (a) what participants had learned about feminist pedagogy, (b) what they planned to try out in their teaching, and (c) what ideas or plans they had for a mini-grant.
4. Observations. The evaluator was present for many institute sessions in order to gauge the scope of content, approaches, level of engagement of the participants, and other aspects of the experience.
5. Document File. Agenda, participant lists and plans for sessions were gathered for content analysis. Photographs were taken during various activities.

Excerpt 4 [Rochester Institute of Technology]

Data Collection Procedures & Schedule:
Describes classroom observation procedure

Project Component

Classroom Observations

Goal
The goal of the Classroom Observations is to quantify over time the gender bias exhibited by teachers in their regular classrooms.

Process
Several teachers who have received training and been involved with the project since its inception were the Coders. In order to code the classes, the participating teachers provided the Coders with a classroom seating chart which specified the name and gender of each student along with an overview of the physical environment of the room.

Evaluation
During the lectures the Coders observed teacher initiated or student initiated questions and interactions and the type and level of response. In addition to the Coders each observation was video taped for review by the teacher and Coder as a method of providing feedback.

Excerpt 5 [Tufts University]

Methodological Approach:
Describe multiple methods of inquiry and sample

II. Evaluation

Evaluation was an integral component of the project from its inception, incorporating both formative and summative components. Data used to evaluate the project was collected from a variety of sources, including:

• Attendance at the initial planning meeting;
• Three observations during the Summer Institute;
• Parent and teacher interviews during the Summer Institute;
• Pre/post student, parent, and teacher questionnaires;
• Pre/post interviews with Tufts University and Science Discovery Museum staff;
• Post project questionnaires and interviews with Tufts engineering student "mentors";
• Review of materials provided to participating girls;
• Attendance at the final project presentations at Tufts University and Opening Night at the Science Discovery Museum;
• Follow-up interviews with teachers focusing on what, if any aspects of the project they have implemented in their classroom following the project.

Data Collection Procedures & Schedule

Instruments:
Describes focus of questions

To look at areas of possible impact on girls, pre/post data was collected on girls' attitudes toward math, science and engineering; their interest in math, science and engineering careers; math and science course-taking plans and as well as educational plans; and involvement in informal science and engineering activities. Pre/post data was received from all twenty-seven of the girls.¹ A variety of formative data was collected from participating girls as well, focusing on why they decided to participate in the project, what they liked most and least about the project and suggestions on how they would improve it.²

Questions for teachers and parents focused on their attitudes and awareness of equity issues and specific things they do, or could do to encourage girls in math, science and engineering, what, if any hands-on science activities they do, as well as strengths and weakness of the project and suggestions for improvement. They were also asked to complete open-ended sentences similar to those given to the girls. Pre/post data was received from all five teachers while pre/post data was available from seven of the sixteen parents who were involved with the project. Data was also collected from two university staff and five museum staff which focused on lessons learned and areas of possible impact.

¹ Participants included five sixth graders, twenty seventh graders and two eighth graders. The participants were white (23/85%) and Asian (4/15%).

² Please see the Appendix for summaries of feedback from participating girls, teachers, parents, college students and Museum staff.

Excerpt 6 [University of Denver]

Information Sources & Sampling:
Describes sample selection and attrition

The target population was sixth through eighth grade girls in middle schools in the Denver Public School System. Approximately 130 girls were nominated by teachers, parents, other school personnel, and girls themselves to participate in the program. All nominated girls and their parent(s) were sent a letter of description and a consent form. 62 families agreed to participate, returned the consent forms, and were included in the initial sample. The 62 girls were then assigned to participate in either the year-long Saturday SEM program (Group 1) or in the Control Group (Group 2) using a modified random sampling procedure. The ratio of 60/40 was chosen so that the 36 slots available for the Saturday program would be filled and at the same time there would be enough girls in the Control Group to have meaningful information. Therefore, 38 girls were assigned to the SEM program and 24 girls to the control group.

Attrition

Group a: Of the 38 girls originally assigned to Group 1, three girls dropped out of the program prior to attending any sessions. The other 35 girls completed the program, yet data will be calculated using all girls that attended any sessions (n=35). See Table 1 for the attendance at each one of the eight SEM sessions, and for the three parent sessions.

…

Data Collection Procedures & Schedule

Mathematics and science teachers from 4 middle schools in the Denver School District were sent letters explaining the SEM program and requesting interviews from 16 teachers. Teachers were informed that the interviews would focus on girls' interest in math and science. The first 16 teachers who responded were chosen for the interviews. Teachers were then contacted by one of two graduate assistants and interviews were scheduled. All interviews were conducted at the teacher's home school. Each of the research assistants interviewed 8 teachers. Of the 16 teachers interviewed, 9 were women. Before each interview, each teacher signed a consent form and was paid $50 for their time and participation.

(…)

Methodological Approach:
Describes how effectiveness will be judged

Meta-Evaluation

In the initial year of the program, we were curious about the effect of the overall program but were equally concerned with the impact of the individual sessions. We learned from the first year that the types of questions asked must be specific and at the cognitive developmental level of the middle school students. Girls had responded with one or two words when asked general questions, providing only minimal information. Therefore, in the extension year, the assessment of individual sessions was conducted with more specific questions in the hope of gaining more relevant information.

(…)

Presents a hypothesis that drives the evaluation

Another interesting consideration coming from the initial year of the project was the importance of undergraduate student mentors. Undergraduate students in biology, chemistry, and engineering were employed to assist in helping girls and parents for all parts of the Saturday programs. Informally it seemed that the mentors contributed a great deal to the learning atmosphere for girls and parents, as well as learned about teaching, helping, and their subject matter. From our own experience we knew that teaching the material is one of the best ways of learning about it in more depth. Many undergraduates, early in their education, transfer out of science, engineering, and mathematics because they struggle with the material. It was our belief that providing the opportunity for undergraduates to become mentors to middle school students might also increase their own enthusiasm for SEM. During the extension year of the program, this component was assessed more directly by interviews with 6 undergraduates.

Excerpt 7 [University of Washington]

Information Sources & Sampling

Thirty-one institutions were identified as having formal Women in Engineering Programs. These 31 institutions were identified from the Women in Engineering Directory of College/University Programs (Wadsworth, 1991), which lists over 180 institutions with some type of activity for women in engineering. To delineate informal from formal programs, the criteria that a specified individual was employed as a program administrator for women in engineering was applied to the list of institutions. Hence, only 31 institutions were identified as having formal programs.

(…)

Instruments:
Describes survey objectives

A survey was designed to gather information regarding the following generic topics: 1) program objectives; 2) target populations; 3) years in operation; 4) organizational structure; 5) budget; and 6) degree earned. Within each of the topical areas, the questions were designed to ascertain the following: 1) commitment of the engineering dean; 2) skills and experience of the designated director of the program; 3) adequacy of the budget; 4) assistance with fundraising; 5) student participation or involvement in the program; and 6) system of accountability or evaluation procedures.

Data Collection Procedures & Schedule:
Describes piloting

The questionnaire was pilot tested with six program directors at six different institutions. The questionnaire took approximately 30 minutes to complete. Examples of modifications were minimal; the revisions were made.

Excerpt 8 [University of North Texas]

Methodological Approach:
Describes extent of program intervention

The BUGS Program and Participants

The BUGS Program provided the opportunity for 32 fourth and fifth grade girls to work in an outdoor science lab after school. The students met 30 times after school. 20 meetings were scheduled at an elementary where an outdoor science lab was available. Additional meetings were scheduled at UNT where the girls worked with adult and university student mentors on collaborative experiments. Four informational parent meetings were held at UNT and an additional evening meeting was scheduled where mentors and mentees shared their experiments with parents. A two-week summer day camp provided a capstone experience for the girls. The attendance for the summer experience was lower than the attendance for the after school and evening parent opportunities. The parents and girls often visited the interactive Web site that was created by the project. Books were added for parents and students at the public library. Interviews indicated that books were appreciated but these were accessed less then the information on the web. A web scavenger hunt informed parents and students about the Web site and seemed motivational for using the site. Parent interviews indicated that the Web site was a convenient way to obtain information and was complimented as user friendly.

Information Sources & Sampling
Describes random assignment to treatment and control groups

At the beginning of the 2001 school year, solicitation for participation was facilitated by teachers, counselors, and administrative personnel within the school district. Interested girls (with parental permission) submitted an online application. These applications were reviewed and then were randomly divided into the BUGS and contrast groups.

Instruments:
Describes combination of standardized quantitative outcome measures and qualitative interviews

Instrumentation and Data Collection

Several quantitative assessments were administered in pre and posttest fashion. These instruments and their subscales included:

In addition to the quantitative indices, qualitative data were also collected via observation and interviews. A comprehensive round of semi-structured interviews were conducted with participating BUGS students (n = 12), parents of BUGS participants (n = 17), the BUGS program teachers (n = 2), and members of the contrast group (n = 14). Contrast group parents were interviewed. (n = 9). Other building teachers were interviewed. (n = 7). The quantitative and qualitative results are presented below and are organized by project goals.