Under-Represented Populations Annotated Report Excerpts

Excerpt 1 [Committee on Institutional Cooperation]

Interpretations & Conclusions:
Present qualitative results

Below are the most frequent responses to the open-ended questions. In some cases responses were selected on the basis of their information. The question from the original survey is labeled A, and the corresponding question from the follow-up survey is labeled B. Responses to the questions from both time points are placed together for comparisons. However, for the more general questions responses are combined.

1. Please provide an example of how you plan to make use of the information/activities presented at this workshop?
   
   
   • Order and use "Classroom Climate Workshop" and conflict resolution videos.
   • Use interactive theater in workshops and training.
   • Plan and organize similar workshops on-campus.
   • Incorporate gender equitable teaching strategies.
   
   
2. Please provide an example of how you've made use of the information/activities presented at the Best Practices Workshop?
   
   
   • Ordered and/or used videos on conflict resolution.
   • Using or planning to use interactive theater in workshops and training.
   • Developing workshops and courses.

1. What was the most important idea that you learned at this workshop?
   
   
   • "Gender conscious pedagogy is better pedagogy for all."
   • The use of interactive theater for TA training.
   • Interest-based negotiation strategies.
   
   
2. What was the most important idea that you learned from the Best Practices Workshop?
   
   
   • Interactive theater.

Excerpt 2 [Anonymous 1]

Recommendations

The following recommendations are based on the cumulative analysis of all survey data:

• Continue to focus on the actual improvement of girls' achievement scores in science and mathematics. The girls' self-esteem will reach higher levels if actual increases in achievement accompany the already existing programs.
  
  
  
• Create tangible goals and assessment methods in science that girls can use to measure their own achievement. Most girls believe they are more successful in math because it is easy and they can monitor their own progress although teacher data and some quantitative data from the girls does not support these conclusions. In reality girls are more interested in different aspects of science and succeed in these areas but are unaware of their successes because they are unsure how to evaluate them.

Recommends consideration of local context

• Recognize that most of the girls are influenced by the words and actions of their parents. Maintain contact with the parents and involve them in as many aspects of the different programs as possible. Parents need to be kept abreast of the program focuses, expectations, and activities so that they can reinforce these concepts and ideas at home.

Cites findings that support recommendation

• Build a teacher-parent partnership focusing on improving achievement and increasing confidence in mathematics, science, and technology. Survey data suggest that parents have a better understanding of their daughters' feelings about their experiences in these programs, especially on how the programs impact their daughters' interest level, enthusiasm, and feelings about their potential success in math, science, and technology. Teachers have a better grasp of the student's actual abilities and behaviors in these areas. A partnership of both teachers and parents will help to create programs, opportunities, and activities which best meet the girls' social, emotional, and academic needs.

Excerpt 3 [Education Development Center]

Interpretations & Conclusions:
Presents lessons learned

Implementation of Modeling Instruction

During this year's pilot implementation, a number of critical issues were raised that have implications for the plans and direction of the project. These issues are described below. They reflect findings from our formative research, and feedback from project participants. These issues will inform revisions to Telementoring environments and materials.

Presents conclusions about local context

• Diversity—Race, gender, and class issues don't disappear on line. Training and preparation need to include some type of sensitivity awareness and training. This is especially true for student facilitators who are responsible for moderating discussions for students from different regions. In addition, there are diversity issues that do arise in the mentor-student matching process where students have indicated a desire to be matched to mentors of a similar cultural background. This runs contrary to some generally held beliefs that on-line environments are devoid of these biases.
  
  
• Maintenance—Maintaining the processes, structures, and project components is time intensive and requires investing in project participants. This year, past participants became future trainers and facilitators. As we continue to scale this program up we will continue to grow experts from within the program, and will also continue to support these experts in their new leadership roles within the Telementoring on-line environments.

Excerpt 4 [Educational Equity Concepts]

Interpretations & Conclusions:
Describes data limitations

It is important to note that student data were collected only over two sites. Pre data were collected prior to any involvement with PS+ while post data were collected after students and group leaders had done PS+ with no leader training. The follow-up data were collected after the group leaders had had PS+ training and were doing PS+ with these students for a second year. The PS+ group leader training was held at these two sites and almost all group leaders from these two sites attended. Thus pre/post comparisons look at the impact of PS+ on students when their group leaders have not had training. Post/follow-up comparisons look at the effect of PS+ on students when group leaders have had training.

Presents and interprets results

After doing PS+, students were more apt to mention specific PS+ activities as the science they did in the after school centers, although science remains a minor part of students' after school experience.

The total number of activities students mentioned when they were asked what they did in their after school program decreased over the year and a half from 4.3 (pre) to 3.7 (post) to 3.5 (follow-up). The number of math activities mentioned stayed at .2, with four students mentioning a math-related activity. The mean number of science activities mentioned by students started at .04, increased to .5 by the post and decreased to .1 by the follow-up. In general, activities students were most apt to mention were sports/gym and computer games. Science activities mentioned included Building with Wonderful Junk, SMART Activities and general science activities.

Excerpt 5 [Georgia InGEAR Consortium]

• All five institutions completed extensive self-study data analysis and began disseminating self-study findings within their institutions, at professional development activities, and at professional conferences
• Board of Regents institutional-level data provided by Georgia Tech to state schools
• High awareness of the potential usefulness of and motivation to use self-study findings to promote educational reform

Interpretations & Conclusions:
Draws conclusion about utility of data

By the end of the second year, the participating institutions had gathered and analyzed an extensive amount of data for self-study. However, participants were often at a loss as to how to use these data effectively as part of the reform teacher preparation programs on their campuses. This is especially challenging for younger faculty members who are less familiar with the administrative and programmatic politics at their institution.

Recommendations:
Presents recommendations for project improvement

Recommendations: During the third year, self-study activities should focus on how self-study data can be used to change the environment of the university. It is recommended that local Leadership Teams each identify a task force to review and recommend use of the data on the local campus

Implementation plans should be informed by the following questions

• Who should serve on the committee or task force that reviews the self-study data and determines policy recommendations?
• What is the best strategy for getting local administrators to welcome the results of the study?
• Through which avenues should the results be disseminated?

Recommendations:
Presents recommendations for evaluation improvement and continuing stakeholder involvement

• Technical assistance provided for developing outcome measures
• Evaluation strategy recommendations received from the Advisory Council
• Comprehensive evaluation process included annual report guidelines and site visits

Recommendations: During the third year of the project, the Project Evaluator will continue to provide technical assistance for developing and implementing outcome measures based on suggestions presented by the advisory council during the first annual meeting. Ideally, at least one outcome measure will be used for each strand during the third year of the project. The possibility of including a question about gender equity on student ratings of instruction instruments should be investigated.

Interpretations & Conclusions:
Presents conclusions

It was recognized during Year 1 that additional meetings were needed to address logistical and organizational issues related to the project as a whole. Consequently, monthly Co-PI meetings were instituted, and the first was held in January. Co-PIs reported that they found meetings with a strong substantive focus (e.g. curriculum revision, professional development, current research findings) particularly rewarding.

In addition, many participants commented positively about the smooth functioning and lack of conflict within the Management Team and CO-PI meetings. With this strong "climate of collaboration," it is possible that participants may be reticent to address conceptual and administrative issues that are of concern. This is especially likely in light of the fact that InGEAR is the first collaborative for several of the Management Team members.

Recommendations:
Recommends greater influence of local teams

Recommendations: The InGEAR collaborative consists of multiple partners with diverse needs. In order to best address local needs, it is important that local teams help shape the agendas of the Management Team and CO-PI meetings so that relevant logistical and substantive issues can be addressed. As appropriate, issues that concern the project as a whole—but may have only been discussed privately and/or informally, should be formally incorporated into Management Team and/or CO-PI meeting agendas.

Excerpt 6 [Girls Inc. of Alameda County]

Interpretations & Conclusions:
Interprets quantitative survey results

Graduates were asked to identify the role that participation in Eureka played in developing their confidence in math, science, computers and sports. Responses indicated that Eureka played a primary role in developing confidence in computers (4.25), science and sports (4.12), and a moderate role in math (3.5). Graduates recognized their increased confidence in these subjects and the significant role that Eureka played in developing their confidence.

Presents qualitative evidence that supports a conclusion

Two comments made during the focus group meeting indicated their increased level of confidence about science and math. When asked if they felt more confident about being in math and science classes they related experiences they had on field trips.

"Like when we go on field trips, experiencing things such as science at Great America or the Exploratorium has science and math principles, and if you have an interest in that you can learn a lot. For instance, the Revolution (at Great America), if you want to figure out how many times it swings before it goes over (makes a complete revolution) you have to know science. It has a lot to do with weight, and you get to calculate how much weight is needed to know how many times it swings before it goes over."

"Through Eureka I have come to understand my abilities as far as what I can do when I'm in the classroom…I know that because I want to run my own business, I have to know how much money that I can put into my business, how much money I'm taking out of my business and how much money I'm going to spend on my inventory. So I mean, I think math and science might not be something that you like to do, but look at it as far as the future and your business. If it's right there in front of you, can you do it, and that's what's important."

Excerpt 7 [Montana State University—Bozeman]

Interpretations & Conclusions:
Presents qualitative survey results

On surveys, participants mentioned a few areas of difficulty: (a) pre-conference text readings were not available until Wednesday morning; (b) conference details were not provided early enough; (c) undergraduate student panelists were "too perfect" and not representative; and (d) confusion over the Friday session—was it optional or not?

Presents conclusion and supporting evidence

Overall sessions were given high ratings. Participants valued the small group discussion and brainstorming opportunities, especially when groups were mixed by gender, and with teacher/counselor Short Course participants. Faculty worked best in groups of 6 to 8 when they had assigned tasks and were required to report back to the whole group. They were observed to draw on personal experiences and compare perspectives.

Presents quantitative survey results that support a conclusion

On the exit survey, 90% of respondents felt the institute helped to establish a cooperative and mutually supportive network of faculty who share a commitment to increase the number of women in science and engineering. 55% of respondents said they learned something about female friendly pedagogy that they didn't know when the institute began. However, only 25% of respondents wrote clear ideas for what they would try in their teaching. 40% of respondents listed definite ideas or plans for mini-grants.

In summary the session-by-session and exit surveys indicated the faculty institute as presented provided a springboard for discussion and created a network of interested people.

Recommendations

The following recommendations are made for the next institute:

1. Use e-mail communication to make pre-institute announcements and begin participant dialogue. Several of the 1997 participants indicated they were not connected to the e-mail list-serve.
2. Distribute preconference readings in advance and use readings to focus discussion sessions.
3. Continue to coordinate short course and faculty institute. Consider starting the initial joint session with an ice-breaker activity to (1) introduce the two groups to one another and (2) focus on common areas of concern. Perhaps both groups could do a common preconference reading and begin with a mixed group small discussion session.
4. When the faculty institute and teacher/counselor short course groups meet jointly, continue to mix breakout groups by gender, title and type of institution. If possible, balance working groups with (1) males and females; (2) teachers, counselors, faculty and administrators; and (3) K-12 school, tribal college and MSU Bozeman people.
5. Continue to model feminist pedagogy at institute sessions: Acknowledge women's experience, empower participants, reject the "open and pour" method of teaching, emphasize critical analysis, connect experiences and knowledge with principles and practices. Build from the participants' knowledge base.
6. Replace the current undergraduate student panel with (1) a panel of recent graduates from the science or engineering areas prepared to talk about their undergraduate and early career experiences; (2) a panel of graduate students prepared to talk about their undergraduate experience (i.e. what helped them to stay in a science or engineering program); or (3) a panel of tribal college students and/or faculty prepared to talk about their perspectives.
7. Have panelists use microphones and arrange seating for better visibility. The Wednesday student panelists were seated in front of a sunny window, so their faces were in shadow.
8. Serve "on-site" lunches each day to facilitate a continuance of morning discussions. Offer lighter snacks and refreshments.
9. In the 1998 institute, showcase successful first year mini-grant initiatives. Start brainstorming sessions for mini-grant ideas earlier in the institute.
10. Prepare a press release for the MSU Bozeman September Staff Bulletin and include examples of teaching strategies featured at this institute. This might become a starting place for continued communication and reinforce institute outcomes.
11. Use local demographics. Incorporate sessions with specific data on MSU science and engineering enrollment and graduation rates. Faculty may not be aware of the numbers broken out by gender and ethnicity.
12. Increase the participation of university administrators: department heads, deans and others. Their active participation demonstrates support for gender equity.
13. The majority of participants were male, so female organizers need to be open about the complexity of gender equity issues. It is important to keep an open-mind, hear all the voices and articulate clearly that "men are not the problem and women are not the solution."
14. Continue to feature national leaders lik e<name of person>, who was a master leader for the Talking Circle Session, and <name of person>, who demonstrated use of e-mail, Student Board of Directors, and other valuable ideas being used in a student-centered chemistry course at the University of Wisconsin—Madison.
15. Some MSU Bozeman faculty participants were observed to attend some but not all sessions. In order to improve sustained participation in 1998, consider holding the institute at a "retreat" setting—an easy drive from campus (away from the lure of offices and labs). The Madison River Inn or Deer Park Chalet are possible locations.

Interpretations & Conclusions

Although suggestions are given, this first MSU and tribal college faculty institute was well received, included appropriate information, and began communication among MSU departments, tribal colleges and Montana school teachers and counselors. SEA made substantial gains in creation of a network of faculty who share a commitment to increase the number of women in science and engineering.

Excerpt 8 [College of St. Scholastica]

Interpretations & Conclusions:
Prefaces conclusions with background information about project

SciCon was created because the staff of the PLUS Center recognized that most of the girls who attended FAST Camp as 6th or 7th graders were not attending NSD when they became eligible as 8th or 9th graders. Even though the girls enjoyed their FAST Camp experience and left camp motivated and excited about science, their enthusiasm did not persist. The return to school and/or home environment that did not provide ongoing support for math and science achievement seemed to be preventing FAST Camp students from maintaining the high level of interest, enthusiasm, and self-confidence that was generated by the summer camp. The SciCon program was designed to overcome this problem by providing special science activities through the school year and by involving parents and teachers in addition to the girls themselves.

Presents conclusions

Characteristics of Successful Activities

The SciCon Saturday Science Workshops were designed to provide the girls with a different type of experience and to introduce a different aspect of science each month. This variety clearly helped to sustain the girls' interest in the program, and it also helped to show them that science is available to them in many different forms. Although every workshop was different, the thematic elements that were most crucial to the success of the program included:

• Hands-on activities,
• Fun,
• Social interaction with friends,
• Teamwork,
• Good science/data collection in a real-world setting,
• Freedom to explore (inquiry), and
• Strong female role models.

These elements formed a common thread that was carried through all program activities.

The Role of Parents

During the two years of the SciCon program it was obvious that parental support was critical for the success of the program. Some of the SciCon girls came from great distances to attend the monthly Saturday Science Workshops at St. Scholastica. In both Year 1 and Year 2, over half of the participants had to travel more than 25 miles to get to Duluth. In Year 1, two girls traveled more than 100 miles, and in Year 2, five girls traveled 100 miles or more. When it was necessary to leave home at 4 a.m. on a dark winter morning, with the temperature below zero and the roads covered with fresh snow, and drive more than 100 miles to get to the college, it was essential that parents were as committed as the girls.

In follow-up sessions and program evaluations, parents also indicated that SciCon helped provide them with at-home activities for stimulating and sustaining their daughter's interest in science.

• "I appreciate the ideas to support and keep our children involved in science."
• "I am not educated in science, but I do try the best I can and this program has taught me a lot on how to keep encouraging my daughter."

Involving Teachers

At the start of the program, the girls were encouraged to identify a special teacher who supported them in science and would act as a mentor. The PLUS Center intended to hold science workshops for these teachers to introduce them to additional ways to encourage the SciCon girls. This idea did not work well, however, because not all students were able to identify a current, supportive, science teacher. As a result, instead of holding special teacher workshops as originally planned, interested teachers were invited to attend any of the SciCon weekend programs. Only a few teachers took advantage of the invitation.

Although participation by teachers was not as significant as originally envisioned, when interviewed by phone in April and May of 1998 by <name of person> of CSS, teachers indicated that the SciCon girls really stood out in their classes. The theme that emerged clearly from the teachers' comments was that prior to participating in SciCon the girls were shy and reluctant to ask questions in class, but after the program, they were much more outgoing and willing to ask questions.

Impact on Students

Based both on quantitative and qualitative evidence, the SciCon program appears to have succeeded in its goal of sustaining participants' interest in science during the school year and bridging the gap between FAST Camp and NSD. Half of the Year 1 SciCon girls have gone on to attend NSD. Seven girls participated in both years of SciCon, and five of those seven attended NSD for a second time in 1998. Prior to SciCon, only one girl had ever attended NSD twice.

The long-term impacts of the program are more difficult to assess. A follow-up survey was sent out to the girls in spring 1998 to collect information about their school activities and accomplishments. The PLUS Center plans to send out a survey every year in order to keep track of the girls' long-term achievements and successes.

Based on the girls' and parents' written comments, it appears that, at a minimum, SciCon gave them the opportunity to meet friends with similar interests, increased their self-confidence, gave them the incentive to work harder at school, and introduced them to potential science careers and strong female role models.

Excerpt 9 [Tufts University]

Interpretations & Conclusions:
Presents a perspective for interpreting results

It is important to place the following results in the context of the selection criteria for girls, which included high interest in math and/or science, good disciplinary standing, demonstrated ability to work as part of a cooperative learning team, and good communication skills.¹ In addition, recruitment efforts for individual girls focused heavily on "popular" students who fulfilled the criteria, as project staff felt these students could potentially be role-models to their classmates and lower-level school mates by showing interest in science. Project staff felt that students who were not popular would influence fewer students because of "weak interpersonal skills" or because of "not being liked" generally. As defined by the project staff for selection criteria, popular students are amicable and well-liked by the general student body and teachers and possess strong interpersonal skills.²

¹ It must also be noted that 23 out of 27 (85%) participating girls had at least one parent who works professionally in a math, science or technology-related field, and nine of the girls had two parents who did. Ten parents were engineers, eight were involved in a high tech industry (including four parents who were presidents/CEO's) and five were scientists. Parents also included a surgeon, math/science teacher and pharmacist.

² Teacher and principal perceptions were used for evaluating the "popularity" of the students.

(…)

Presents the perspectives of different participant groups about program impact

Areas of Perceived Impacts on Girls

Adults participating in the project felt that the girls gained in abilities in problem-solving and working in groups, areas that were not measured in the evaluation. They also felt girls had "increased confidence," which was consistent with girls' increases in the sentence completion measures. In addition, some adults feel that the impact of the project may not be felt for several years, when girls begin to make college and career choices.

When asked what impact, if any, they saw the program having on the participants, the eight college students interviewed were most apt to say that the program broadened the girls' minds by exposing them to the field of engineering (5/63%). They felt that in the future girls will now explore careers in engineering which they would not have considered before the program. They also felt girls learned to work in a group and cooperate (5/63% each). Two of the college students noticed a boost in the girls' confidence levels. Other areas of impact mentioned were how to do a big project, the fact that things do not always go perfectly, and conjointly, how to problem-solve (25%/2 each).

Parents were remarkably consistent in what they felt was the greatest impact on girls. Overall, parents felt girls gained skills in teamwork (7/70%), learned how to use various tools and technology (7/70%) and gained knowledge of engineering (7/70%). In the words of one parent, (who happened to be an engineer): "My daughter and the other girls learned what engineering really is—10% inspiration and 90% perspiration, to misquote Edison. I hope it didn't scare them!" Parents also felt girls learned how to execute a major project (6), gained confidence (4), learned how to solve problems creatively (4) as well as gained knowledge of scientific concepts (2).

The five teachers felt the major impact on girls was more ability to work in groups (3/60%), confidence (2), a sense of accomplishment (2) and learning the importance of setting goals and adhering to them (2). Through this, teachers felt "Girls learned to do even the most complex tasks that at first may seem scary and/or impossible." In addition, teachers also felt that girls had understood complex math and science concepts behind their exhibits (3).

Impact on Teachers

Project staff hoped that by involving teachers as members of the core team, the impact of the project would extend beyond the summer. Initially, project staff hoped that the project would enhance teachers' aptitude with science and engineering and increase their use of hands-on science, and that teachers would continue project-related activities, such as visiting the exhibits at the Acton Discovery Museums as well as other museums with their classes. It was expected that teachers would have continued contact with girls, and that the project would improve teachers' awareness of gender equity both in curricula and classroom climate. Teachers came into the project very interested in hands-on science, spending between 25% and 80% of their science time doing hands-on science activities (mean: 59%) and doing between four and sixteen hands-on science activities (mean: 8.6). Some of the hands-on activities teachers mentioned were: vacuum pumps, building simple machines, constructing a solar house, heat capacity and exchange, models of plate tectonics and photography to investigate chemistry and optics.

Presents conclusions and recommendations for a modified, generalizable model

II. The project has the potential to be replicated, although the cost of high caliber human and material resources may make replication of the current model difficult.

The project was implemented with a group of 27 middle school girls with pre-existing interest and achievement in science and mathematics. Girls had the benefit of a high adult to student ratio, access to state-of-the-art technical facilities, and a cadre of university faculty and museum staff to provide technical and conceptual advice. While replication with similar resources and a similar target audience is possible, the project may want to explore ways to expand replication audiences. One way to do this is to implement the model with more diverse target populations, including girls who are less academically gifted, girls from lower socioeconomic levels or girls who are African American or Hispanic. This may involve modifications in the project based on the target population. For example, more information on careers may be needed for girls whose parents aren't scientists and engineers. In addition, more time might be needed on technical skills development for girls who have had fewer experiences in these areas.

The project may also want to explore lower cost variations of the project, such as depending more heavily on graduate students than professors for technical assistance, increasing the number of girls per group or reducing the time teachers come in during the summer.

(…)

Presents results combining qualitative and quantitative data

As part of their final questionnaire, the twenty-seven participating girls were asked what parts of the project they liked most, least and how they would improve it. A summary of their responses follows.

What Girls Liked

When asked what they liked best about the project, 25 of 27 girls (93%) mentioned at least one activity related to designing and building the exhibit, such as using tools and working as a team. Overwhelmingly the one thing girls liked best about the project was using tools (12/44%). In the words of one girl: "I liked working with power tools. I don't know many girls who know how to use them and now I do!" In addition to using tools, girls liked being able to work as a team (9/33%), building the exhibit (8/30%), using computers (8/30%) as well as the overall exhibit design and development process (7/26%): "I liked the planning part, choosing a design and exploring options." Girls also liked "being treated as adults" (7/26%), working with the engineering students (3/11%) and that the "project was fun" (3/11%). Individuals also liked the Tufts staff, meeting new people, learning the concepts of the exhibit, being able to use the Tufts facilities, learning about engineering and "figuring out stuff."

Areas to Improve

There were several things that girls did not like about the project and would change. Most frequently, girls felt that days were too long, and suggested providing shorter days but more weeks (10/37%). Five girls felt some "take over" from adults:

Although it was a girls in engineering project, I felt as though some of the people in the workshop—men—when you would ask a simple question they would take over and do the whole thing for you rather than just tell you stuff.

Other areas girls "liked least" were lectures on concepts and working with girls from other groups (mentioned by four girls each); working out design problems and issues, "the project felt too much like school," the stress and pressure of having to complete an exhibit in a month (three girls each); and daily summaries and project organization (two each). There also appeared to be feelings of competitiveness between groups. In the words of one girl:

Some groups had to do less building than others. It left the groups with more building, like our group, feeling left behind from other groups such as the earthquake group.

Presents survey respondents' suggestions for improving the project

Girls had many suggestions to improve the project, including:

• More recreation/free time (mentioned by five girls)
• Spend less time on peripheral activities such as building the birdhouses and PowerPoint (4)
• Snack break (3)
• Cut down on first week activities (3)
• Better luncheon workshops: "the luncheon workshops were boring" (3)
• Stipulate the amount of building required, i.e., how much can be "shopped out" (2)
• A list of objectives for the month, broken down by task (2)
• Make roles for everyone clear (2).

Individuals also suggested "all female professors," more trips to museum "so that when we are told where the exhibit will be, we know exactly what that area looks like" and more structured daily schedules.

Excerpt 10 [University of Denver]

Interpretations & Conclusions:
Presents conclusions derived from examining responses on composite scales

The girls in Group 1 (SEM Program) and Group 2 (Control Group) were similar on several variables prior to the beginning of the SEM Program. On the Science Career Predictor Scale Total (and the Subscales) the girls in both groups showed high perceived interest and ability in science and mathematics. This perception remained constant during the year program for both the girls that participated in the SEM program and those who were not involved in the program. Girls were also similar on the Persistence Scale, suggesting that all girls, regardless of group, were persistent in how they approached problems, and had high perceived ability and interest in math and science.

(…)

Presents a perspective on key quantitative results

Since the girls all came to the program with high interest and ability, it would be difficult to increase these already positive attributes. Some interesting general information was gained by the girls on several aspects of School Climate, and SEM in general. The following information relates to the entire group of girls at Time 3 [Time 1=pre-program activities; Time 2=during program activities; Time 3=end of program activities].

• All girls believe it is Important (n=6) to Very Important (n=38) to their mothers that they attend college.
• All girls stated that it is Important (n=7) to Very Important (n=37) to their mothers that they attend college.
• All girls stated that it is Important (n=6) to Very Important (n=36) to their fathers that they attend college.
• The majority of girls stated that their mothers and fathers had talked to them about attending college, and about specific majors including science, mathematics, and engineering.
• All girls stated that they had the ability to graduate from college with degrees in SEM.
• Many girls thought it would be more difficult for females to get accepted into a science, engineering, or math degree program: Time 1 (57.5%) and Time 3 (41.5%).
• Most girls felt that teachers encouraged them to pursue math and science interests (76.2%).
• Most girls felt that there was no competition between girls and boys when it came to science and mathematics (67.4%).
• Most girls believed that boys and girls had: the same amount of interest in science and mathematics, received the same grades, had the same amount of knowledge in science and mathematics, and were treated similarly by the teacher.
• The difference between Groups 1 and 2 at Time 3 was in the areas of KNOWLEDGE about career options, years it required to complete a degree in SEM, and which high school courses were needed as the foundation for science, engineering, and mathematics degrees.
• Girls in the Saturday Program knew much more about the above areas compared to the Comparison Group, suggesting that the Program was effective in providing knowledge to girls who were already interested in SEM.

(…)

Draws conclusions from results of statistical significance (t-test)

There were significant increases in the pre- and post-test scores for the amount of information students perceived they had concerning careers in science, engineering, and mathematics (Questions 1-3). Another area tapped by the questionnaire was how well their mother and father could answer their questions in the three SEM areas (Questions 4-9). Except for Question 4 concerning students' perceptions of the mothers' ability to answer questions in science, students increased their confidence in both mothers' and fathers' ability to respond to questions. Additionally, students also stated that they gained information about careers in the three SEM areas at the end of the program.

Describes data limitations; draws implications for future research

Although these data need to be interpreted with great caution due to the small sample size, particularly for males, there are several considerations for future research. First, students stated that: (a) their teachers were quite positive about science and mathematics, (b) girls thought teachers treated boys and girls equally, and (c) girls described other girls at their school as having a positive attitude about science and mathematics. This is important because some research shows that girls lose interest in science and mathematics around 13 years old. Since these girls are in 6th to 8th grade, it is not known whether these girls' attitudes will change over the next few years. Longitudinal studies are needed to answer this question and to date, cross-sectional data have been used to study this question. If the conclusion is correct that girls lose interest in science and mathematics around age 13, then it will be necessary to intervene early. Data from this model program suggest that girls do still feel positive about science and mathematics in middle school.

Excerpt 11 [University of Washington]

Interpretations & Conclusions:
Presents generalizable conclusions

Given these findings, it was reasonable to conclude that indicators of a successful Women in Engineering Program include: a) percent of undergraduate degrees awarded to women; b) the size of the annual Women in Engineering budget; c) fundraising assistance for the program; d) fundraising responsibility of the director; e) use of evaluation, and f) major discipline of the director of the Women in Engineering Program. Neither student involvement nor enrollment statistics were found significant indicators of success. With respect to student involvement, everyone responding to the survey indicated student involvement; consequently, the measure was not fine enough to detect differences in degree of involvement. On the other hand, the small sample size probably accounts for the reason that enrollment did not appear as a significant indicator of success.

Expresses limits to the conclusions' generalizeability

As a practical application, these findings provide the criteria for evaluating the effectiveness of existing Women in Engineering Programs. As an example, the criteria could be used be either internal or external reviewers, including Deans as internal reviewers or federal agencies as external reviewers. However, one should be cautious when applying these criteria without consideration to the climate for women in engineering at an institution.

(…)

Presents generalizable conclusions

In conclusion, the findings of this study strongly suggest that Women in Engineering Programs are having an impact on increasing the number of women receiving degrees in engineering. The study found that there are six prerequisite conditions for successful programs and six criteria of success to be used for evaluation purposes. It also pointed out that commitment from the Dean, a designated full-time director, a WIE budget, faculty and student involvement, and program evaluation were critical components of a successful Women in Engineering Program.

Describes limitations of evaluation data

The primary limitation of the study was the small sample size. It is important to repeat the study again in 1993-94. As a result of the national technical assistance and training offered through WEPAN, the numbers of Women in Engineering Programs have increased significantly throughout the U.S. A larger sample size of Women in Engineering Programs would provide the opportunity to confirm the findings of this study, and to investigate other relationships such as type and size of institution with other variables.

Stakeholder Review & Utilization:
Describes results of use to multiple stakeholders

Finally, the findings in this study are important to a number of audiences: 1) provides deans with guidelines for the conditions that are needed to establish a successful Women in Engineering Program; 2) provides directors of Women in Engineering Programs with strategies to make the program successful; 3) provides funders with measures for evaluating success; and 4) provides policy makers with indicators of successful intervention programs, as well as a perspective on the national need to increase the number of women pursuing careers in engineering.

Excerpt 12 [Montana State University-Bozeman]

Interpretations & Conclusions:
Presents quantitative and qualitative responses

Findings

Wednesday AM: "Changing Pedagogy"
Mean = 4.1 (good+)

Highlights: Videos were very good.

Talking with teachers from schools to discover different approaches that they would use—very enlightening.

Suggestions for Improvement: Ideas could be kept to be compiled as a record of ideas and typed as bullets for the class.

There are many types of "self-esteem" type activities that are used at workshops as warm ups, etc. Maybe these could be used to break up sessions some so we don't sit so much. Northwest labs has some they use. I really like the handbook!!

Excerpt 13 [Indiana University]

Recommendations:
Recommends strategy for increasing survey return rate

F. Findings

The Survey Analysis Report can be found in Appendix B.

The surveys were used to evaluate the success of the program in meeting the three objectives of providing research skills, presentation and communication skills, and mentoring opportunities, peer support networks, and role models for women in science. The surveys also helped identify any trends or relationships regarding women and their retention in the sciences. There were difficulties in getting all Fellows and Faculty to complete the surveys. In the future, it would be useful to provide a greater incentive for survey and activity participation, perhaps by withholding part of the stipend till surveys are completed. A more structured mentoring program would also be more useful to the participants, along with an incentive for mentees to participate in the SURF surveys.

Interpretations & Conclusions:
Identifies strengths and weaknesses of the program

Survey Analysis: Fellows

The SURF Fellows clearly were intelligent and committed to working in the sciences. They had experience with supportive environments, both familial and educational. Many of them had strong science backgrounds prior to coming to Indiana University, yet it didn't appear that any of them had extraordinary experiences. Many of the Fellows were engaged in a number of activities at IUB. These women were quite focused in their educational and career interests. The majority of the Fellows knew prior to the Program that they were going to be science majors and none of them changed to majors outside the sciences while they were in the SURF Program.

The SURF program provided increased self-confidence and an increased commitment to continuing in the sciences for most of the participants that responded to the surveys. The Fellows gained perspectives on issues and concerns for women in the sciences. Throughout the program, most of the women respondents were engaged in the activities designed by the program and by their faculty advisors/mentors. The students appreciated the opportunities provided on learning about a number of different topics and hearing faculty speakers.

The Mentoring Program did not seem to go as well. Many of the students desired more structure. The contact with mentees was very limited for the majority of the Fellows. Contrastingly, the students appeared overall to be very comfortable with their faculty advisors/mentors. Only a few students offered negative feedback.

The research experience was reported to be a wonderful learning experience from almost all of the participants. They appreciated the laboratory experiences, the writing experiences, and the data collection experiences. Having the opportunity to write up research and present research appeared to have a positive effect on most of the research fellows.

Survey Analysis: Faculty

The SURF Program was assessed as a strong program by the faculty who responded to the surveys. The individuals who had the opportunity to work with the undergraduate fellows saw the benefits for the students and appreciated their assistance with their own research. Constructive feedback from the faculty centered on communication issues with the SURF program/WISP office and the relationships built with individual fellows. Overall, the program was assessed as having multiple benefits and particular advantages for undergraduates having the opportunity to be involved in research.

Excerpt 14 [Anonymous 4]

DISCUSSION AND PROGRAMMATIC IMPLICATIONS

Findings from this evaluation provide evidence of the academy's substantial success in providing a novel educational experience that extends students' experience of science and technology and affords a view of how home and school factors intersect with students' sense of themselves in relation to the academy's subject matter and activities. Findings can inform planning for future programming. In addition, data from the study can serve as a base for continued inquiry into the long-term impact of the academy for students and as a means to gauge the implications of diverse programmatic follow-ups initiated in participants' schools.

Interpretations & Conclusions:
Summarizes interview findings

Students' Initial Impressions

Students came to the academy with some preexisting view of it that had been shaped by teachers, parents, and peers, and, sometimes but not always, informed by their own sense of themselves in relation to science both as a subject and an activity. While teachers were said to have inspired a number of girls to participate in the academy, parents were said to have had more influence over boys' participation. Findings indicated appreciable differences in students' motivation toward the academy depending on age and grade level—younger students were motivated by fun and learning generally while older students had more purposeful interest. Academy participants drew parallels among an array of first-hand or vicarious and formal or informal educational experiences as similar to the activities included in the session. Students at the University L session were more likely than students at University K to feel they had some previous experience pertaining to robotics.

Recommendations

Programmatic Implications of Students' Initial Impressions of the Academy Everyday relationships and everyday activities and objects are potential resources for the academy to capitalize on in extending the reach of both its preparation of students for the academy and its continuation of influence after the session. More overtly drawing connections to everyday objects and processes might validate students prior related experiences, inspire future projects at home, and further draw significant people in students' lives into engagement in and support of their developing interests.

Interpretations & Conclusions:
Summarizes interview findings

Students' Experiences During the Academy

Building and programming were the two most frequently mentioned activities preferred by participants in the academy. In general, girls were equally drawn to programming and building, while boys were comparatively more enticed by building. Active hands-on activities were far more popular than didactic components of the sessions, although students generally recognized the necessity of both. Characterizing appealing activities as "hands-on" is accurate but incomplete. In addition to their active quality, tasks inherent in the creation of a robot entail numerous feedback loops in which students immediately see the effects of their work, which is simultaneously a social, cognitive, kinesthetic, and emotional experience. With varying amounts of intercession by academy staff, students selected their projects, organized themselves as groups and experimented with planning, design, building and programming until they produced the robot they envisioned. Depending upon a complex mix of students' age, previous experience, disposition, group work skills and personal work habits, students responded differently to the challenge inherent in project-based science inquiry. The major activity that each group undertook during the session was appropriately called the "challenge project," and challenge was a powerful theme characterizing much of the week. It is likely that for some students, pursuing challenge through an open-ended process-oriented project constituted a central learning of the academy.

Individual students responded differently to challenge and its necessary complement frustration. And therefore, the maintenance of an appropriate balance between challenge and frustration was a central preoccupation for academy staff in their facilitation of students' experiences. Staff maintained a vision of the ideal trajectory of the student robotics learning that guided their provision of assistance to students, but they continued to actively grapple with the issue of how much and when to offer assistance. At the end of each day staff met to discuss how the students were progressing, sharing perspectives on individual students and of partner groups and critically reflecting on their own facilitation of students' experiences. This was productive in provoking staff reflection and giving an open-ended discovery quality to staff's experiences.

Recommendations

Programmatic Implications of Students' Experiences During the Academy In addition to substantive learning about robotics and engineering, students' management of the technical and social aspects of undertaking a relatively long-term project is a central learning supported by the academy. The academy entails processes such as negotiation and planning, and emotions such as challenge and frustration. Effective facilitation of the academy includes managing students' own management of these processes and emotions, in addition to conveying substantive information about robotics. Staff's vision of a robotics-learning trajectory, included sequencing of concepts and activities, is a necessary pedagogic guide, although actual facilitation was equally creative as formulaic. Current school reforms typically include testing which tends to focus student and teacher attention to the end results of teaching and learning. This product-orientation is important to education but does not necessarily encourage students (or teachers) to seek challenge and therefore open-ended learning beyond task completion and assessment. The academy has the strength of affording opportunity for ongoing challenge and for constructive learning from errors and mistakes, through activities in which processes and outcomes are somewhat conflated. The cultivation of students' pursuit of challenge, rather than the mere completion of a functional robot, should likely be understood as the central pedagogic goal of the academy.

Interpretations & Conclusions:
Summarizes interview findings

Sex Composition of the Academy

Students held complex and diverse opinions about the sex composition of the academy. Of students who had a preference, girls were nearly evenly split between those who preferred a single sex academy and those who preferred a mixed session. Boys, in contrast, were most likely to prefer a mixed session. There was a slight tendency for older youth, both boys and girls, to prefer a mixed session and for younger girls to prefer a single sex session, but this pattern was not consistent between or within the two University K sessions. Female proponents of a girls-only session in the all girl University K academy were as adamant about its superiority as were those girls in the mixed University K academy who advocated the benefits of a mixed session. Given the option of selecting between attending an all-girl and mixed session, girls in University K tended to have purposefully enrolled in their session of choice.

Although some girls felt that the absence of boys would automatically create an improved learning environment, a few girls in the mixed session in University K characterized girls as more distracting and potentially more threatening than boys. Similarly, several boys in University K foresaw troubling distractions in a hypothetical all-boy session. Boys' and girls' opinions were informed by their experiences in school interacting with the same and opposite sex. At University L, several girls anticipated that a mixed session would provoke competition since in their experience boys believe that they are smarter than girls. While boys appeared to have consistent expectations of girls, some girls qualified their expectations of boys, conceding that, "it depends" which boys are in question. Regardless of their sex composition preference, girls who were most enthusiastic toward the academy tended to value "seriousness" in others, specifically the discipline not to "goof off" and the maturity to abstain from preoccupation with status, jealousy, and petty comparisons during the session.

Age was as influential as sex in students' partner selections during the sessions. Problematic partnerships could result from mixed age single sex partnerships; and in contrast, generally effective partnerships could result from similar age mixed sex partnerships. There was some indication that boys were more likely than girls to extricate themselves from undesirable partnerships—that is girls tended to quietly accept the objectionable arrangement while boys more actively moved to change their situation.

Recommendations

Programmatic Implications of Sex Composition of the Academy Findings from the study present no clear cut implications for how best to organize the sex composition of the academy: girls' preferences were mixed and contradictory, and while some had clear-cut preferences others were neutral; boys, similarly, were often neutral toward sex composition, but those who expressed a preference uniformly wanted a mixed group session, less because they wanted to work with girls than that they wanted to avoid inclusion of too many boys. In addition, despite differences between and within the three sessions—for example, the absence of preference for a mixed group session at University L compared to University K—the complexity of contextual differences between the three sessions makes generalizations problematic. However, the complexity of the findings does not support the conclusion that sex composition did not matter. On the contrary, sex composition mattered a lot to some girls and boys. Even students who neutrally accepted or preferred the presence of the opposite sex, could prefer to work in partnerships with students of the same sex. Consideration of sex composition and gender dynamics should likely not stop with the decision to organize a single or mixed group, but should be influencing partner selection and group work. In addition, issues of sex and gender manifest not only between the sexes, but also within the sexes. Specifically, findings indicated that age composition intersected with sex composition. Students of different ages responded somewhat differently to gender issues and sex composition and age mixing instigated issues among students of the same sex.

Interpretations & Conclusions:
Summarizes interview findings

Teachers in the Academy

Recommendations:
Discusses lack of clear implications

Programmatic Implications for Teachers in the Academy It is unclear whether students' reactions to their teachers' participation in the University L session would play out similarly in other contexts, but given the positive response, more experimentation with this format seems warranted.

Interpretations & Conclusions:
Summarizes interview findings

Students' Future Career Thinking

Most students in the academy held some view of their future academic and career trajectory although boys were more likely than girls to express indecision. While the majority of students saw some form of science as a viable option, some foresaw multiple possibilities that entailed divergent trajectories or incoherent assumptions. For instance, both cosmetology and psychiatry were posited as possible options by one eighth grader, and another considered becoming a pediatrician or a childcare worker equally possible. A number of boys envisioned a possible career in professional sports, although most also had alternative plans as well. Family members, especially mothers, were often referenced as playing a role in career decision-making. Several students at the University L session who were interested in engineering spontaneously mentioned that the academy had informed their thinking,

Recommendations

Programmatic Implications Ideas about students' prospective future academic and career paths link to complex views of personal identity— "the kind of girl (or boy) I am"— as well as to views of personal capacity. Findings indicated that awareness of engineering increased for some students, namely girls in the University L session. Students had generally high interest in science, and therefore connecting robotics to other sciences, such as biology and health, might further bolster the academy's relevance to students' emerging interests. It is possible that longitudinal research will indicate additional types of influences the academy experience had on students' career-related interests, sense of validation of interest, view of personal capacity toward science and technology, and embrace of group work and project-oriented inquiry.