Evaluation Report 1996-1997
Biotechnology Initiative for Systemic Change in the Teaching of Science (BISCITS)

This evaluation report was prepared by Clarion University of Pennsylvania.

Table of Contents:

1. Background
   • Project Description: Project Features, Project Participants, Audiences & Other Stakeholders
2. Project Evaluation: Overview of Design
   • Evaluation Overview: Evaluation Purposes, Evaluation Questions
   • Design: Information Sources & Sampling
3. Evaluation Results
   a. Question #1
   • Design: Instruments
   • Analysis Process: Quantitative Analysis
   • Results & Recommendations: Interpretations & Conclusions
   b. Question #2
   • Design: Instruments
   • Analysis Process: Qualitative Analysis
   • Results & Recommendations: Interpretations & Conclusions
   c. Question #3
   • Evaluation Overview: Evaluation Questions
   1. Concerns:
   • Design: Instruments
   • Analysis Process: Quantitative Analysis
   • Results & Recommendations: Interpretations & Conclusions
   2. Biotechnology Teaching Self-Efficacy
   • Design: Instruments, Data Collection Procedures & Schedule
   • Analysis Process: Quantitative Analysis
   • Results & Recommendations: Interpretations & Conclusions
   3. Other Attitudes
   • Design: Methodological Approach, Instruments
   • Analysis Process: Quantitative Analysis
   • Results & Recommendations: Interpretations & Conclusions
   d. Question #4
   • Design: Methodological Approach
   • Results & Recommendations: Interpretations & Conclusions
   e. Question #5
   • Design: Data Collection Procedures & Schedule
   • Results & Recommendations: Interpretations & Conclusions
   f. Question #6
   • Analysis Process: Qualitative Analysis
   g. Overall Impact of Summer Program
   • Results & Recommendations: Interpretations & Conclusions
   h. Effective Aspects of the Program
   • Results & Recommendations: Interpretations & Conclusions
   i. Areas Needing Improvement
   • Results & Recommendations: Interpretations & Conclusions, Recommendations
   j. Labs
   • Results & Recommendations: Interpretations & Conclusions, Recommendations
   k. Summary of Evaluation Findings
   • Results & Recommendations: Interpretations & Conclusions, Recommendations

Evaluation Report 1996-1997
Biotechnology Initiative for Systemic Change in the Teaching of Science (BISCITS)

Return to Table of Contents

Background

BISCITS is an intensive program, funded by the Teacher Enhancement Division of the National Science Foundation to support middle and high school teachers in developing contemporary understandings of molecular biology and biotechnology subject matter, career options in these fields, and ethical issues in current experimental work in these areas. In addition, it is intended that the participants will explore and use biotech-related resources on the Internet, and develop new teaching methods for use in their classrooms.

A four-week summer residential program was held from July 7, 1996 through August 2, 1996, at the Clarion University campus located in Clarion, PA. Through daytime and evening sessions, participants engaged in lectures, laboratory activities, guest speakers, discussions, role plays and actual biotech-related teaching to peers and minority high school students. Twenty-four participants attended the 1996 summer session. The median age of participants was between 30 and 39 years and the average participant has 11 years of experience teaching science.

The project continued into the 1996-97 school year, when participants integrated BISCITS-inspired activities, lessons, experiments, etc. into the curricula in their middle school or high school classrooms. In addition, participants provided leadership to their peers in the area of biotechnology by providing many professional development workshops over the course of the academic year.

The BISCITS team offered support to each of the participants in order to maximize the impact of the program. Through a travelling biotechnicican, loans of equipment and supplies, and sharing of successful biotechnology-related activities, the BISCITS staff was available so that teachers were able to integrate biotechnology activities and issues into existing school curricula.

More specifically, the BISCITS program endeavored to:

1. Develop participants' understanding of contemporary biology and biotechnology.
2. Develop the skills (including leadership) and materials such that the participants will be able to share newly adopted curricular materials with their peers.
3. Disseminate laboratory experiences which have been demonstrated to be effective and have high potential for implementation.
4. Develop new classroom application from their understanding and skills in molecular biology and biotechnology, and the related science-based societal problems including ethical issues.
5. Develop networks between biologists, science educators, and secondary teachers.
6. Develop curricula (which is at least 50% laboratory based) and pedagogy applicable to all students in life science and chemistry in junior high and high schools.
7. Provide support mechanisms to increase the likelihood of implementation in the participant's classroom or laboratory.
8. Establish partnerships between the university, public schools, the government, and business and industry.
9. Evaluate cognitive, attitudinal, and the effects of the workshop on teaching behaviors and the impacts of extended inservice.

Return to Table of Contents

Project Evaluation: Overview of Design

The primary purpose of the evaluation is to provide data-based feedback to the project directors and program officers about the ways in which BISCITS is meeting its objectives. This report is a summative for the participant group that joined in the summer of 1996. That group completed a four-week intensive summer program, integrated BISCITS-inspired activities and issues into their existing curricula, and provided leadership to other science educators by offering professional development experiences connected to the BISCITS agenda.

The 1996-97 evaluation addressed a number of questions about the implementation and impact of BISCITS. The evaluation is structured around these guiding questions:

1. How do participants' understandings of contemporary biology and biotechnology change as a result of participation in BISCITS?
2. In what ways do participants' use of biotechnology-related lessons change as a results of participation in BISCITS?
3. In what ways do participants' concerns about biotechnology teaching, teaching self-efficacy, and other attitudes change as a result of BISCITS?
4. What was the impact of support mechanisms, such as networks with scientists and the traveling biotechnician on promoting the objectives of BISCITS?
5. To what extent and in what ways do participants share BISCITS-inspired ideas about content and pedagogy with peer?
6. How could BISCITS be improved to better support the development of science teachers in the areas of understanding and teaching microbiology and biotechnology-related topics and concepts?

The 1996-97 evaluation of BISCITS involved the collection and interpretation of data from multiple data sources. Primary data sources included: 1) content knowledge tests, 2) participant responses to a variety of questionnaires, 3) field notes from observations of classrooms where participants implemented BISCITS-related biotechnology lessons, 4) individual and focus group interviews with participants, and 5) artifacts collected from participants and their students while implementing biotechnology lessons (e.g., lesson plans, student assessments). All 24 participants are included in the evaluation. Demographic data including variables such as gender, years of teaching, number of students taught, etc. were collected to explore and possibly explain differences noted in participants.

Return to Table of Contents

Evaluation Results

This section is organized by the six major questions guiding the evaluation study. Under each sub-section evaluation sub-questions, data collection, and analysis techniques are described.

Question #1

How do participants' understandings of contemporary biology and biotechnology change as a result of participation in BISCITS?

A content examination was constructed by the project directors for administration in a repeated measures design. Items for this test were developed by guest and resident lecturers based on the content of program activities. The pre-test was administered on the first day of the summer program and the post-test was given on the last day. All 24 participants completed both the pre-test and the post-test. The test had two scales: one scale tested understanding of biology and biotechnology concepts and the other assessed understandings of science pedagogy. Hence, sub-questions were pursued about participants' performance on both scales as well as the entire examination.

A paired t-test was conducted to determine if there was a difference between pre-test and post-test scores on the overall test and on both scales of the test. Table 1 summarizes the performance of the participants on the test. Analysis indicates that the differences between pre-test and post-test scores on both scales and on the test overall are significant at the p< .001 level leading to the conclusion that participants increased their understandings of both content and pedagogy over the course of the summer program.

In addition, participants were asked on a course evaluation form administered at the end of the summer to indicate to what extent they agreed with the statement that BISCITS increased participants' knowledge substantially. Nearly 92% of all participants indicated that they strongly agreed or somewhat agreed with that statement.

Observations of participants during the school year provide further evidence of this gain. In an interview conducted prior to or following each observation, participants were asked to describe the extent to which the BISCITS program provided a knowledge base for the lesson observed. In all cases the participants said that the lesson observed had not been taught previously and that BISCITS helped the participants develop understandings necessary to teach the topic well. For example, Teacher A's lesson was about recombinant DNA and she interwove ethical issues that might arise from recombinant work. She claims that her knowledge of this technology as well as the ethical dimensions were introduced to her for the first time during the summer program.

Return to Table of Contents

Question #2

In what ways do participants' use of biotechnology-related lessons change as a result of participation in BISCITS?

Primary data for this question comes from responses to a Levels of Use Questionnaire and from classroom observations, classroom artifact review, and participant interviews. Sub-questions are:

2a. How does participants' behavior in teaching biotechnology-related lessons change from the beginning of the summer to the end of the 1996-97 academic year?

2b. How has BISCITS inspired particular lessons, activities, student investigations, and assignments during the 1996-97 school year?

Question 2a is addressed in this report through an analysis of participants' responses to a questionnaire developed from the Levels of Use dimension (Hall, Loucks, Rutherford & Newlove, 1975) of the Concerns Based Adoption Model (Hall, Wallace & Dossett, 1973). A questionnaire was developed using the interview guide provided in the Levels of Use manual (Loucks, Newlove, & Hall, 1975). It was administered in July 1996, August 1996 and May 1997 and questions based on the questionnaire were asked during interviews throughout the year. The initial administration took place on the first day of the summer institute and provided baseline data from which to compare changes in the use of biotechnology teaching.

The LoU questionnaire was analyzed using the procedures described by Loucks, Newlove, and Hall (1975). Two raters examined participant responses to items on each administration of the questionnaire in order to place each participant at an LoU for each category and to assign an overall LoU rating. All responses were reviewed independently by the two raters and tallied were kept on a LoU Rating Sheet. Each LoU category and the overall LoU were rated separately by each rater. An important aspect of the analysis is the determination by the rater of the "global picture" of the LoU for each participant. This final rating is not directly derived from a scoring grid but requires interpretation by the rater taking into account all of the participant responses. Data from 3 participants were not used due to incomplete or unreadable responses on the baseline questionnaire. In all, questionnaires from 21 participants were used.

After assessing questionnaires separately, the two raters discussed each of the responding participants. There was agreement on the remaining from 16 participants from the outset of the discussions. In the case of the overall LoU of 5 participants, the discussion can best be characterized as a negotiation. Discussion continued until both raters agreed on the overall LoU for all 21 participants (see Table 2)

At the beginning of the BISCITS program in early July 1996, there appeared to be a relatively clean tri-modal distribution of participants on the overall LoU rating. While each of the overall levels will not be defined at this time, those levels in which participants have been grouped will be described.

Eight participants were assigned an overall rating of Level O or Level I. Level O is a state of non-use where the participant has little or no knowledge of biotechnology and has had limited involvement in teaching biotechnology-related lessons. All of the teachers at these levels say they do not teach biotechnology topics at present. Of those assigned to Level O, there is a mix between participants who do not see how biotechnology can fit in their curricula and those who feel they do not have adequate knowledge or skills to teach these topics. In addition, two participants felt that middle school students cannot comprehend biotechnology topics and offered student developmental abilities as a reason for non-use. The one respondent assigned to Level I is highly articulate in her interest in and commitment to teaching biotechnology once she overcomes her knowledge limitations. She expressed that this need to know more is what motivated her to participate in BISCITS.

Eight participants were assigned to Level III, where use of biotechnology teaching is generally limited to short-term, day-to-day, mechanical aspects. Among mechanical aspects specifically mentioned, getting and maintaining equipment and meeting curricular constraints were mentioned often as factors influencing the use of biotechnology teaching. Finding ways to make biotechnology topics like genetic engineering concrete was mentioned by one participant as a major interest, as he only lectures about it right now.

Level IVB, to which 6 participants were assigned, is a level of use that is characterized by participants who already teach biotechnology and are seeking refinement in their biotechnology teaching. Participants at Level IV seem most concerned with maximizing impact of biotechnology teaching on students. In general, this group expressed much knowledge about and enthusiasm for biotechnology teaching in schools. All of these 6 participants raised concerns that this field is rapidly changing and that they need to keep abreast of advances in knowledge, experimentation techniques that are do-able in schools, and general pedagogy. They feel that they teach biotechnology in an acceptable way at present but look to BISCITS to help them refine their understandings and skills, especially by providing new ways to teach biotechnology topics. The desire to enhance pedagogy and keep it interesting to students through real-life examples like the O. J. Simpson criminal trial and information about genetically-engineered mutants is a clear indication of refinement of use.

In contrast, Levels of Use data collected at the conclusion of the 1996-97 school year in May 1997 indicate that all participants are users of biotechnology education. Table 3 provides a comparison between the beginning and end of the program. Three of the participants can be characterized as occasional biotechnology teachers. Demographic data on these three individuals suggest that two of these participants are not in teaching contexts where biotechnology easily fits into existing curricula. The other person states that he hopes to "do more with BISCITS stuff next year." Ten of the participants articulated interest and excitement about their biotechnology teaching and were seeking additional information about ways to be more effective. The other eight participants can be characterized as advanced users of biotechnology education. Interestingly, all eight expressed a desire to become even better biotechnology educators.

Given the apparent trimodal distribution at the beginning of the program and the bubble of participants at the end of the program near the higher levels of use, it can be inferred that these participants were not only able to incorporate activities presented to them during the summer program but were able to actually internalize the value of biotechnology education for their own classrooms. Some other interesting "use" data:

• Eight participants stated that they initiated field trips to biotech companies or university labs for the first time during this project in order to augment their curricula.
• Twenty-three participants reported using "many" of the activities presented during the summer program. Rebops, DNA spooling, paper electrophoresis, rainbow electrophoresis, and cell soup were mentioned most frequently by participants as activities they used during the school year.
• Two participants mentioned that they were impacted by the need to make activities more inquiry-based and adjusted many typical labs and classroom activities to make them more student-cantered.
• Three people indicated that they had or were about to have the opportunity to rewrite curriculum guides and that biotechnology would feature prominently.

Return to Table of Contents

Question #3

In what ways do participants' concerns about biotechnology teaching, teaching self-efficacy, and other attitudes change as a result of BISCITS?

This question was pursued in three prongs:

3a. What were participants' concerns about biotechnology teaching and how did participants' concerns change over the course of the year?

3b. What were participants' perceptions of their biotechnology teaching self-efficacy and how did that change over the course of the year?

3c. What other attitudes or beliefs about biotechnology and biotechnology teaching were held by participants and how did those change over the course of the year?

Return to Table of Contents

Concerns

Question 3a is grounded in the major theoretical underpinnings of this evaluation project, the Concerns Based Adoption Model (Hall, Wallace & Dossett, 1973). The Stages of Concern Questionnaire (SoCQ) is the tool designed to capture teachers' concerns about adopting an innovation. The SoCQ uses a Likert-scale response format to measure seven hypothesized stages of concern individuals have toward implementing an innovation. The SoCQ consists of 35 statements (five items for each stage) which allow respondents to describe a concern they feel at a given point in time. The seven hypothesized stages are awareness, informational, personal, management, consequence, collaboration, and refocusing. Test reliability was reported by Hall to be satisfactory, based on internal consistency estimates of .64 to .83, and test-retest reliability of .65 to .86. The SoCQ was administered pre-workshop, post-workshop, and end of the school year. Stages of Concern data plotted over time will allow for greater insight into the affective stance that the teacher is taking toward implementing biotechnology.

Stages of Concern profiles were constructed for each of the participants. Responses from both administrations of the SoCQ were analyzed and plotted on a grid. Pre-test/post-test profiles for each participant were constructed for closer scrutiny of the SoCQ data on an individual level. Those profiles are not included in this report. Interestingly, the data from pre and post summer vary from CBAM theory. Theoretically, peak stage scores of participants should progress from lower Stages to higher Stages as individuals move from awareness and nonuse into beginning use and then more sophisticated use. It is hypothesized by Hall, George and Rutherford (1979) that "concerns develop from being most intense at Stages 0, 1, and 2, to most intense at Stage 3, and ultimately to most intense at Stages 4, 5, and 6" (p.34). What is noted in this study is that participants' concerns in the lower stages increase and concerns in the upper stages decrease at the conclusion of the summer program. The group data also show that overall concern about biotechnology teaching increases at the end of the summer, as evidenced by higher overall percentile scores in five of the seven stages. Table 3 and Figure 1 summarize the SoCQ group data set.

The raw scores can be useful as well in interpreting the SoCQ data. As the raw score data are derived from responses to a Likert-type scale, ordinal level data are available for analysis. Robust parametric analyses of ordinal data (e.g., ANOVA) is not advisable, given that the basic assumptions about population parameters cannot be met. A more conservative, non-parametric analysis to test statistical significance is most appropriate in this case. A Wilcoxon signed rank test was used to determine if their was significance difference in the pre-test/post-test stage scores. This test is analogous to the t-test. The data from pre-summer and post-summer were subjected to this test. From this test a significant difference exists on Stage 0, Stage 1, and Stage 2 scores. Table 4 summarizes the results of this test.

Likewise, the end of the year data are useful for comparison. A Wilcoxon signed rank test was used to determine if there was significant difference in the pre-summer and end of the year stage scores. Significance was noted in Stage 1 and Stage 6 as noted in Table 5. From the table it can be noted that the end of the year scores on Stage 1 decreased significantly while the end of the year scores on Stage 6 increased significantly. Overall, this patter was expected and serves to confirm that BISCITS participants decreased in their need for information about biotechnology education and increased in their interest/concerns about refocusing or revising their approach to biotechnology education.

The deviations from the predicted pattern in responses to the SoCQ in the pre and post summer data set is likely to be best explained by changes in participant perceptions about all that is involved in molecular biology and biotechnology teaching. Participants self-selected to participate in BISCITS for two main reasons: 1) They already teach biotechnology and want to learn new and better ways to teach, and 2) They do not teach biotechnology and know little about it. Post-summer data show that all participants saw dimensions of molecular biology and biotechnology they never knew before. The summer BISCITS workshop apparently helped participants identify aspects of teaching molecular biology and biotechnology that were not fathomed as the summer began. The significant increase in scores of Stages 0, 1, and 2 suggest that even those participants who came to BISCITS with biotechnology teaching experience recognized that they had limitations in their knowledge base about teaching biotechnology. At the conclusion of the summer session all participants seemed to be saying that they had an increased need for more information about biotechnology teaching and for understanding their role in teaching biotechnology in the classroom.

In summary:

• Stages of Concern data suggest that at the end of the school year most participants are users at some level, with about a third of them able to be classified as very experienced.
• Experienced or not, there seems to be a very high interest among all participants in collaborating with others. In fact, 13 participants scored in the 80^th percentile or above on this dimension on the SoCQ. The anecdotal data further indicate that all participants are very attracted to biotechnology education, and think the topic is "hot." It is unclear why they wish to share what they know and can do with their colleagues. While in the cases of most participants it appears that they wish to share because they are seeking to provide leadership, it is possible that at least some of the participants feel uncertain in this area and want support from others.
• Most of the 96-97 group seems relatively unconcerned with management issues - this might reflect either confident veteran teachers or relatively experienced users of this innovation (or both?).
• In the data set there is about an even split between those participants relatively unconcerned about consequences, those moderately concerned, and those highly concerned.
• At the end of the year, about half seem relatively unconcerned with getting some more information about the innovation, the other half seem as if some more information would be welcome.
• There seems to be only moderate to low personal concerns for almost all respondents.
• There's about an even split between those with high refocusing concerns and those with moderate refocusing concerns, only two seem to have relatively little concern in this area.

Return to Table of Contents

Biotechnology Teaching Self-Efficacy

Evaluation question 3b asked "What are participants' perceptions of their biotechnology teaching self-efficacy and how does that change over the course of the year?" Teaching self-efficacy is a psychological construct proposed by Ashton and Webb (1982). Riggs and Enochs (1989) carried the idea one step further to the study of science teaching self-efficacy as they proposed that teaching efficacy beliefs appear to be dependent on a specific teaching context. The biotechnology teaching instrument was designed to capture teacher-participants' perceptions of their own biotechnology teaching self-efficacy. Biotechnology-specific items were modeled after the STEBI-Science Teaching Efficacy Beliefs Instrument (Riggs & Enochs, 1982). The items were modified to reflect a biotechnology context. The original instrument has two scales: Personal Science Teaching Efficacy and Science Technology Outcome Expectancy. The biotechnology instrument has two scales as well: Biotechnology Teaching Efficacy Beliefs (BTEB) and Biotechnology Teaching Outcome Expectancy (BTOE). The instrument to measure biotechnology teaching efficacy will be a by-product of the evaluation of BISCITS over the next several years and has not been established at this time. Response choices ranged from strongly disagree (1) to strongly agree (5). Sample items from this part of the questionnaire are:

I generally teach biotechnology topics ineffectively. (BTEB)

The low achievement of some students during a biotechnology unit cannot generally be attributed to their teachers.(BTOE)

The first administration of the biotechnology teaching self-efficacy occurred on the first day of the summer institute. The second administration occurred on the last day of the summer workshop. An additional administration occurred at the end of the school year. Group means for both scales are report in Table 6. The biotechnology teaching self-efficacy of the BISCITS participants increased throughout the year. A statistically significant increase is noted between the pre-summer scores and the end of the year scores. Although not shown here, a significance difference was also obtained by comparing pre-summer and post-summer data. This finding suggests that the summer program appears to be an important factor in developing a feeling of self-efficacy.

To determine if the entire BISCITS year had a positive influence on either the enhancement of biotechnology teaching self-efficacy and/or outcome expectancy, a nonparametric sign test was employed since a Likert-type scale was used on the self-efficacy instrument. The results of the tests yielded a significant difference on the self-efficacy scale but not on the outcome expectancy. (See Table 7).

Return to Table of Contents

Other Attitudes

Other attitudes and beliefs about biotechnology and biotechnology teaching were explored via items on a questionnaire administered in a repeated measures design. Participants responded to a series of items regarding their perceptions of biotechnology and their attitudes about teaching biotechnology-related lessons. Response choices ranged from strongly disagree (1) to strongly agree (5). Sample items from this part of the questionnaire are:

I have a strong working knowledge of biotechnology topics.

I like experiments in biotechnology.

The pattern of participant responses on both the pre-test and post-test is included in Appendix A in Part B of the questionnaire. Statistical tests for significance resulted in significant positive change in perceptions on the following items:

1. I could explain biotechnology to someone who didn't know anything about it. (p <.001)
2. I like biotechnology. (p < .05)

30. I like experiments in biotechnology. (p < .05)

1. I have a strong working knowledge of biotechnology. (p < .000)
2. I feel I have the materials and resources necessary to implement biotechnology curricula. (p < .005)
3. I have sufficient knowledge and skill to use electronic technologies such as computers to enhance my biotechnology teaching. (p < .05)

Changes noted on these additional attitudinal items support other data from this evaluation that suggest that participants perceive great benefit from the summer experience. Item 32 in particular supports the idea that participants developed much greater understandings of the breadth and depth of molecular biology and biotechnology as a result of the summer workshop, lending additional support to the notion that participants held limited conceptions of the nature of biotechnology at the beginning of the summer. Furthermore, the change noted in item 38 serves as an additional testament of the value of including Internet and hands-on equipment time in the summer workshop.

Return to Table of Contents

Question #4

What was the impact of support mechanisms such as networks with scientists and the travelling biotechnician on promoting the objectives of BISCITS?

The support features of the BISCITS program were described by each and every participant as a facilitating factor. About three quarters of the participants stated that the traveling biotechnician and the load of supplies strongly influenced their ability to implement BISCITS-inspired lessons. In six cases, the participants felt they would have been completely unable to offer certain laboratory experiences to their students if BISCITS did not provide support in terms of supplies. In addition, nearly all participants mentioned at some point that they felt the BISCITS staff provided "teacher tested" classroom ideas that were very likely to be successful with their own students.

The support features appears to be a useful safety net in allowing participants to get their feet wet in integrating BISCITS-inspired activities and ideas into the curriculum. Five of the participants mentioned that they were concerned that it would take many years to build up the resources needed to effectively teach biotechnology now that they know what specialized materials are available and how to use them. In one case, a participant said her budget for laboratory supplies was cut for next year to $0. Another participant stated that the lack of proper specialized equipment was his greatest limitation.

In addition, ten participants said a lack of a double period or extended laboratory time in the school schedule makes it nearly impossible to teach certain labs that they see as essential investigations. One participant reported that having only 40 minute periods required more planning time in order to sequence certain lab activities into manageable segments. One participant said that he only had a single period for his biology class and that he was not sure if breaking certain activities into parts over several days was productive for students.

It may be helpful to follow up with these participants during the next school year to determine the extent to which the perception of limited supplies or time frames impacts the curricular choices of participants. In particular, it would be useful to learn how participants navigate the supplies, budget, and time issues and come up with creative, low-cost solutions.

Return to Table of Contents

Question #5

To what extent and in what ways do participants share BISCITS-inspired ideas about content and pedagogy with peers.?

Data about peer instruction and professional development experiences has been monitored by the in-house BISCITS staff.

Return to Table of Contents

Question #6

How could BISCITS be improved to better support the development of science teachers in the areas of understanding and teaching microbiology and biotechnology-related topics and concepts?

Data to answer this question come primarily from questionnaires participants completed at the end of the summer program and the end of the school year. Supporting data come from informal conversations with participants, field notes taken during my visits to the summer program and follow up sessions, and electronic mail.

Data from the evaluations were analyzed inductively. An open-coding procedure gave some organization to participant comments. Codes and their respective quotes from the data set were clustered resulting in three main themes: 1) overall impressions and personal impact, 2) aspects of the program that were effective, 3) aspects of the program that need improvement.

Return to Table of Contents

Overall Impact of Summer Program

Overall, there is a high level of satisfaction with the summer program among participants. All of the participants (100%) reported on the end-of-the-course evaluation that BISCITS was useful in that it provided information and experiences that were applicable to their teaching and the probability was high that BISCITS would have significant, long-term benefits to their teaching. The following quote from one of the participants captures the positive impressions nearly all of the participants reported:

I have many good impressions from the BISCITS workshop. The instructors were extremely prepared, organized, and incorporated a wide variety of biotechnology activities, labs and discussions. The activities allowed us to think instead of having an activity that we would know the answer. It is this type of hands-on lab that I want my students to experience. The topics were relevant to all of us and can affect everyone by its uses and applications. I would describe this as an excellent program for all teachers. The most important impression that I leave this program with is my progress. This workshop has allowed me to go full circle from a person with little confidence and working knowledge of biotechnology to someone who feels comfortable and eager to teach these topics to my students. (brief eval, 8/2/96).

Another person echoed those feelings and added that BISCITS is better than other professional development workshops: "I personally have learned more in these 4 weeks of theBISCITS workshop than in any workshop I have ever attended." (brief eval, 8/2/96).

A broader awareness of the nature of biotechnology and new understandings of pedagogical strategies for teaching biotechnology appear to be two areas where participants report the greatest impact. Most participants indicated that their conceptions of molecular biology and biotechnology were rather restrictive prior to BISCITS and the greatest impact was a new understanding of the science areas that related to biotechnology:

My definition of biotechnology has expanded to include more topics than the traditional concept of biotech, i.e., only genetic engineering. (follow-up, 10/96)

I learned so much about DNA. I had to read up on topics in the library almost every day because I realized I had limitations in my basic knowledge of the topics we were discussing. (brief eval, 8/2/96)

The workshop has broadened my idea of biotechnology. There are many areas that relate to it that I hadn't connected before, i.e., meiosis, enzymes, Mendelian genetics, ethics. (follow-up 10/96)

I was not even aware of a biotechnology section in the science catalogs. It is not only an awareness but it's a new view to the future of my own science teaching in molecular biology. I went home and ordered $10,000 of biotechnology equipment. (follow-up, 10/96)

Additionally, participants' written evaluations reflect a new commitment to teaching molecular biology and biotechnology to all students:

As a result of the 4-week BISCITS workshop my interest in teaching molecular biology and biotechnology has increased tremendously. I will be teaching a six to eight week short course on molecular biology and biotech in my Biology II class. I will also be introducing molecular biology and biotechnology into my General Biology classes. (follow-up, 10/96)

I'll be teaching a semester course (18 weeks), divided into genetics and biotechnology. The BISCITS program has provided information and experience that will greatly help my presentations and lab activities during the 9 weeks of biotechnology. (brief eval, 8/2/96)

I will definitely incorporate more lab activities dealing with Biotechnology into my Biology classes. I will also use Ethics Discussions concerning Biotechnology. Such discussions really force a student to think about controversial topics and to be able to explain why he or she feels that one choice of action is better than another. (brief eval, 8/2/96)

In summary, participants report that BISCITS had the greatest impact in helping them broaden their personal understandings of molecular biology and biotechnology, gain motivation and confidence to include more biotechnology in their teaching, and develop a collection of classroom and laboratory activities that support good biotechnology teaching.

Return to Table of Contents

Effective Aspects of the Program

The summer BISCITS program was filled with many opportunities to learn molecular biology, biotechnology and pedagogy. Everything that was done during the summer was mentioned at least once by at least on participant as being effective. Overall, three events from summer 1996 drew attention and reaction from participants: laboratory investigations, sharing/peer teaching, and teaching actual students.

Investigations in the laboratory were billed by the project directors as a major tool in the summer program to help participants understand the content and pedagogy of biotechnology, and it was noticed by the participants as it was mentioned by all 24 participants as the best part of the program. Actually doing laboratory activities as they might do them with students resonated among the participants as an effective way to help them learn content as well as the laboratory procedures (e.g., equipment use). One participant said, "Using and doing labs is really the only way that a teacher feels comfortable trying it in the classroom. That's what made this experience great" (brief eval, 8/2/96). Other supportive quotes include:

The most effective part for me was the many interesting labs we performed. (brief eval, 8/2/96)

We did labs like I want my students to experience. (brief eval, 8/2/96)

My lab skills improved tremendously over the past four weeks. (brief eval, 8/2/96)

Sharing teaching ideas with other teachers and conducting peer teachers were also rated high as effective components of the summer program. One participant said that "I always learn a lot when I listen to fellow teachers who are willing to share what works for them." (brief eval, 8/2/96). Other participants shared this perception about the role of collegiality and collaboration in their learning during the summer session:

We (participants) learned from each other. The program was excellent for me because it brought science teachers together for information exchange. (brief eval, 8/2/96)

Peer teaching and sharing lessons was great! Teachers need more time for collaboration! I learned so many ways to involve new ideas and incorporate new activities into my classroom. (brief eval, 8/2/96)

I really enjoyed the peer teaching nights where we shared ideas from our fellow teachers. (brief eval, 8/2/96)

The intensity of the program over the four weeks in July, coupled with participants living and working together, helped create a sense of collegiality that seems extremely important in developing comfort with new aspects of teaching and building confidence. Many of the teachers supported the comment above about teachers wanting and needing collaboration. They expressed frustration that school constraints such as tight schedules and too much work does not support collaborative work environments. BISCITS allowed teachers to bond with each other. As additional evidence of the value and importance of this aspect of the program, all 100% of the participants indicated that they strongly agreed or somewhat agreed with a statement on a final course evaluation that asked their opinion about the workshop providing opportunities to "describe my work or learn from other participants."

Another aspect of the summer program that was identified as effective was the time that the participants taught students from the STIMM and Talent Search programs. BISCITS program directors arranged a coordination of BISCITS and these programs which bring minority high school students to campus. Over two days, BISCITS participants were able to apply what they had learned during the summer and see firsthand how high school students might react to lessons or activities they had never taught before. Some participant quotes about this aspect of the program include:

Another highlight was the two days we had the students and we could actually try some of the Biotechnologies that we had learned during the workshop. (brief eval, 8/2/96)

Actually working with students… makes me feel more confident in taking ideas back (to my school) and using them with my students. It was also a great experience because I got to work with other teachers and see their instructional methods. (brief eval, 8/2/96)

Implementing what we learned with the Talent Search and STIMM students was the best event. I was able to see how students respond the this technology, its use and importance in our lives. It convinced me that I must work this technology into my chemistry teaching. (brief eval, 8/2/96)

Return to Table of Contents

Areas Needing Improvement

All programs have components that are not rated very high by participants. Information about aspects of BISCITS that participants did not like is useful in organizing the remaining follow-up sessions and planning for the next summer program. Six areas were identified by participants as needing improvement. Four of the six areas were mentioned by only one or two participants and therefore will not be reported here. The other two areas - time and labs - warrant discussion as they were mentioned by a majority of participants.

Time. Time was the program aspect most often identified as needing improvement. Two kinds of time were reported: overall duration of the program and use of time during the program. Fourteen participants stated that the length of the program - four weeks - was too much given the other demands on their lives during the summer. Twelve of these participants suggested that a three-week format would have accomplished the same effects as the four week session. The issue of overall duration and the impression that it could have been shorter is closely related to the issue of the use of time.

Nearly all participants expressed frustration and concern that time was not used efficiently during the four-week session. Comments from the end of the summer evaluation form that support this interest in better time management include:

Too much lag in time between activities.

I am frustrated with the daily schedule - too much time was down time.

Scheduling and planning needed major improvement.

The evening sessions were totally unnecessary and added nothing to the program - just made me even more tired. I wish I could have had the evening to catch up on reading or just chatting with the other teachers bout ideas for the classroom.

I found there was a lot of "down" time in the lab when 24 people needed to use only one or two sets of materials.

Some of the speakers went on and on. What they had to say could have been said in 10 minutes.

Some days dragged on in an endless manner.

We put in 158 hours! The days were way too long (my brain became a saturated solution).

I needed more time away from Pierce [the building].

Opportunities for teachers to get together to combat information overload is definitely needed.

Too much information scattered across a very long day (8:30 a.m. - 9:00 p.m.). I know the lecturers/presenters could "get to the point" in a shorter amount of time reducing the need for evening sessions.

Interestingly, some of the same people who said that the extended time was useful in that it allowed them to explore the molecular biology and biotechnology topics in-depth. Several participants commented that they had not learned so much about science since they were in college. In fact, one participant said, "While I think 4 weeks is too much time, I have no idea how they could have done it in 3 weeks" (brief eval, 8/2/96). Another said, "There was a lot to learn. Shortening the program would make it too grueling and maybe not afford as many opportunities" (brief eval, 8/2/96). It appears that the concern in overall duration is related to the concern in time management. While the schedule was organized and planned in advance, the project directors may wish to examine the use of time in the daily schedule carefully.

Participants' concern about the time aspect of the program warrants some creative thinking about ways in which participants are engaged during the day and evening. As several participants suggested that they had no time to review or organize their extensive notebook and handouts, and others said they needed to visit the library for "background" reading, it may prudent to build into the evening program structured and unstructured time for this kind of work. Structured time might consist of "open lab" time when participants can return to the BISCITS library laboratory, sort through notebooks, browse through background articles and texts that the BISCITS staff puts on "reserve," and meet with BISCITS leaders for informal discussion. An increased amount of time that is completely unscheduled is needed to better allow participants to take care of personal needs, recreation and reflection.

Return to Table of Contents

Labs

It is interesting to note that the laboratory activities were mentioned often as the most effective aspect of the summer program, and were also mentioned as needing improvement. This intensive interest in labs serves to support their centrality to the program: participants value the labs so much that they have strong feelings about making them better. The biggest concerns about the labs are two sides of the same coin. Participants felt that the staff "went overboard with the inquiry lessons" (brief eval, 9/2/96) and that there was a "lack of direction in the lab" (brief eval, 9/2/96).

Participants clearly recognized the inquiry-orientation built into activities, but it appears that the connection between doing inquiry lessons and being prepared to teach inquiry lessons may have been lost. Participants said:

As a teacher, I need to understand what I am doing before I begin the lab. I want to look for instructions or trouble areas that I adjust for my students while I am doing the lab. (brief eval, 9/2/96)

Too much inquiry-based lessons. I was so focused on trying to figure out what to do that I didn't have time to think about adapting the inquiry lesson for my students. (brief eval, 9/2/96)

I definitely have a different perspective than the instructors of what an "inquiry" really is. Not giving out the directions before a lab does not make it an inquiry lab. It only makes it unsafe. I won't take that chance with my students. (brief eval, 9/2/96)

I had difficult with some of the lab instructions because of my prior knowledge. (brief eval, 9/2/96)

A discovery lab does not meet "do not give instructions." Discovery means that a cookbook approach to lab is modified. Students do not know how to do a lab without instructions. A good discovery lab must be processed so students can get the most of it. That's what I'll be doing when you come visit me." (e-mail, 10/96)

Clearly, strong feelings were expressed by several of the participants about the nature of inquiry portrayed during BISCITS. It was obvious from many of the comments that understandings of what is meant by inquiry vary widely across participants, and possibly across instructors. One participant quoted above believed her lack of knowledge about molecular biology created difficulty for her in the lab. As most middle and high school students will likely have even less prior knowledge about these topics, it appears that the inquiry approach used by instructors needs to be revisited.

A more focused and explicit definition of inquiry seems to be needed. Perhaps the explanation of the nature of inquiry in the National Science Education Standards might be useful as the common tool. Participants could receive a copy of the Standards (or photocopies of relevant passages) for their notebooks. The rationale for "inquiry" was never constructed by participants in a convincing way - doubts remain to the viability of inquiry with middle school and high school students. In the future, instructors will want to make explicit their thinking behind pedagogical strategies employed during activities and laboratory investigations so participants do not have to worry about following a procedure they have never done before on equipment they have never seen, as well as wonder about how to make it work in their classrooms.

Other concerns about labs center on the pragmatic. Several participants expressed concern that adaptation for specialized equipment were rarely discussed. "We need to know how to do some of these things with materials we have available to use like fruit flies and frogs," wrote one participant. Another expressed concern that expectations for her products from the lab were never made clear, suggesting a need to check participants' understandings of expectations.

Return to Table of Contents

Summary of Evaluation Findings

The preponderance of data suggest that the 1996-97 BISCITS program was highly successful in:

• Increasing participants' understandings of molecular biology and biotechnology,
• Using BISCITS-inspired laboratory activities and experiments,
• Increasing participants' interest and desire to include biotechnology in their teaching,
• Increasing participants' interest and desire to include ethical issues in their teaching,
• Increasing participants' biotechnology teaching self-efficacy,
• Facilitating peer collaboration and idea sharing as well as connections of university biologists,
• Making participants aware of the resources and materials available for biotechnology teaching through BISCITS, supply houses, and the Internet.

Data also suggest that the program can be improved in three ways:

• Rethinking how time is used during the summer experience, in terms of overall duration and in terms of time use in the daily schedule, and
• Exploring the issue of inquiry more thoroughly with participants and making explicit the ways in which the staff use this understanding of inquiry for the lessons and labs designed for the workshop.
• Explicitly discussing the concern about lack of extended lab periods, exploring ways certain complex laboratory investigations can be implemented in a single period manner, and indicating which aspects of an investigation or activities can be scaled back or eliminated in a time restricted setting.

Nearly all of the participants expressed a need to further information to support their teaching of biotechnology that is beyond the introductory level. Three participants want additional courses, workshops, resource books and videos to be offered by BISCITS. Two others stated that they would like BISCITS to serve as a clearinghouse for biotechnology education in Pennsylvania so that they could easily locate assistance. The BISCITS staff may wish to explore what they want their role to be in this regard and how such apparently needed support mechanisms can be sustained for past participants.

Return to Table of Contents