Curriculum Development Stand-Alone Report 1 (Progress)

Outside Evaluation Report of Drexel University's Enhanced Bioscience Education Program
1996-1997 (EBE)

Table of Contents:

1. Executive Summary
   • Executive Summary
2. Background
   • Project Description: Project Features
3. The Goals of EBE
   • Project Description: Project Features
4. Evaluation Approach
   • Evaluation Overview: Evaluation Purposes, Evaluation Questions
5. Evaluation Methods
   • Design: Methodological Approach, Information Sources & Sampling
6. Observations
   • Analysis Process: Qualitative Analysis
7. Pre-Survey Data
   • Analysis Process: Quantitative Analysis, Qualitative Analysis
8. Pre-Survey Results Summary
   • Analysis Process: Quantitative Analysis, Qualitative Analysis
9. Post-Survey Data
   • Analysis Process: Quantitative Analysis, Qualitative Analysis
10. Post-Survey Results Summary
    • Analysis Process: Quantitative Analysis, Qualitative Analysis
11. Comparison of Pre/Post Surveys
    • Analysis Process: Quantitative Analysis, Qualitative Analysis
12. Focus Group
    • Analysis Process: Qualitative Analysis
13. Portfolio Experience Reflective
    • Analysis Process: Qualitative Analysis
14. Results Summary
    • Results & Recommendations: Interpretations & Conclusions
15. Conclusions & Recommendations
    • Results & Recommendations: Interpretations & Conclusions, Recommendations

Outside Evaluation Report of Drexel University's Enhanced Bioscience Education Program
1996-1997 (EBE)

Return to Table of Contents

Executive Summary

An outside evaluation of the Enhanced Bioscience Education Program at Drexel University was conducted to determine to what degree program goals had been met, what the participants’ experiences were, what the dynamics and direction of changes were, what the faculty and students learning outcomes were, what skills had the students developed, what constraints and problems prevented the program goals from being realized, what were the critical events, what were the significant program highlights, and improvements die participants suggest. The methods included pre- and post-surveys of student attitudes and ratings of their skills, formal and informal observations at presentations, poster sessions and meetings, the analysis of student journal entries on their portfolio development, careers and a look back at EBE, and a focus-group interview.

The evaluation results from multiple sources confirm that the EBE program goals have been met at remarkably high levels. This community of investigative learners are practicing science in complex ways – conducting investigations, working in teams, solving problems, and present, and in some cases publishing, their own results and conclusions. The EBE program has met to a great degree or exceeded expectations (82%) and had a positive impact on their feelings of accomplishment and self-confidence. Moreover, students demonstrated growing abilities in using the language of science; active learning; creative, critical and reflective thinking’ growing awareness of larger issues, such as ethics and environmental concerns; and more insight into career paths and professional practice.

Students’ skills have improved at significant levels in communication, computers and teamwork. Almost all the students (91%) rated the EBE faculty as above average and outstanding. In the students’ view, and important contribution to their success was the EBE environment that provided support, faculty interactions, networks, coaching, resources and extended labs.

Suggestions for improving EVE include better alignment between lectures and labs, more student choice in lab projects, improved communications, and more labs, computers and field trips. Faculty and TA’s suggest increased planning and faculty development, more resources, and adding an educational study group.

Recommendations are that EBE develop a strong internal infrastructure as EBE expands and changes. A performance-based assessment model is recommended for future program dissemination and adoption.

Return to Table of Contents

Background

The Enhanced Bioscience Education (EBE) program is designed to increase the effectiveness of undergraduate education in the biological sciences, providing a curriculum and methodology that incorporates a problem-solving, integrative approach. The curriculum includes biological investigation, hands-on lab activities, group work, and the study of Bioscience, Physical Sciences and Mathematics. Integrated with the biosciences is the Humanities’ course in writing and reading that includes technical writing.

The emphasis on active learning through investigation requires assessments beyond the quiz and exam. Students keep a course journal throughout the program and develop a portfolio. The University Seminar engages students in reflective analysis on their learning, including their journals writing. In this fourth year, the program has been expanded to include students from Food and Nutrition Sciences and Teacher Preparation.

Return to Table of Contents

The Goals of EBE

1. Improve the effectiveness of science education by teaching science the way it is practiced, emphasizing problem solving, investigative, laboratory approaches.
2. Improve science education by incorporating the use of technology and computers in all phases of study.
3. Develop the students' ability to work together for problem-solving and research projects. Self-assessment, journaling and portfolio analysis are included.
4. Prepare students more effectively for their advanced courses, research projects and professional careers.
5. Utilize an integrative curriculum to alleviate problems of content overload and curricular fragmentation while helping students develop the ability to integrate knowledge across the disciplines.
6. Attract and retain more students in science, especially those from underrepresented groups.
7. Improve science education of pre-teachers using these same, hands-on, integrative laboratory approaches.
8. Provide a support network for students for their academic courses.
9. Evaluate the program using various qualitative and quantitative approaches.
10. Disseminate the results of the program nationally for possible adoption at other institutions.

Return to Table of Contents

Evaluation Approach

The evaluation frame is holistic; it considers the meaning and perspectives of the participants in the study, the relationships regarding the structure, occurrence, and distribution of events, and what constrains or prevents program goals from being realized. Second, as program participants describe what the program experience is like, they discover what is significant and meaningful for them. Third, the multiple measures result in a detailed model for the participants and future beneficiaries to understand the program’s goals, the change processes, and what works and doesn’t work–the lessons learned.

These evaluation questions were used to guide the evaluation:

1. What are the student and faculty learning outcomes?
2. How well did the students develop their skills in teamwork, communications, and using computers? Which instructional formats and activities did the students prefer?
3. What constraints and problems did participants experience?
4. What re the telling evens and relationships within the program?
5. What do the participants view as meaningful program highlights and characteristics?
6. What suggestions do the EBE participants have for improving the EBE program?

Return to Table of Contents

Evaluation Methods

The evaluation plan includes both quantitative and qualitative approaches. Quantitative methods can yield data that is generalizable and qualitative methods can provide depth and insight into the specificity of change processes in a particular context. For this evaluation the following methods were used:

1. Conduct a faculty workshop on newer assessment methods
2. Observations of program key events and student project presentations
3. Pre/post surveys of EBE freshman students
4. A focus group discussion
5. Analysis of three focused journal entries; career plans and the future, the EBE program, and the portfolio experience
6. A senior survey
7. Artifact analysis

A matrix that shows the relationship of the evaluation questions, evaluation methods, data collection and timeline is show on the next page.

Return to Table of Contents

Observations

Student Presentations, December, 1996

Grouped in teams in Stratton 309, students waited their turns to present the results and conclusions of two months of scientific investigations. They used presentation software to show their hypothesis, the methods they used, the results they got and their conclusions. Over the course of the hour and a half, students showed their newly gained expertise in several ways:

Using the language of scientists

Students described their investigations in scientific language, using the terms hypothesis, methodology, procedures, findings and results. One team even explained the index theory to the audience; the focus was on evidence and accuracy. In one presentation, a TA asked the students to define the term best. "What does it mean?" he asked. "Try to get out of the habit of using best which means different things to different people."

Defending their work

The questions from the instructors, TA’s and other students required the teams to defend, review, analyze, re-think, and see the significance of their experiments and results. Questions, such as the following, sharpened the discussion:

• what are the different properties being tested?
• Which temperature caused the most growth?
• Where does bacteria exist?
• What does that suggest to you?
• Did you notice anything about the color?
• What types of organisms form spores?
• Do you think the spore test was the best test given where you wanted to be?

In each presentation, teams had to take the results and conclusions one step further to show they would do things differently and make connections to real world problems.

The discussion had many teachable moments. Dr. Duwel told one team that "if you make an observation that a micrococcus couldn’t grow at 42 degrees, it might have been a juncture to determine something more."

Another time, Bob Johnston (a TA) asked a team about a graph they used to represent the data. He suggested that they might use a more appropriate representation for showing measurement over time – a line graph – rather than a column graph which just shows the comparison of data.

High expectations and continuous learning

The teams showed various levels of expertise in their science experiments and presentations, but students were also learning during the presentations. As they watched each other in this community of developing scientists, they often explained how they would improve their methods and approach if they were to re-d the experiment. One could sense some peer pressure as each one presented, but it wasn’t until the last group had finished that it was clear that one team stood out above the others. It had set the highest standard for all the groups.

At the close of the presentations, I spoke with Dr. Duwel and the TA’s about their impressions of the students’ work. One said that "students do well on specific procedures, but need to go beyond that."

Another added, ‘Students are given criteria for presentations and it is suggested that they go beyond the single question for relationships, connections to the real world, and to broader implications."

(picture)

Dr. Laura Duwel is viewing one of the posters at the EBE poster sessions exhibited in Stratton Hall foyer during May, 1997.

Poster Sessions

The spring EBE poster sessions were conducted in a large area on the first floor of Stratton Hall at the end of the spring quarter, 1997. It was titled "EBE Catalase Convention" and a booklet of all the abstracts was available on several tables and benches. (A page from the booklet can be viewed in the Appendix.) Students and faculty closely examined the posters and each team presented its results and conclusions to a faculty member. As I observed several of these presentations, I was struck by the students’ active involvement in their own work and in their observations and evaluations of the other posters displayed there. Students in pairs or trios were huddled at posters as they evaluated each one based on the evaluation criteria.

I spoke to several students that day about their experiences. One student, Jason Boethe, felt that overall the posters were pretty good. "Some are lower than others," he commented, "but these are much better than last semester. We are more used to what we have to do. In high school, we were given a lot more research to do, but now it’s more complex and in-depth. It gets critiqued a lot more."

Adrienne Richards said that a lot of the ideas were well researched and creative. "It has helped me learn a lot from looking at different posters. Also it’s held here – down the main hall, not in a place where no one sees them."

I watched as Dr. Dickstein and Dr. Duwel conducted several poster sessions, seeing how involved the students were with presenting their data. One experiment, "The Effect of Ripening on Catalase Activity in Bananas," produced contradictory results. Tom McGee and Mike Castagna found that as a banana ripens, the catalase activity increases, a result that was just the opposite of what they had predicted in their original hypothesis.

Each session lasted a half-hour or more as the faculty took their time with each aspect of the presentations. Once could see the teams warm up as they got to the results and explanations about their methods.

Dr. Dickstenin was pleased with the results. "The students had to think critically," she said, "although there were big differences in their ability to cope and work in a group.

Return to Table of Contents

Pre-Survey Data

N = 105 (Not all students answered all questions.)

In the first week of Classes, EBE students were surveyed to investigate why they enrolled at Drexel and EBE and to ask how they would rate their own skill levels for teamwork, computers, communications and several other factors.

1. Briefly describe why you decided to enroll at Drexel University and in the EBE program.

The largest group of students (30) listed Co-op and Drexel’s reputation (12) as reasons for enrolling at Drexel. Reasons cited for enrolling in EBE included strong interest in Biology (13), requirements for medical/graduate school (7), and interest in hands-on learning, small classes, and recommendations from Teacher Preparation. Individual students listed the following:

• Innovative style of teacher prep similar to EBE
• Interested in environmental biology
• Impressed with the lab
• Sounded fascinating
• To keep the world’s environment safe
• Transferred out of engineering for a biomed career
• want to help people
• EBE will be challenging

2. What are your expectations about how this program will develop your skills and knowledge required for your educational and career objectives?

Students expected to increase their knowledge and understanding (25), increase their options for a job and career (13), and develop their lab, communication, and group learning skills (13). Specific expectations included:

• Solid background in bioscience
• Better research skills
• Writing skills
• Hands-on experience
• Computer skills
• Problem-solving
• Developing ideas
• To be comfortable with teaching the subject
• Help me be more focused
• Get a chance to see if I can do science
• Make me an independent learner
• Help me observe and be more curious
• Presentation skills

3. A majority of the students (66%) expected the coursework to be rather difficult, while 22% felt it would be of average difficulty. Only 11% listed it as very difficult.

4. Most students (71%) rated their computer skills at a fair level, and 20% rated this skill at a poor level. Only 9% rated this skill at the mastery level.

5. A few students said they were using the following programs at this point:

• Netscape Navigator
• Word
• ClarisWorks
• PowerPoint
• Maple

6. Just over half (52%) rated their communication skills (writing technical reports, presentation skills, speaking, etc.) at a proficient level, while 42% rated it as fair.

7. Half of the students listed speaking skills as one that they needed to develop. One third listed presentation and technical writing as well.

8. A large majority (85%) has not had experience developing a portfolio; 15% have had this experience.

9. A majority of students (65%) has had experience in journal writing. This experience was gained in English classes, a Chemistry lab, Humanities, and at other school.

10. 62% of students have had a great deal of experience working on a team, and 37% have had some experience.

Students’ experiences included:

• Chem lab
• Sports
• Community clubs
• Work
• Committee

11. Most students (71%) rated their ability to work on a team at the proficient level. Only 16% considered themselves expert and 13% said it was fair.

12. Most of the students (72%) tend to study and do homework in informal study groups occasionally.

13. Students said the greatest benefits of working in groups or on a team are:

• You feel confident you have accomplished something
• The course is less intimidating
• You learn what other people have to offer
• Communication skills
• Less work for each student
• Helping others as others help you
• Access to other people’s genius
• Different points of view
• Things get done faster and better
• Fun
• Working as a unit
• Get to meet people

14. 59% of the students said they had one or two courses that require group work outside of the EBE program; 32% had none. These occurred in Chemistry, Nutrition, Humanities, and Food Science.

15. Just over half of the students (53%) rated their ability to make an informed decision about their future career/education as good, while 28% rated it as fair. Only 13% rated it as excellent.

Return to Table of Contents

Pre-Survey Results Summary

1. The major reasons that students listed for enrolling at Drexel was the Co-op program and Drexel’s reputation. They identified their own interest in Biology, requirements for medical or graduate school, the attraction of hands-on learning in small classes, and recommendations from Teacher Preparation as the main reasons for enrolling in EBE.

2. Students in this first week of the term expect to:

• develop their knowledge and understanding of bioscience;
• increase their options for jobs and careers; and
• develop skills in communication and teamwork.

3. A majority of students (66%) expected the coursework to be rather difficult,. Most of the students (72%) tend to study and do homework in informal study groups occasionally.

4. Most students (71%) rated their computer skills at a fair level and use the programs ClarisWorks, Netscape, Word, Powerpoint and Maple.

5. Just over half (52%) rated their communications skills (writing technical reports, presentation skills, speaking, etc.) at a proficient level and would most like to develop their speaking skill.

6. A large majority (85%) have not had experience developing a portfolio, although a majority of students (65%) have had experience in journal writing that was gained in high school English classes, Chemistry labs, Humanities, and at other schools.

7. A majority of the students (62%) have had a great deal of experience working on a team in experiences in Chemistry lab, sports, community clubs, committees and work. Most students (71%) rated their ability to work on a team at the proficient level and said they had one or two courses that require group work outside of the EBE program. They listed many benefits of working on a team, including getting different points of view, efficiency, getting to meet people, and getting support.

8. Just over half of the students (53%) rated their ability to make an informed decision about their future career/education as good.

Return to Table of Contents

Post-Survey Data

This survey was conducted in May, 1997. N = 45

1. Students expectations were met to a great degree (58%) and above expectations (24%).

The reasons they gave included the following:

• The tests are short essays where you have to know your stuff.
• I never thought we’d accomplish so much!
• You did things that were quite interesting yet difficult.
• The interactions with professors and the hands-on problems.
• The bio lab that we started right away.
• The curriculum is well integrated.

2. Almost half (48%) of the students rated their coursework as rather difficult. 34% rated it as of average difficulty, and 16% said it was very difficult. The combined ratings of higher difficulty was 64%.

3. Twenty percent of the students liked all the learning activities. The rank order from highest to lowest are as follows:

4. About half of the students rated their computer skills at the mastery level and nearly half (44%) of the students rated them as fair.

5. Many more students (64%) rated their communications skills (writing technical reports, presentation skills, speaking) at the proficient level at the end of the year. (See next section).

6. The communication skills students have developed the most include presentation, technical writing, and speaking.

7. 64% of the students rated their ability to work on a team at a proficient level and 29% rated it at an expert level. (See comparison in the next section).

8. The benefits of working in groups or on a team are:

• catching mistakes
• sharing work
• different expertise
• different and new ideas
• get to meet others
• getting help
• ability to do more difficult tasks
• communication
• efficiency and working together

9. The problems of working in groups or on a team are:

• having to depend on someone who doesn’t do their work
• uneven work ethics and abilities
• compromising
• finding time as a group to meet
• when ideas conflict and decisions can’t be made
• weak links
• frustration
• failure to understand input
• contention, disorganization
• someone always tries to be in charge
• sometimes a lack of communication
• sometimes people drop out and take a lot of the work with them
• personal conflicts
• not showing up

10. A clear majority of students by the end of the year (78%) said they had one or two courses outside of EBE that require group work. 13% had more than two.

11. More students (53%) rated their ability to make an informed decision about their future career/education at a good level, and 20% rated this item as fair.

12. Almost all the students rated the EBE faculty's teaching and support as outstanding and above average.

13. How would you improve the EBE program?

• Better organization and planning of course deadlines
• More supplies and lab equipment
• Calculus course needs improvement
• More TA’s
• More lab time - longer periods
• New computers
• Give more partial credit
• Encourage more questions & group discussions
• Make EBE I & II as involved in lab as Dr. Dickstein’s approach
• More field trips
• Teams (at least 3 or 4)
• Better communication
• Be more specific on procedures and evaluations needed
• Make it more open: student should have more control on labs
• Just have instructors that love teaching
• Matlab/Simulink course needs a lot of work
• We shouldn’t have to pay for Matlab/simulink
• More projects on a wider variety of topics
• Have professors and TA’s explain how the lectures and labs relate
• better computers
• Rotate lab partners

14. When you look back over the past year, what stands out as the most significant activity that has influenced your bioscience learning and educational goals?

• Planning my own experiments
• Labs have definitely influenced my bioscience learning
• Poster sessions and slide shows helped us communicate to an audience in a scientific manner and it helped us think faster when faced with a question
• Tinicum and Haverford field trips
• Presentations
• Interaction with faculty
• Portfolios
• Dissections and the study of human anatomy
• I have learned to be a leader
• Hands-on work
• Review sessions
• Using computers
• EBE coaches (Bob and Andy)

Return to Table of Contents

Post-Survey Results

1. Most students expectations were met to a great degree (58%) and above expectations (24%) by the year’s end. Students identified several reasons for their satisfaction, including meaningful assessments, hands-on labs, interactions with the professors, and the curriculum integration.

2. Almost half (48%) of the students rated their coursework as rather difficult. The combined ratings of higher difficulty was 64%.

3. One fifth of the students (20%) liked all the learning activities. The highest ratings were given to lab activities, experiments, teams work, computers, presentations, and portfolios.

4. The number of students who rated their computer skills at the mastery level rose to 49% from 9%, a significant change.

5. Many more students (64%) rated their communications skills (writing technical reports, presentation skills, speaking) at the proficient level at the end of the year. The communication skills students have developed the most included presentation, technical writing, and speaking.

6. Sixty-four percent of the students rated their ability to work on a team at a proficient level and 29% rated it at an expert level. The benefits of working on a team include meeting others, doing more difficult tasks together, and sharing different expertise. The problems of working on a team were uneven work ethics and abilities, finding time to meet, lack of communication, and personal conflicts. A clear majority (78%) said they had one or two courses outside of EBE that required group work.

7. Fifty-three percent (53%) of the students rated their ability to make an informed decision about their future career/education at a good level, the same level as in the pre-survey.

8. Almost all the students rated the EBE faculty's teaching and support as outstanding and above average.

9. Suggested improvements to EBE included better organization of course due dates and the relationship of lecture to lab; giving students more options for lab projects, improve communication; and arrange more lab time, computers, and field trips.

10. Significant activities that have influenced students’ learning and goals were lab experiences, poster sessions, presentations, portfolios, interactions with faculty, TA coaching, field trips and leadership development.

Return to Table of Contents

Comparison of Pre/Post Surveys

Several shifts in the pre/post surveys in computer skills, communication skills and team skills are significant.

Students shifted from 9% to 49% on the mastery level by the end of the year. The expert level showed up for the first time and the poor level went from 20% to 7%.

The proficient and expert levels increased, while the fair level decreased.

The percentage of students saw themselves as expert (from 16% to 29%) almost doubled, while fewer students viewed themselves as proficient or fair.

Senior Exit Surveys

Seniors were asked to take a survey about their educational experiences in Bioscience. The number who responded (9 EBE and 4 non-EBE) was too small a sample for data analysis and only one student responded to the longer essay question. The survey was given too near the end of classes.

There was, however, one meaningful difference between the EBE and non-EBE group. EBE students checked off more types of learning activities that they found effective (from three to six) compared to the non-EBE students who checked only two: experiments and lectures. This pattern might indicate that the non-EBE students’ learning experiences were more traditional compared to the EBE students. Future exit interviews for seniors will be scheduled earlier in the term for wider participation.

Return to Table of Contents

Next Page