Technology Annotated Report Excerpts

Excerpt 1 [Middlesex County College]

Interpretations & Conclusions:
Presents findings about the extent to which project objectives were met—findings obtained through mixed methods

Objective 3. Conduct a series of summer institutes at Middlesex County College and partner institutions at which teams of practicing high school teachers and pre-service teachers will learn how to integrate STEM fundamentals and active learning strategies into high school curricula and extracurricular activities.

The Building for Tomorrow Summer Institutes comprise the focal point of the project. To date, four summer institutes have been held—one in Year 1 at Middlesex County College, and three in Year 2 at MCC and partnering institutions St. Louis Community College and San Diego City College. A total of 70 high school educators from 19 institutions have participated in the BFT Institutes. (Sixteen of the 19 schools participated with teams of 3 or more educators.) The MCC Institutes were fully subscribed, while the St. Louis Community College and San Diego City College Institutes fell short of the goal of having 5 full teams participating. A number of teams did not have administrators. Furthermore, not all teams had the requested diversity of disciplinary representation. (At least one team had no mathematics or science teachers.)

Activities at the BFT Institutes included a mix of presentations and hands-on activities. Team building, project management, and strategies for obtaining resources from external sources were topics covered in all institutes. Most of the week-long workshop was spent working in teams to design and build a robot that would traverse a pre-determined track and perform certain functions laid out in the competition rules.

Post-workshop evaluations were overwhelmingly positive. Table 3 summarizes the ratings of workshop sessions. Comments provided by the participants were also very positive. Many participants cited the building of the robot and the team building sessions to be among the most useful. Also highly rated were the sessions where other educators presented an overview of their robotics competition experiences. In some cases, these teachers brought their students along to provide testimonials. Sessions where students presented were very highly rated.

The original objective called for the participation of pre-service teachers in the institute. The intent was to work with the College of New Jersey to attract some of their teacher education students. The project team decided to focus on inservice teachers instead, and no pre-service teachers participated in the institutes.

Objective 4. Obtain commitment from the participating high school administrators and faculty teams to integrate (fully or in part) components of the inservice training into instruction during the Project FIRST competition period.

Acknowledges limitations of data collection

There has been no formal mechanism requiring schools to commit to integrating what they learned into their instruction. Rather, the BFT project team worked informally during the program to discuss what was covered and how participants might be able to use the concepts in their classes. A number of the high schools that participated in the Building for Tomorrow Institute have integrated BFT concepts into their robotics competition team training. However, specific data has not yet been gathered to provide a general summary of how effectively BFT has been woven into the high school curriculum and extracurricular activities.

Excerpt 2 [Middlesex County College]

IV. DISCUSSION AND RECOMMENDATIONS

Building for Tomorrow is an exciting project that has the potential to impact thousands of students from largely urban high schools through focused professional development opportunities for teachers.

The project is on schedule. Four teacher summer institutes have been held to date, and 70 high school educators from 5 states across the country have participated in these workshops. Four additional summer institutes are planned for Year 3. Feedback from the workshops has been overwhelmingly positive.

Recommendations:
Describes unanticipated outcomes

Two unanticipated positive outcomes of Building for Tomorrow are:

1. The addition of a fourth partner, College of DuPage, which will conduct a BFT workshop in the summer of 2003 in the Chicago area.
2. The creation and ongoing sponsorship of New Jersey RoboRocks, a statewide robotics competition that grew out of the recommendations from the first BFT summer institute. NJCATE received a grant of $6000 from the College's Retail Services Corporation to run the program. This competition has also been emulated by the team from San Diego City College, which held their first competition in the fall of 2002.

Identifies project areas in need of attention

The project areas in need of attention are:

1. Extension and enhancement of recruitment efforts. While two of the BFT Institutes were fully subscribed, the other two workshops had fewer participants than anticipated. At least two of the registered institutions dropped out at the last minute.
2. Securing commitments from participating schools regarding teams and follow-up plans. A number of the BFT teams did not have the optimal number or make-up of participants. The project is intentionally designed to accommodate multidisciplinary teams. The participation of an administrator on each team is also critically important. Furthermore, it is important to have the high schools develop action plans for participation in subsequent SMET competitions and/or incorporation of technological concepts and activities into high school curricula.
3. Collection and evaluation of impact and effectiveness data especially regarding students. Data collected to date has been largely anecdotal; more formal, ongoing tracking of schools and students is needed to determine the success of the project.

Makes recommendations

The following recommendations speak to the areas noted above:

Recommendation 1: Recruit early, and consider a waiting list to ensure a full workshop. Partnering institutions might consider communicating with each other to see if schools from one region might be able to attend a workshop in another region. Schools who have participated in BFT can be enlisted to help recruit other schools. The project team is encouraged to utilize the promotional video that they developed.

Acknowledges limitation of data collection

Recommendation 2: Secure more formal commitments up front from high schools. Everything should be done to ensure that the optimum team make-up (4 teachers from different disciplines, plus an administrator) is not compromised. Furthermore, it might be appropriate to have participants develop an action plan for follow-up activities during the workshop. Strategies are encouraged that increase the likelihood that students will benefit from post-BFT Institute activities.

Recommendation 3: Work with the evaluator to collect and evaluate data that assess the impact and effectiveness of Building for Tomorrow on students. The primary objective of BFT is to engage students in technologically exciting, fun activities to draw them into STEM careers. Tracking of students from BFT participating schools through competitions such as RoboRocks is a cost-effective way to assess project impact and effectiveness.

Excerpt 3 [Maricopa Community College]

DISCUSSION AND RECOMMENDATIONS

PROJECT INFRASTRUCTURE

The physical MATEC facility and its electronic capabilities (local area network and Web site server) are fully functional and expertly maintained. This functionality continues to be a strong advantage as MATEC plans for future growth. The MATEC web site has been re-designed and will be functioning in April 2002. MATEC has a very efficient electronic office system and local area network that enhances the productivity of all operations and provides each employee with access to a main database of contact information for individuals, partner schools and industries, and other organizations. The MATEC online Catalog is connected to this database and allows purchase information to be combined with other records in order to assess trends among clients. All MATEC materials and products are produced with high standards for quality and professional appearance. Weekly staff meetings are held for the purpose of sharing information, solving problems, and planning for the future.

MATEC has had many personnel changes during the last project year. New staff is in place who seem to be well-suited and skilled for their tasks. A new energy was obvious as the evaluator met with the staff. Each staff member is excited about taking personal responsibility for their tasks as MATEC moves into the future. MATEC is very fortunate to have strong support with matching and in-kind contributions from their education and industry partners, and from the Maricopa County Community College District. Appropriate plans and infrastructure are in place to support MATEC as they move toward self-sufficiency.

Presents strengths and weaknesses of project documentation

PROJECT PLANNING, MANAGING, AND DOCUMENTATION

MATEC continues to respond quickly to potential opportunities with industry partners. They have been able to take advantage of funded projects from industry partners (e.g., Intel EHS project, the Intel/MATEC Summer Workshop, Intel PC Architecture curriculum) and produce curriculum modules and other materials on a tight schedule. Planning and development processes for new opportunities are more stable, but are still not well-documented. Projects with outside funding will become more and more important as MATEC develops their own funding base. Intel utilizes MATEC for many projects, and is obviously very happy with the work MATEC produces, however, these products tend to be Intel specific and must be re-purposed for generic use. MATEC should continue to search for ways to utilize all of their industry partners and guarantee a generic industry approach and not just an Intel viewpoint.

Recommendations:
Makes recommendations for more documentation

MATEC continues to make progress toward determining and documenting its operating policies and procedures. The areas of Faculty Development, Network and Web Site, and Office Administration are well documented and have replicable processes in place that assist in the day-to-day operations. Two remaining areas require further and more complete documentation: 1) curriculum development, formative evaluation, and implementation efforts; and 2) outreach efforts to partnership schools. MATEC has not yet documented a consistent curriculum development approach but they are using a more consistent development process with very competent staff and documentation should be easily accomplished. The Faculty Development tasks for MATEC are being handled by a new staff who will need time to adapt to their new roles and responsibilities. They will be focusing in new areas and it is anticipated that procedures for Faculty Development Workshops will be revised and new procedures will be put in place for the new program assessment process.

GOAL 1: CURRICULA AND MATERIALS DEVELOPMENT

The Curriculum Development group has refocused personnel assignments for module design. This current approach is working well and permits the completion of modules in a timely way. Consistent project planning and tracking for this large and complex instructional system will become even more important as MATEC completes their initial set of modules and begins a revision process for older modules and continues to work with industry to define new modules.

Makes recommendations for collecting observational data

MATEC curriculum materials are well aligned with industry needs and have been carefully designed in a competency-based format. One of MATEC’s original goals was to produce instructionally sound materials that included all aspects of good instruction. MATEC needs to find a way to assess how faculty are using or customizing the materials and how many students are being impacted. MATEC curriculum staff suggested on-site observation by an evaluator or other non-interested party as a viable approach to gathering important information about the curriculum and its use with learners. Adopting this approach would align an evaluation of MATEC’s curriculum modules more easily to the manner in which they are being used. By learning how the faculty are successful with the materials MATEC may be able to concentrate their efforts more effectively in the development of reference materials, animations, simulations, examples, and hands-on exercises while leaving the assignment and assessment of learning objectives to each individual faculty that uses and/or adapts the materials.

Interpretations & Conclusions:
Present findings

Faculty report that they take the curriculum materials and adapt them to their own needs and learning objectives, may or may not share them with students, or just use the materials as supplemental information. The revised approach to module delivery (MEPSS) should enhance a faculty member’s abilities to utilize the curriculum quickly and easily. Adopter’s Workshops can give faculty more opportunities to explore the full curriculum with staff support. This activity might be the initial step for an evaluation process of the actual materials as well by training faculty to report the type of data necessary for an evaluation process.

There is no clear owner of the curriculum evaluation process. The curriculum development area, curriculum adopter workshop providers, and the faculty development group all have access to data about curriculum use, but these groups have not taken a combined look at the problem. The evaluator and MATEC have been trying to find indicators or measures that would show the curriculum materials had an impact on students, a continually elusive goal. The evaluator is suggesting that a more system-wide view of the problem of curriculum evaluation might begin discover a workable approach to providing solid information about the validity and effectiveness of MATEC’s curriculum efforts. A focus on how faculty utilizes the materials should provide very rich data about the materials and their utility in learning settings. Student impact data may be easier to identify once the faculty impact is understood.

GOAL 2: FACULTY DEVELOPMENT AND SUPPORT

The Faculty Development arm of MATEC has been very successful. The new personnel will be hosting focus groups of industry and faculty representatives in six regions to gather current information about faculty development needs. Faculty Development workshops have been very successful and well received by faculty attending them. A mechanism should be devised for collecting evaluation data about workshop attendees that can assess their learning, their intent to utilize the materials in their teaching, and provide information about possible student impact.

The MATEC Web site seems to be well used, but there are no comparative figures to show if the number of visitors to the Web site is average for an organization of this type. MATEC continues to very successfully produce the ATESM Annual Conference that connects community college faculty to industry trends and issues. Attendees have asked that industry be represented more at this conference.

GOAL 3: WORKFORCE DEVELOPMENT

This goal continues to be difficult to define and measure. MATEC provides quality workforce development material for many venues but their effectiveness is difficult to measure. It will be necessary to find viable measures that can be used across schools and industry in order to understand if MATEC is having an impact. Since each school tracks students differently and industry may not track school attendance or graduation, an easy and common measure may not be available to answer our evaluation questions. MATEC has begun a data collection effort for the enrollment and graduation numbers for SMT programs.

MATEC’s work with new partner schools and potential new SMT programs could be a significant revenue source. MATEC should continue to define how they utilize the School Assessment Reports for potential new SMT programs within MATEC. These assessments have narrative and quantitative data that are useful to the Faculty Development and Curriculum Development areas as they plan their materials and outreach activities, but this information is not currently being utilized.

MATEC has plans to create a Technician Certification program that would be an entrée to employment in the semiconductor industry. Many community colleges currently offer successful certification programs for their more technical programs. These certifications are valuable to students and employers alike. The MATEC Technician certification would be based on the MATEC curriculum and industry validation. This approach would also give MATEC valuable information about the soundness of the curriculum materials and their impact on students.

RECOMMENDATIONS

In addition to the comments noting areas for improvement throughout this document the evaluator would like to make the following recommendations to MATEC.

1. Work with industry to define and validate a Semiconductor Manufacturing Technician Certification process and test for Semiconductor Manufacturing Technicians. Incorporate evaluation mechanisms for curriculum impact and student employment within the certification process.
   
   
2. Develop a MATEC-wide approach to the curriculum evaluation process and evaluate new curriculum modules to ensure their validity as well as how faculty utilizes the materials. Determine what information will be useful to MATEC for development and delivery of modules, faculty needs, student needs, and the impact of the materials on student learning and employment (if possible).
   
   
3. Develop and document a methodology for working with potential SMT programs and building the number of Partner School Agreements.
   
   
4. Strengthen the evaluation process for the Workforce Development goal by continuing to define indicators, locating and gathering data, summarizing, reporting and utilizing this data.