: Instruments : User Scenarios

home reports instruments plans

User Scenarios for Instruments

Described below is a scenario for using the evaluation instrument resources. For more information on these and other resources, see the overview for the user scenarios, as well as the frequently asked questions about instruments.

Activity Description
Goal: Preparing data collection instruments for the evaluation of a Teacher Education project Professor Robert R. directs a three-year EHR project designed to improve the instructional approaches of secondary and community college chemistry teachers. Some of the data are available from the school and college records, such as years of teaching experience, performance reviews, awards, student reviews of instructors, and student grades and course credits. Data about the effectiveness of the professional development activities and new course materials will need to be collected by using instruments developed for the project. Professor R. meets with Dr. Paula Q., the external evaluator for the project, to confer about the data that they need to collect for the project.
Review framework linking data collection instruments to evaluation questions They refer to the data collection framework Dr. Q. included in the evaluation plan of the EHR proposal. The framework proposes types of data collection for each of the evaluation questions. They plan to use two types of instruments: a focus group interview for community college professors and a student survey. They turn to the OERL instrument resources to search for suitable instruments.

Go to the OERL Web site

Together, they go to the OERL web site to access the evaluation instruments that have been used in other Teacher Education projects. They click on the Instruments tab.

Review criteria for quality of data collection

Since Professor R. is less familiar with instrument selection and development, they review the criteria for instruments.
Question: What kinds of instruments are used to evaluate similar EHR projects? After reviewing the criteria, Professor R. and Dr. Q. click on the Teacher Education tab on the left hand side of the page, and then on the link to Teacher Education Instruments.

Examine the types of sample instruments for Teacher Education projects

They look at the types of instruments that the collection holds. They find teacher/faculty instruments, student instruments, case/study protocol instruments, and observation instruments.

Question: What chemistry instruments are in OERL?

Search for chemistry instruments in OERL

Because their project focuses on chemistry, they are particularly interested in instruments used in evaluating chemistry projects. They decide to search the Web site for chemistry instruments, so they click on the Search tab in the upper right hand corner of the page. The search page lists several types of searches available. They click on the Instruments search. A form appears allowing them to specify the features of the instruments they are searching for. They choose instruments from projects in the chemistry content area, choosing "everything" for all the other features available. Then they click the Search button. A listing of search results appears organized by Project Type. Each result describes the purpose, length, etc. of the instrument and displays information about the project from which it comes as well as a link to view the instrument. They spend some time looking through the instruments in turn and clicking the "Back" button on the browser to get back to the search results.

After viewing one instrument in particular, they get diverted and leave the page with the instrument on it. This time, instead of clicking "Back" they click on the Search tab again. Back at the search page, this time they click on "most recent search" and the search results they were viewing before reappear, allowing them to continue perusing the instruments in the search results.

Examine the OERL table of instruments, which overviews the content and format of each type of instrument

Look for instruments that can be used or adapted

After reviewing all the relevant chemistry instruments, Professor R. and Dr. Q. decide to spend more time examining the Teacher/Faculty instruments within the Teacher Education area. Because community college professors will be involved in their project, it is imperative that they locate an instrument that has been designed to assess standards-based science reform in a similar group. They click on the links below the heading for Teacher/Faculty Instruments. They compare the instruments used in the evaluations of Teacher Education projects with the types of data specified in their data collection framework.
Use particular questions and formats from OERL instruments as components of new instruments Within the Teacher/Faculty Interviews, they find an instrument that has been used in focus groups to capture instructors' changes in beliefs about standards-based reform. This instrument matches their needs as specified in the data collection framework, so they make a copy of the focus group interview instrument.
Search for instruments that can be used or adapted The pair now read through the OERL overview matrix for each type of instrument to compare the topics covered with the topics relevant to the evaluation questions for their project. They had planned to use student surveys, but not classroom observations or student assessments. After examining the types of instruments available, they realize that observations and student assessments would better address their evaluation questions. Thus they add the two types of instruments to their data collection framework and make copies of the items and formats used in the observation and assessment instruments that meet their needs. These items and formats will be incorporated into newly developed instruments.
Plan for review and pilot testing of instruments Dr. Q., the external evaluator, intends to pilot all the instruments to be used. She will ask community college professors and students to review the instruments and pilot test them. A brief description of the pilot testing and the technical quality data for these instruments will be reported in the evaluation.
Bookmark the OERL Web site Professor R. and Dr. Q. bookmark the OERL Web site on their browsers, anticipating that they may browse more of the resources once the full set of instruments is drafted.