: Plans : User Scenarios

home reports instruments plans

User Scenarios for Plans

Described below is a scenario for using the evaluation plan resources. For more information on these and other resources, see the overview for the user scenarios, as well as the frequently asked questions about plans.

Activity Description
Goal: Planning the evaluation of a systemic mathematics course reform project Professor Katherine S. plans to submit a proposal to EHR for redesign of Drake University's freshman mathematics courses. She meets with Dr. Craig J., the external evaluator, to confer on the details of the evaluation plan. Dr. J. has reviewed the description of the project's goals and the strategies for revamping the mathematics course series and integrating the courses with the community college curriculum. He has drafted some evaluation questions he has derived from the project goals.
Go to the OERL Web site Together, they go to the OERL website to browse through the evaluation plan resources that have been used in other EHR curriculum reform projects. First, they examine the introduction to the library on the OERL home page to identify which resources might be most useful.
Review criteria for sound evaluation plans They click on the Plans tab to locate the planning resources. They review the criteria for sound evaluation plans that overview the major components of a sound EHR evaluation plan.

Goal: Determine evaluation purposes

Question: How do evaluation plans for similar EHR projects specify their purposes and questions?

Scroll through Evalation Overview plan excerpts for Curriculum Development projects

Because Professor S. and Dr. J.'s project focuses on the reform of mathematics curriculum, they click on the Curriculum Development tab on the left hand side of the page. To get an idea how plans describe and organize the purposes of the evaluation, they click on Evaluation Overview under Annotated Excerpts and scroll through the brief plan excerpts, looking for evaluation purposes.

Dr. J. notes that one of the plans specifies three types of project outcomes to evaluate: effects on students, quality of materials, and effectiveness of instructional training. He and Professor S. agree that these three areas should also be used to organize their proposed evaluation. In another excerpt, Professor S. likes the questions specifying five effects of the courses on students: continuation in mathematics, attitudes, basic skills, problem solving, and the ability to reason and discuss a concept in depth. They copy these questions.

Question: What evaluation designs have been used in other Curriculum Development projects? A key issue for the evaluation is the design that will be used to document project effectiveness. Can the project provide evidence of value-added in relation to previous approaches (pre/post) or in contrast to other ongoing approaches (comparison groups)? Professor S. and Dr. J. click on the plan excerpts for Design. They discuss the advantages and feasibility of alternative designs, while examining the designs used in EHR curriculum development project evaluations in OERL. They keep in mind their project goals, institutional contexts, and proposed budget. They find that most of the designs in the plan excerpts have contrasted the revamped, experimental courses with other parallel, but differently designed sections of the courses. They discuss the feasibility of this approach for their evaluation.
Question:What kind of sampling plan should be used? There are a large number of students enrolled in the reformed and unreformed mathematics courses that will be compared in Professor S. and Dr. J.'s evaluation. They turn to OERL for information on sampling. After reading several excerpts describing the sampling plans used in different evaluations, they elect to use random sampling of students in the reformed and unreformed courses. The use of a random sample will be cost-effective, since fewer students will be tested and the generalizability of the evaluation results will be increased.
Refer to plan excerpts to find a table that links data collection procedures to outcomes Professor S. and Dr. J. agree that a combination of qualitative and quantitative methods will be used to evaluate their project. To get some different ideas about how to design the evaluation, they leave the Curriculum Development section of OERL and examine plans from another NSF program area, Teacher Education. To locate the Teacher Education plans, they click on the Teacher Education tab on the left hand side of the page. They then click on the link to Teacher Education plans. Finally, they click on Design under Annotated Excerpts. In the design excerpts they find a table that efficiently identifies the data collection procedures to be used for addressing evaluation outcomes. They copy this table to adapt for their plan.
Use a search to look at resources

They decide to look specifically at plan excerpts for mathematics, so they click on the Search tab in the upper right hand corner of the page. The search page lists several types of searches available. They click on the Plans search. A form appears allowing them to specify the features of the plans they are searching for. They click on Plan Excerpts (as opposed to Complete Plans) and projects in the mathematics content area, choosing "everything" for all the other features available. Then they click the Search button. A listing of search results appears organized by Project Type. Each result describes the content of the plan excerpt and displays information about the project from which it comes as well as a link to view the excerpt. They spend some time reading through the excerpts in turn and clicking the "Back" button on the browser to get back to the search results.

After viewing one excerpt in particular, they get diverted and leave the page with the excerpt on it. This time, instead of clicking "Back" they click on the Search tab again. Back at the search page, this time they click on "most recent search" and the search results they were viewing before reappear, allowing them to continue perusing the plan excerpts in the search results.

Scroll through sample instruments To develop their data collection procedures and timeline, Professor S. and Dr. J. decide to look through the sample instruments included in OERL for evaluating curriculum development projects. To examine these instruments, they click on the Instruments tab at the top of the page. After getting to the section on Instruments, they click on the link to Curriculum Development instruments. They tentatively identify some instruments that have useful sets of questions.
Skim complete plans Professor S. and Dr. J. look over the notes and materials they have gathered from the OERL plan resources and decide to review some complete plans to see how all the components have been integrated.
Bookmark the OERL Web site They bookmark the OERL Web site on their browsers, anticipating that they may browse more of the resources once the evaluation plan is drafted.