: Reports : User Scenarios

home reports instruments plans

User Scenarios for Reports

Described below is a scenario for using the evaluation report resources. For more information on these and other resources, see the overview for the user scenarios, as well as the frequently asked questions about reports.

Activity Description
Goal: Evaluation reporting for a Teacher Education project Professor Alice J. is an external evaluator for a Teacher Education project funded by EHR. The project, which provided professional development opportunities to approximately 200 undergraduate physics faculty, has completed its final year. Consisting of a series of workshops held during the academic year and summer institutes, the project provided physics faculty with new pedagogical and technical skills to teach introductory physics courses online. Professor J. is preparing to write the evaluation report and is turning to OERL to clarify several concerns. Her first concern is what types of information she should present in describing the project.
Go to the OERL Web site To gather information on project descriptions, she goes to the OERL web site.
Review criteria for sound evaluation reports She clicks on the Reports tab at the top of the page. She begins by examining the criteria for sound evaluation reports that focus on project description/documentation. After she reviews the criteria, she clicks on the report excerpts for Teacher Education programs.
Scroll through report excerpts for Teacher Education Project Description Professor J. scrolls through all the report excerpts listed in the Program Description component and notes that many include a list of project goals or objectives and short descriptions of project components or activities. Only a few contain context analyses that document demographic characteristics of participating teachers and the educational context in which they work. Because the project she is evaluating has to do with increasing the technological skills of faculty, she believes it is important to document the educational and technological resources that were available to participants.
Question: How do reports include program descriptions? Thus, Professor J. decides that the Program Description section of her evaluation report will contain a table describing characteristics of participating teachers and documentation of the technological resources available to them.
Question: How do reports specify limitations in data collected?

Professor J.’s second concern has to do with the attrition of participating teachers during the second and third years of the evaluation. Although Professor J. understands that the attrition must be reported, she is interested in the level of detail that other evaluators report when describing limitations in the data collected. Professor J. clicks on the Analysis Process excerpts to better understand how such information is presented.

Among the Analysis Process excerpts she finds several relevant examples. The first reports the limitations of data collected with a survey instrument in which participants provided only very brief responses and there was no follow-up to further elaborate their responses. Another excerpt reports the limitation of not having a random sample of project participants, which reduced the generalizability of the findings. Professor J. notes that in both these excerpts there is a brief description of the limitation, which is followed by implications for the analysis and interpretation of the evaluation data. Using these two excerpts as models, she elects to document the attrition in each year, provide plausible hypotheses for its occurrence, and provide a cautionary note concerning the effect of attrition on the generalizability of the evaluation findings.

Question: What kinds of graphic displays are used to report survey data?

As part of the Results section of the evaluation report, Professor J. will present findings from a survey that was conducted for each year of the three-year project. She is considering reporting the data graphically but is unclear about how much complexity can be clearly conveyed by using simple bar graphs. She needs to report results for several categories of participant characteristics (e.g., years of teaching experience, geographic location, gender, age of participant, etc.) and wants to report the data in a format that is readily accessible to the evaluation stakeholders. Professor J. clicks on the Results & Recommendations excerpts to see graphic displays of quantitative results.

She finds several graphs in the Results & Recommendations excerpts. In addition, she reads several excerpts that report survey results by subcategories of participants in prose only. By comparing the prose and graphic results, she elects to use a combination of approaches. She will present basic survey findings with prose explanations, but rely on graphic displays to compare selected subcategory results by year. Professor J. makes copies of the relevant excerpts and graphics.

Skim complete reports Professor J. decides to skim several of the complete reports to see how all the basic components of sound Teacher Education reports have been integrated. She goes to the top of the page and clicks on the Reports tab, which returns her to the Teacher Education Reports page. Then she clicks on the link to each of the complete reports in turn to view them.
Bookmark the OERL Web site She bookmarks the OERL Web site on her browser, anticipating that she may browse more of the resources as she prepares a draft of the report.