Alignment Table for Report Components

Describes the type or types of analyses conducted (e.g., quantitative, qualitative, mixed methods) and procedures used for examining results and ensuring their trustworthiness, such as:

• training conducted to ensure reliable coding and scoring of data
• checks of the data to remove errors
• procedures for reducing and summarizing the data
• descriptions of analyses, that identify a pattern of results

This section also describes results non-interpretively (e.g., without being subject to values, perspectives, and conceptual frameworks).

Describes procedures taken to analyze numeric data:

• organizing the data
• verifying it
• summarizing it
• presenting purely descriptive information about the project (e.g., percentages of different responses to a survey question; percentages of different scores on a test item) that could lead to patterns and trends
• examining relationships among variables (e.g., Pearson Product Moment correlations, multiple regression, factor analyses)
• using inferential statistical techniques to test for significant differences between comparison groups (e.g., t-tests, analyses of variance, analyses of covariance)

The quantitative analysis procedures should be appropriate to the evaluation questions being addressed and the characteristics of the information being analyzed. The practical significance (e.g., effect sizes) and replicability, as well as statistical significance, should be considered when drawing inferences and formulating conclusions from quantitative analyses. Analyses of effects for identifiable subgroups should be considered, as appropriate, because a program may have differential effects for them.

In addition, the number of informants who actually provided data should be reported. (Informants who fill out a survey are called "respondents," and the percent of those solicited who actually respond is called the "response rate." This will help reviewers determine the extent to which the informants are representative of the total population.

Potential weaknesses in the quantitative data analysis, along with their possible influence on interpretations and conclusions, should be described.

A8 Analysis of Quantitative Information
Quantitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A7 Systematic Information
The information collected, processed, and reported in an evaluation should be systematically reviewed, and any errors found should be corrected.

Describes the qualitative analysis procedures used to compile, analyze, and interpret the data in order to find themes, patterns, and trends.

The qualitative analysis procedures should be appropriate to the evaluation questions being addressed and the characteristics of the information being analyzed. As the evaluation progresses, the accuracy of findings from qualitative data must be confirmed by gathering evidence from more than one source and by subjecting inferences to independent verification.

Potential weaknesses in the qualitative data analysis, along with their possible influence on interpretations and conclusions, should be described.

A9 Analysis of Qualitative Information
Qualitative information in an evaluation should be appropriately and systematically analyzed so that evaluation questions are effectively answered.

A7 Systematic Information
The information collected, processed, and reported in an evaluation should be systematically reviewed, and any errors found should be corrected.