go home

Select a Professional Development Module:
Key Topics Strategy Scenario Case Study References

Introduction |  Step 1  |  Step 2  |  Step 3

Step 1: Conceptualize an instrument triangulation strategy (P), (R).
(P) = plan example
(R) = report example

Substep 1a. Identify the information you need and the groups from whom you need to collect it (R), (P).

To address some research questions, it may be necessary to obtain information from a variety of individuals (e.g., teachers, students, administrators, trainers, classroom observers, etc.) or data repositories, using a variety of data collection approaches (e.g., surveys, interviews, direct observation, collection of work products, etc.). Approaches for identifying information needs and suitable respondents were addressed in Step 3 of the Methodological Approach and Sampling module.

Groups who are participating directly in the project, such as teachers and students, are primary candidates from whom to collect information. Nonparticipating stakeholders who have a vested interest in the project because of a relationship with the participants would also be candidates, provided that they (1) are needed to corroborate information gathered from participants; (2) have a role in implementation, even if only indirect or occasional; or (3) are affected by the project in some way. For example, in an evaluation of a middle school science program designed to bolster the interest and confidence of students in doing science, data from parents as stakeholders may improve your understanding of the impact of the program.

Substep 1b. Identify sources of archived data (P), (R).

It is usually less costly to use existing data than to collect new data. Such existing data may be in quantitative data sets such as those managed by federal agencies (e.g., U.S. Census, National Assessment of Educational Progress), or in documents (e.g., school registration records, reports, logs, newspaper articles). If existing data are to be used in the evaluation, they should pertain explicitly to the people, places, events, or time periods for which you need information, and should meet your requirements for representativeness (see the Methodological Approach and Sampling module for more on these topics).

Table 1 shows examples of archived quantitative data and how they might be used in the evaluation.

Table 1. Examples of archived quantitative data for a project evaluation

Project description: A project designed to encourage more female students to major in science at undergraduate institutions
Evaluation question: Since the project's inception, are more female students majoring in science at the participating institution?
Age of the project: Three school years

Type of archived data available: Should the data be used in the evaluation?
Data showing each current female student's major Yes. It would also be beneficial to have the same data for the precedign two years as well, as a means of identifying possible changes inthe pattern of student majors.
The female students' grades in the science courses No, because the question is about majoring in science, not about improving performance in science courses.
Records of how many female students at the institution majored in science in the years before the project began Yes, as long as the data are used only as a comparison group to determine if there is a project effect, and only if a case can be made that the earlier populations of female students sufficiently resembled the current population.
Data on only the female students who are members of minority groups No, because the data would not be representative of the entire participant population, which also includes nonminorities.
Data on only 40% of the female students Yes, if (1) it can be shown the data are not biased (i.e., the data represent a cross-section of the entire population of the female students at the institution) and (2) there will be a large enough pool of data from the 40% to detect, with sufficient statistical power, the predicted effect (see the Methodological Approach and Sampling module for more on sampling requirements).

Table 2 shows examples of archived qualitative data and how they might be used in the evaluation.

Table 2. Examples of archived qualitative data for a project evaluation

Project description: The use of a new computer-based simulation tool in high-school physics classes
Evaluation question: Are the students learning physics better with the new tool?
Age of the project: One school year
Type of archived data available: Should the available data be used in the evaluation?
Teachers' written evaluations of individual student progress Yes, as long as there are comparable evaluations, by the same teachers, of student progress in comparison classes.
A newspaper article that describes a reporter's visit to a classroom where students were using the tool and the high level of interest in the tool that she observed among the students No, because the article captures only anecdotal data at one point in time, from one person's perspective. A better argument could be made for using a series of articles published over a period of time that might yield a pattern.
Student responses to open-ended tool related questions posed to them on an assignment that required the use of the tool Yes, as long as the assignment was about physics and not about mastering the user interface or some other topic not related to physics.

Substep 1c. Identify the types of instruments you will need for collecting new data (P).

If you cannot find archival data, your next recourse is to collect new data from the groups you have identified. To do so, you need instruments. The most common types of instrument are questionnaires, interview protocols, observation protocols, and learning assessments (see the various instruments of other modules for more information about the selection, design, and administration). Questionnaires and interview protocols are designed to elicit reporting from individuals. Observation protocols and learning assessments are designed to provide a record of something that the individual exhibits, be it knowledge, skills, or behavior. Each instrument type has advantages and limitations. Tables 3, 4, and 5 list the advantages of different instruments for different data collection needs, their limitations, and criteria for determining whether they should be used in the evaluation.

Table 3. Advantages and limitations of questionnaires and interview protocols, and criteria for using them

Advantages of questionnaires:
Good for systematically gathering background information, opinions, facts, and informed judgements.
Limitations of questionnaires:
Their standardized structure imposes constraints on the types of information that can be obtained from them. Respondents are often unwilling to offer detailed written explanations.
Advantages of interview protocols:
Greater potential than questionnaires for capturing deeper insights (e.g., more detail, greater articulation of problems because an interviewer can probe an individual's responses).
Limitations of interview protocols:
The presence of an interviewer may inhibit a respondent from answering questions candidly.
Criteria for using questionnaires and interview protocols:

  1. Individuals from the target population must have useful information on the issues. To satisfy this criterion, you must be able to say "yes" to one or more of these questions about the members of that population:
    • Do they have opinions on the issues that interest you?
    • Are they qualified to give you facts for informed judgements on the issues?
    • Are they the best or only source of information on the issues (i.e., information that is either not available or not as desirable to obtain from alternative sources, such as test scores or other records)?
  2. You should have a clear idea of the variables for which you want to gather data, and reasonably sure you will get those data by posing standardized questions.
  3. You should be confident that the individuals who are reporting can be relied on to give you the information you need, perhaps with incentives. This means that they must be
    • proficient enough in the language of the instrument to comprehend the questions and respond properly,
    • motivated enough to complete the questionnaire or interview and respond thoughtfully, and
    • knowledgeable and candid about the topics being addressed.

Table 4. Advantages and limitations of observations, and critieria for using them

Advantages of observations:
Good for capturing the dynamics of behavior taken from actual situations (e.g., the interplay of participants, characteristics of the setting, and sequence of events as they unfold).
Limitations of observations:
An individual observation can capture events in only one setting at one point in time. Hence, the results may represent exceptional rather than typical circumstances. Also, because an individual observation is carried out by one observer, the results may be inaccurate or misleading if the observer is not fully qualified to carry out the observation. To increase accuracy of the observations, videotaping can be done. However, videotaping can be costly and may require individual permission from all individuals who appear in the video (even those who appear only incidentally).
Criteria for doing observations:

  1. Identify the behaviors you need to observe (e.g., the amount of student time spent searching for information on the Internet, the number of open-ended questions posed by the teacher in a class discussion) and in what settings they occur (e.g., classroom, tutoring situation, extracurricular club).
  2. Be reasonably certain that you will get the desired data in the particular setting and in the particular time period you have chosen.

Table 5. Advantages and limitations of learning assessments, and criteria for using them

Advantages of learning assessments:
Good for capturing how much people have learned. They require learners to demonstrate rather than self-report their level of understanding.
Limitations of learning assessments:
One learning assessment by itself is unlikely to fully capture what a student has learned because it provides evidence of learning with only one particular set of tasks, usually in only one setting, and usually at only one time.
Criteria for learning assessments:

  1. Identify the concepts and processes you want to learners to demonstrate.
  2. Be reasonably certain that you will get the desired data in the particular setting and in the particular time period you have chosen.
  3. Make sure that the learners are proficient enough in the language of the assessment to comprehend the questions and respond properly.
  4. Make sure that the learners are motivated enough to complete the assessments thoughtfully.

Examine the advantages, limitations, and criteria in Tables 3, 4, and 5 with respect to your needs for information, and determine which instruments would be most appropriate to use with the different groups participating in the evaluation.

Substep 1d. Decide how you will triangulate to take optimal advantage of your instruments, and produce an instrument triangulation table.

Table 6 presents an example of three specific types of instruments that will be used for triangulation in a particular evaluation. Because each has its own advantages and limitations, the triangulation will maximize their strengths while minimizing the risks that would arise if any one of them were used alone.

Table 6. Examples of instruments that could be selected for use in a particular evaluation to achieve instrument triangulation

Evaluation goal:
To determine how well students are learning introductory biology in a new problem-based introductory biology curriculum.
Instrument options :
Classroom observation protocol:

  • Advantage: The observer can detect how many students manifest signs that they are learning (e.g., participating constructively in group projects and discussions) or not learning (e.g., acting distracted or confused, asking questions they should already be able to answer).
  • Limitations: The students may be tired or distracted on the observation day and hence not demonstrate their full capabilities to the observer.

A learning assessment composed of a series of selected-response items:

  • Advantage: relatively easy to score, easy to administer, and good for assessing a wide breath of knowledge in a limited time.
  • Limitation: not good for capturing the depth of student knowledge.

A learning assessment composed of a series of constructed-response items:

  • Advantage: good for eliciting evidence of deep thinking.
  • Limitation: relatively difficult to score and administer, and impractical for assessing a wide breath of knowledge because constructed-response items typically take longer to complete than selected-response items.

As a culminating exercise in conceptualizating a triangulation strategy, it is useful to identify the parameters of your strategy in an instrument triangulation table. In the table, you can identify the instruments needed for particular types of data. Use the first column of the table to specify the evaluation questions, the second column to specify what types of information you will need to answer the questions, and the third column to denote what archives or instrument types you want to use to obtain the information in a triangulated manner. The example in Table 7 has one evaluation question, two types of needed information, and, for each of these two types of information, two optimal information sources (archives and instruments). The example is about the evaluation of a project that has created a Web-based digital library of resources for the professional development of undergraduate engineering instructors.

Table 7. Example of an instrument triangulation table

Evauation question Information needed Instrument triangulation stratgey
What is the impact of the digital library on the professional practice of engineering instructors? Extent of use of the digital library Archives of numbers of visits to the digital library
Questionnaires to instructors
The nature of changes to professional practice by the engineering instructors in response to the availability of the digital library Questionnaires to instructors and to samples of their students
Protocols for interviews with the observed instructors

It is useful to generate a draft of your instrument triangulation table before you search for specific instruments to use or adapt. If you do not find enough existing instruments of a type listed in the table, and a determination is made that the advantages of triangulating in this instance are not worth the cost of developing new instruments, you may have to revise the strategy. The table can then be modified to reflect these adjustments.