Inappropriate aggregation cannot only damage the construct validity of a measure but also disguise systematic patterns in the data and lead to misleading substantive conclusions. Aggregation can be inappropriate and can potentially disguise important information whenever facets of the data box interact (i.e., when differences between objects on one dimension differ systematically on another dimension). Consider differences in grades on three facets, the person (student) facet and two method facets: the teacher facet and the type of exam facet (oral versus written). Assume that every student received a better grade from Teacher A in an oral exam than in a written exam, whereas Teacher B gave a better grade to every student in a written exam than in an oral exam. Assume further that grades differed consistently among students across all four methods. Figure 2.1 schematically depicts the entire data pattern. Aggregation across students is appropriate because grade differences are perfectly generalized across the other two facets. However, aggregation across the two method facets masks two sources of method bias. Aggregation across teachers suggests that exam type does not matter. Aggregation across exam types suggests that teachers make no difference. Although these conclusions are technically correct on the level of grade averages, they preclude a deeper understanding of the methods by ignoring an interaction between the teacher and exam type facets. Aggregation thus results in a loss of information that might be of theoretical interest and great practical importance for avoiding method bias. Method bias occurs in this example when only written or only oral exams are administered and if some of the students are tested by Teacher A, whereas the rest are tested by Teacher B. For a comprehensive treatise of the method bias issue, see Hoyt (2000).
Was this article helpful?