What is the relation between scales based on responses from new and existing self-concept instruments? Historically, self-concept research was plagued by a surfeit of idiosyncratic instruments that hindered communication among researchers and research syntheses. This situation invited the Jingle-Jangle Fallacy (Marsh, 1994) whereby researchers mistakenly assumed that two scales with the same label measured the same construct or that two scales with different labels measured different constructs. The MTMM design in which the multiple methods are the different instruments has been highly effective in addressing this critical issue, as illustrated by two studies summarized in this section.
In a classic example of this MTMM approach, Marsh (1989, 1993b) examined the relations between three academic self-concept traits (math, verbal, and general school) measured by three different instruments. The 9 scores representing all combinations of the 3 traits and 3 methods were based on multi-item scales, and the three instruments had strong psychometric properties. Consistent with theory and considerable prior research, math and verbal self-concepts were nearly uncorrected with each other and were substantially correlated with school self-concept. In the CFA MTMM analysis, the trait factor loadings were consistently large (convergent validity), the trait factor correlations were small or moderate (discriminant validity) , and the correlated uniqueness (method effects) were small to moderate. Method effects were smaller for the first instrument than the second and particularly the third instrument, whereas trait effects were smaller for the third instrument. Hence the results supported the convergent and discriminant validity of the self-concept responses and provided useful diagnostic information about each of the three instruments.
Examining preadolescent responses to three self-concept instruments. Marsh (1990b) demonstrated the CFA approach to MTMM data based on preadolescent responses to three widely used self-concept instruments. He demonstrated two interesting variations on the typical MTMM design in that not all traits were assessed by all the different measures. First, two instruments contained self-esteem scales, whereas the third did not. Hence, the general self-trait factor was represented by two indicators instead of three. Second, one instrument contained two separate physical scales representing physical appearance and physical ability; one instrument contained only one physical scale apparently representing physical ability; one instrument contained only one physical scale apparently representing physical appearance. An evaluation of alternative MTMM models demonstrated the need to separate physical ability and physical appearance trait factors. The MTMM analyses provided support for convergent validity for all three instruments and for the divergent validity for two of the instruments, but also contributed to understanding the multidimensional structure of self-concept and particularly the physical facet of self-concept.
In summary, the MTMM design in which the multiple methods are different instruments is very useful in the construct validation of new and existing measures. Importantly, this multimethod approach also provides clear tests of jingle-jangle fallacies based on clusters of seemingly similar constructs based on different instruments, different theoretical perspectives, and results from different research teams.
Was this article helpful?