Analysis of variance. The application of analysis of variance (ANOVA) models has a long tradition in multimethod research. To analyze the conver gence of several methods measuring the same trait, ANOVA models are routinely applied (Mill-sap, 1995b; Tinsley & Weiss, 2000). In general, two types of factors can be considered in ANOVA models: random and fixed factors. Random factors are considered when the levels of a factor are a random sample from a population and the research goal is to generalize to the population. For example, if different raters are randomly selected to rate the trait of different individuals, ANOVA with random factors can be applied to test convergent validity. In this case, variance components and intraclass correlation coefficients can be estimated to indicate the convergence of the different methods (Shrout & Fleiss, 1979; Tinsley & Weiss, 2000). In the case of structurally different methods, the factor can be considered fixed, and differences between the methods can, for example, be analyzed by planned contrasts. ANOVA designs can easily be adapted for MTMM studies if one considers the three factors person, trait, and method. Millsap (1995b) discussed the advantages and limitations of ANOVA models for MTMM research. He concluded that ANOVA designs are most appropriate when method influences generalize across traits but that ANOVA models have problems detecting method influences that are trait specific, that restrict variances (such as the central tendency response bias), and that are related to rater halo effects.
Generalizability theory. The classical ANOVA framework has been a starting point for many theoretical and methodological extensions from which generalizability theory has become very influential. Cronbach, Gleser, Nanda, and Rajarat-nam (1972) developed generalizability theory based on the ANOVA methodology as a theoretical framework for analyzing the dependability of psychological measurements on different sources of influences (e.g., methods). Several coefficients for evaluating the generalizability of the results of a study can be estimated (Hox & Maas, chap. 19, this volume). Moreover, generalizability theory builds a fruitful theoretical framework for the conceptualization and the analysis of multi-method studies because it allows the considera tion of different types of methods (random, fixed) and different types of method structures. For example, the same raters can rate all individuals (crossed design) or raters can be specific for one individual, for example, friends (nested design). Multivariate generalizability models also allow multiple indicators for a trait-method unit, for example, by considering indicators as a further facet or by referring to multivariate models of generalizability theory (Jarjoura & Brennan, 1983). Hox and Maas (chap. 19, this volume) give an introduction to generalizability theory.
Multilevel modeling. In recent years, multilevel analysis, which represents another extension of linear models such as ANOVA and regression analysis, has become very popular in psychological research (Bryk & Raudenbush, 1992; Goldstein, 1995; Hox, 2002). Multilevel models have been developed to analyze data that are hierarchically ordered. For example, if an individual is rated by several friends who are chosen from his or her group of friends, this is a typical nested design with raters nested within targets. Multilevel models are particularly appropriate for these data structures, as they allow a very flexible modeling of method effects for these designs. For example, the number of friends chosen could be different for different target individuals. Multilevel models particularly allow a very flexible analysis of interchangeable methods, such as randomly selected raters, although other types of methods can also be considered.
Was this article helpful?