Most researchers were exposed to classical test theory during their training and so understand meas ures in terms of ideas related to reliability and validity. Oftentimes they have memorized some types of reliability and several types of validity. This simple approach to measurement can aid an investigator to proceed with his or her work, but is woefully inadequate compared to the psychometric sophistication we now possess. Some investigators have also been exposed to generalizability theory, and a few understand the triangles in Campbell and Fiske's (1959) multitrait-multimethod matrix. Our hope is that this volume has exposed readers to a deeper understanding of measurement that they can apply in their own research. A reader of this volume might no longer ask whether a test is valid, but will ask whether it is valid for certain purposes, and how a measure triangulates with other measures of the same supposed construct.
The statistics for analyzing the psychometric properties of our measures have gone beyond simple zero-order correlations coefficients. Modern statistical approaches can be used to model method-specific influences that correspond to theoretical assumptions about method effects. Data-analytic approaches such as loglinear modeling, item response theory, multilevel modeling, and models of generalizability theory, as well as structural equation modeling, enable researches to test hypotheses about the sources and generalizability of method effects in an appropriate way. Moreover, latent variable approaches allow us to separate unsystematic measurement error from systematic method-specific effects and to measure latent variables that can be related to other variables to explain trait and method effects.
MULTIPLE METHODS, NOT JUST MEASURES
Several authors in this volume argue that the need for multiple methods can be extended in new directions. Although some are despairing of multiple methods of measurement because of their frequent lack of convergence, others call for more applications of the basic idea of multiple methods. Miner and Hulin, in their chapter (chap. 29, this volume) on organizational research, call for a longitudinal dimension, with sampling over time, as a type of multimethod, and Marsh, Martin, and Hau (chap. 30, this volume) make a similar argument for cross-cultural measurement as a type of multiple method. Burns and Haynes (chap. 27, this volume) extend multiple method measurement to include more dimensions, including not only the method of measurement, but also including dimensions, facets, settings, modes, and occasions. In addition, multiple methods can be extended to experimentation, where multiple treatments and control groups can allow researchers to gain greater insight into the causal mechanisms in any given area (Smith & Harris, chap. 26, this volume). We welcome these extensions of multiple operations and yet want to remind readers that many of the same issues will apply to them as apply to multiple measures.
When using multiple occasions, experimental manipulations, organizations, cultures, and so forth, researchers need to be prepared for the fact that there might only be modest convergence between them. Just as with multiple measures, it is a relief when other types of multiple methods produce similar conclusions. But one should not despair if this is not the case—as long as one is patient and understands that scientific progress takes time. Much can be learned when different cultural patterns are pursued in further research, when a pattern is sought in longitudinal differences, or when experimental manipulations of supposedly the same construct lead to different outcomes. Indeed, we would argue that it is in these circumstances, just when researchers sometimes give up in despair, that the conditions are right for important advances in science. If the researcher persists, he or she is likely to discover interesting points in nature, where a construct is really two distinct concepts or where the effects of a variable depend on the context. Thus, multiple methods are likely to lead to a more complex and sophisticated science.
Was this article helpful?