For our examples, we intentionally used four sources rather than multiple methods. The use of multiple methods such as interviews, rating scales, and direct observations would at first glance appear to represent an ideal multitrait (ADHD-IN, ADHD-HI, and ODD) by multimethod (interview, rating scale, and direct observations) matrix. Unfortunately, this type of matrix often contains a number of confounds that can make the interpretation of the results difficult (Burns, 1980; Cone, 1979).
One complexity concerns the time frame for each method. If the time frame for the diagnostic interview was the past 6 months, the past 1 month for the rating scale, and the past 5 school days for the observational measure, the interpretation of the findings would be problematic because of the varying time frames for each method. To eliminate this confound, it would be necessary to hold the time frame constant for each method.
A second complexity involves the source of the information across the three methods. Consider these possibilities. For the rating scale method, the source is the teacher. For the diagnostic interview method, the source represents a combination of the information provided by the parent and decisions made about the information by the interviewer, whereas the behavioral observations during recess are made by the school counselor. Here source is confounded with method as well as situation. A possible solution would be to use the same source for each method, although this raises additional complexities.
A third complexity with multiple methods concerns the possibility that the content of the trait may be specific to the method. For example, the content of the diagnostic interview may be slightly different from the rating scale with both these methods having different content than the observational measure. Although such content differences across methods are at times appropriate because different methods have different goals (e.g., diagnosis versus treatment outcome evaluation), content differences can create problems in the use of CFA to model the matrix. Here different outcomes across the methods could be a function of the different representations of the traits in each method. To eliminate this confound, it is important that similar traits contain similar content for the different methods (e.g., the content of the ADHD-IN trait is similar for the interview, the rating scale, and the observational methods).
These complexities can make the interpretation of results from multitrait by multimethod by multi-source analyses difficult. However, with careful planning, CFA can still be used to model such complex matrices and therefore provide a richer understanding of clinical psychology measures. We now turn to a review of the use of the procedures in clinical psychology.
THE AMOUNT OF TRAIT, SOURCE, AND ERROR VARIANCE IN CLINICAL PSYCHOLOGY MEASURES
An attempt was made to locate all published studies on topics in clinical psychology that used the correlated trait-correlated method CFA approach to model multitrait by multisource matrices. A search was made through PsycINFO from 1980 to the 2003 with the terms "multitrait-multimethod" and "confirmatory factor analysis." Many of the studies used the correlated uniqueness approach to separate trait from source variance (e.g., Cole, Martin, Peeke, Henderson, & Harwell, 1998; Crystal, Ostrander, Chen, & August, 2001). In addition to the possibility that the correlated uniqueness approach artificially inflates the amount of trait variance in the measures (Lance et al., 2002), this approach also combines the source with the error variance, thus making it impossible to determine the amount of source variance in each measure (Lance et al., 2002). Given our purpose to summarize the amount of trait, source, and error variance in measures used in clinical psychology, our review covers the correlated trait-correlated method studies.
Table 27.1 summarizes the results from these studies. Two of the studies focused on the amount of trait, source, and error variance in parent and teacher ADHD rating scales (Burns et al., 2003; Gomez et al., 2003). Here the source effects were strong, being consistently larger or equal to the trait effects. The one exception was that the teacher ratings of the ADHD-HI symptoms consistently showed slightly more trait and source variance (see also Gomez, Burns, Walsh, & Hafetz, 2005). For measures of depression and anxiety in children (e.g., the Child Depression Inventory and the Revised Manifest Anxiety Scale; Cole, 1990; Cole, Truglio, & Peeke, 1997), the source effects were consistently stronger than the trait effects with there often being little trait variance in the measures of depression and anxiety.
There was only one study with two sources (mothers and fathers) in the same situation (Rowe & Kandel, 1997). Here the trait variance was greater than the source variance for measures of externalizing problems (Child Behavior Checklist aggressive and delinquent behavior scales) but not for internalizing problems (Child Behavior Checklist withdrawn, somatic complaints, and anxious/depressed scales). For teacher and mother ratings of conduct problem behaviors across 3 years (Fergusson & Horwood, 1989), the amount of source variance was slightly greater than the trait variance for mothers with the reverse occurring for teachers. In a study that used the Psychopathy Checklist—Revised, source effects were also either equal to or stronger than trait effects for interviewer and therapist ratings of personality (e.g., callous
Was this article helpful?