Clinical Versus Mechanical Prediction

The most comprehensive and sophisticated review of studies on clinical versus mechanical prediction was conducted by Grove et al. (2000). In addition to locating more studies than anyone else, they published the only meta-analysis in this area. Their review will be described in detail.

In their search of the literature, Grove et al. (2000) included only studies in the areas of psychology and medicine. Studies were included if clinicians and mechanical procedures were used to "predict human behavior, make psychological or medical diagnoses or prognoses, or assess states and traits (including abnormal behavior and normal personality)" (p. 20). Also, studies were included only if the clinicians and the mechanical procedures had access to "the same (or almost the same) predictor variables" (p. 20). After an extensive search of the literature, 136 studies were found that qualified for inclusion.

The results reported by Grove et al. (2000) favor mechanical prediction. Mechanical prediction techniques substantially outperformed clinical prediction in 44%, or 60%, of the studies. In contrast, clinicians substantially outperformed mechanical prediction techniques in 6%, or 8%, of the studies (results were calculated from their Figure 1, p. 21). In the remaining studies, clinical predictions were roughly as accurate as mechanical predictions. On average, mechanical prediction rules were about 10% more accurate than clinicians.

Overall, the results of the meta-analysis support the general superiority of mechanical prediction. However, in light of these findings, comments made by statistical prediction advocates seem too extreme. For example, Meehl's (1986, p. 374) claim that there are only "a half dozen studies showing even a weak tendency in favor of the clinician" no longer seems accurate. As noted by Grove et al. (2000), "Our results qualify overbroad statements in the literature opining that such superiority is completely uniform" (p. 25).

Grove et al. (2000) also reported additional interesting findings. The general superiority of mechanical prediction holds across categories: "It holds in general medicine, in mental health, in personality, and in education and training settings" (p. 25). They also found that mechanical prediction was usually superior regardless of whether clinicians were "inexperienced or seasoned judges" (p. 25). With regard to a third result, one variable was notable in the eight studies in which clinical judgment outperformed mechanical prediction: In seven of those eight studies, the clinicians received more data than the mechanical prediction rules. One implication of this finding is that optimal information has not always been used as input for mechanical prediction rules. One more result will be mentioned. Mechanical prediction rules were superior to clinicians by a larger margin when interview information was available. Limitations of interview information have been described in the clinical judgment literature (Ambady & Rosenthal, 1992; Garb, 1998, pp. 18-20).

To check on the integrity of their findings, Grove et al. (2000) conducted additional analyses:

[We] examined specific study design factors that are rationally related to quality (e.g., peer-reviewed journal versus chapter or dissertation, sample size, level of training and experience for judges, cross-validated versus non-cross-validated statistical formulae). Essentially all of these study-design factors failed to significantly influence study effect sizes; no such factor produced a sizable influence on study outcomes. (p. 25)

Thus, roughly the same results were obtained in studies varying in terms of methodological quality.

The Grove et al. (2000) meta-analysis is a landmark study, but it does not address many important issues. For example, specific mechanical prediction rules that clinicians should be using are not described; nor are obstacles to developing better mechanical prediction rules. Finally, conditions under which clinical judgment should be preferred to mechanical prediction are not described. These issues and others will now be discussed.

Was this article helpful?

0 0

Post a comment