Critique Of Mechanical Prediction Automated Assessment

As already noted, automated assessment programs consist of a series of if-then statements. They are written by clinicians on the basis of their clinical experiences and their knowledge of the research literature and clinical lore. They are considered to be mechanical prediction rules because statements generated by automated assessment programs are 100% reproducible.

Several strengths of automated assessment programs can be described. First, they are written by clinicians who are generally thought to be experts. Another advantage is that they are mechanical prediction methods, and thus test-retest reliability is perfect (e.g., given a particular MMPI-2 test protocol, the same test report will always be written). Also, the general superiority of mechanical prediction methods was supported by Grove et al. (2000), although results were not analyzed separately for automated assessment programs and statistical prediction rules.

A number of weaknesses can also be described. First, in empirical studies, alleged experts often have been no more accurate than other clinicians (for reviews, see Garb, 1989, 1998; Garb & Schramke, 1996). Second, although test-retest reliability is perfect, interrater reliability is not. Computer-based test reports generated by automated assessment programs are generally used by clinicians along with other information (e.g., history information). One should not assume that psychologists will make similar judgments and decisions when they integrate all of this information. Finally, and perhaps most important, many automated assessment programs for interpreting psychological test results are not validated (Adams & Heaton, 1985; Garb, 1998, 2000b; Garb & Schramke, 1996; Honaker & Fowler, 1990; Lanyon, 1987; Matarazzo, 1986; Snyder, 2000; Snyder, Widiger, & Hoover, 1990; but also see Butcher, Perry, & Atlis, 2000). Thus, automated assessment programs can "lend an unwarranted impression of scientific precision" (Snyder, 2000, p. 52).

0 0

Post a comment