Info

Note. ni+ represents the number of times Educational Psychologist A chooses "hyperactive" or "normal," respectively. The corresponding frequencies for Educational Psychologist B are denoted by n .. These marginals are obtained by adding the cell counts of the corresponding row (or column, respectively).

Note. ni+ represents the number of times Educational Psychologist A chooses "hyperactive" or "normal," respectively. The corresponding frequencies for Educational Psychologist B are denoted by n .. These marginals are obtained by adding the cell counts of the corresponding row (or column, respectively).

an individual. For example, the two ratings are associated if A assigns "hyperactive" to some of the pupils while B rates the same pupils as "normal," although there is no agreement (which only occurs when, e.g., both simultaneously rate "hyperactive" or "normal," respectively). Thus rater agreement can be seen as a special variant of association. In this chapter, the focus is on rater agreement, which plays an important role in the analysis of diagnostic accuracy. A high level of agreement between raters does not guarantee an individually correct diagnosis, but disagreement between raters often indicates a lack of diagnostic accuracy (Uebersax & Grove, 1990).

The association between variables and the extent to which methods or raters agree depend on two major criteria. First, it is important that both raters can distinguish well between any pair of categories. Distinguishability between two categories increases if the ratio of concordant ratings to discordant ratings of different observers increases. The second criterion is the lack of bias (Agresti, 1992). According to Agresti's definition, the amount of bias depends on the comparison of the marginal distributions: If raters use the response categories with the same frequency, their marginal distributions are homogeneous, indicating that none of the raters prefers a particular category compared to the other raters. However, homogeneous marginal distributions do not imply that all raters judge the subjects correctly compared to the subjects' true status, but they show that they use the response categories in a similar way. If all raters distinguish between categories in the same way and their marginal distributions are similar, subjects are more congruently assigned to the categories of a variable, thus providing hints that observers define the categories in a similar way.

Was this article helpful?

0 0

Post a comment