All general agreement indices described so far fail to provide more detailed information about various types and sources of agreement and disagreement. This kind of information can be obtained by modeling association between variables using loglinear models. Moreover, for special cases of association, effect sizes can be estimated representing the degree of association between variables. Conditional probabilities of receiving a particular response by an observer given the responses of other observers can be computed. Finally, residuals can be determined that compare the frequencies with which certain types of agreement and disagreement occur compared to what would be expected with some predicted pattern (Agresti, 1990, 1992).
Since the 1970s, the analysis of categorical data by means of loglinear models has strengthened its position as more and more investigators successfully applied loglinear models in their research. Many extensions of the models in several directions have been developed as, for example, the ordinary loglinear model, the standard latent class model, and the loglinear model with latent variables (for an overview, see Agresti, 1990; Hagenaars, 1990, 1993).
Table 17.3 presents a typical situation for the analysis of multimethod data. Two educational psychologists rated the behavior of 153 pupils as hyperactive, dyslexic, or normal. A rated 25 pupils as hyperactive, 20 dyslexic, and 108 normal. B classified the pupils' behavior in a similar manner (26 hyperactive, 19 dyslexic, and 108 normal). Both raters agreed on 16 hyperactive diagnoses, 15 dyslexic diagnoses, and 99 normal diagnoses. In sum, they agreed on 130 ratings and disagreed on 23 ratings, whereas the majority of discordant ratings is found in the categories "normal" and "hyperactive."
Artificial Data of Pupils' Diagnoses by Two Educational Psychologists
Educational Psychologist B
Hyperactive Dyslexic Normal nu
Educational Hyperactive 16 3 6 25 Psychologist A Dyslexic 2 15 3 20 _Normal_8_1_99_108
Loglinear models aim to capture sources of associations between different categorical variables, and these associations are mirrored by different effects in the loglinear model. To understand the special meanings of loglinear models for the analysis of rater agreement, the most general loglinear model—the saturated model—will be introduced first.
Loglinear models are implemented to reproduce the joint frequency distribution of empirical data situations. Thus, the expected frequencies (ey) implied by a model have to match the observed frequencies. Expected frequencies can be determined by the multiplicative form of the model:
The expected cell frequency (e.; with i = 1,. . . I and j = 1, ... ,J denoting the categories) are computed by the product of the overall effect Cn), two one-variable effects ), and the two-variable effect(VB).
The overall effect (r|) represents the geometric mean of all cell frequencies and is, thus, nothing other than a mere reflection of the sample size
(Hagenaars, 1993). The one-variable effects [x*, x*)
reflect deviations of the geometric mean of all cells belonging to the ith (jth) category of a variable.
Finally, the two-variable effect (t^) depicts deviations of the expected frequency of a particular cell beyond the overall and one-variable effects. The parameters can be estimated for the example given in Table 17.3 as follows:
T| = ^16x2x8x3x15x1x6x3x99 = 6.49,
Was this article helpful?