To remedy the shortcomings of the proportion agreement index when the prevalence of a critical obser vation is very low or high, the occurrence and nonoccurrence agreement indices can be used. The computation of both indices is quite similar to the proportion agreement index, whereby the occurrence (or nonoccurrence, respectively) agreement index only reflects the number of times both raters agree on the occurrence (nonoccurrence) of the critical category and the number of times both raters disagree in general (on occurrence and nonoccurrence). The occurrence index (pocc) should be used when the prevalence rate falls below .20. When the prevalence rate is higher than .80, the nonoccurrence agreement index (p ) should be used (Kelly, 1977). The occurrence index is defined as p
í ore occurrence agreements occurrence agreements + disagreements independence of two ratings. The expected cell frequencies in the independence model are computed as the product of the row and column sums, divided by the total number of observations. If a researcher is interested in any kind of association between variables, one has to test the joint distribution of two variables against the assumption of independence. In addition to the independence model, other hypotheses can be tested by comparing the expected frequencies implied by a particular hypothesis with the observed frequencies. For instance, if the object of interest is the agreement of two novices' ratings compared to the agreement of two experts' ratings, it would be necessary to set the frequencies of the experts' ratings contingency table as expected values. In general, the %2 value can be computed by
By substituting the occurrence agreements with nonoccurrence agreements, the nonoccurrence agreement index can be computed. Given the data in Table 17.1a, where the prevalence rate is .11, the occurrence agreement index should be used. The occurrence agreement index provides a value of 40
40+ (15+ 20) 17.1b, the occurrence agreement index yields a value of zero, indicating that both raters did not agree for at least one critical observation. Thus occurrence and nonoccurrence agreement indices correct for most of the inflation by chance, but they do not correct for the total inflation by chance (Suen & Ary, 1989). One limitation of the occurrence agreement index can be viewed in the fact that often no prior knowledge about the prevalence rates exists, that is, knowledge that would enable a theoretically founded application.
Was this article helpful?