Reliability and Multilevel Measurement

Raudenbush, Rowan, and Kang (1991) discussed the issues involved in multilevel measurement. One convenient way to model data such as these is to use a three-level model, with separate levels for the items, the pupils, and the schools. Using a model with no explanatory variables except the intercept, the variance between items is decomposed into variance components at the item, pupil, and school level. This model can be presented as

where y000 is the intercept term, and the subscript h refers to items, i to pupils, and j to schools. The variance components of items, pupils, and schools are 0.845, 0.341 and 0.179, respectively. The variance component <J2Uem can be interpreted as an estimate of the variation that is due to item inconsistency, C72pupil as an estimate of the variation of the mean item score between different pupils within the same school, and cr2schooi as an estimate of the variation of the mean item score between different schools. The item level exists only to produce an estimate of the variance that is due to item inconsistency. The error variance in the mean of p items equals o2 = <72j(em/p, which for the example data equals 0.141.

The pupil-level internal consistency is given by °Wa = a\jv)- For our example, data apupU is 0.71 and reflects the consistency in the item scores from different pupils in the same schools. The internal consistency coefficient of 0.71 indicates that this variability is not random error, but that it is systematic. It could be systematic error, for instance, response bias such as a halo effect in the judgments made by the pupils, or it could be based on different experiences of pupils with the same principal. This could be explored further by adding pupil characteristics to the model. The school-level internal consistency can be calculated by (Raudenbush et al., 1991, p. 312)

In Equation (8), p is again the number of items in the scale, and n. is the number of pupils in school j. Because the number of pupils varies across schools, the school-level variability also varies. An indication of the average reliability can be calculated by using Equation (8) with the mean number of pupils for n.. In our example, on average 8.9 pupils per school provided judgments, and the school-level internal consistency is ttschool = 0.77. The school-level internal consistency coefficient indicates that the school principal's leadership style is measured with reasonable consistency.

The school-level internal consistency depends on four factors: the number of items in the scale, the mean correlation between the items on the school level, the number of pupils sampled in the schools, and the intraclass correlation at the school level. The school-level reliability as a function of these quantities can be determined by kn.p.r a =_ILl_ (Q\