Multilevel Models for Generalizability Analysis

Generalizability theory can be viewed as a special case of multilevel analysis. In the one-facet nested design, the nesting structure is clear: The items are nested in the persons. In the one-facet crossed design, the nesting structure is arbitrary: Items can be seen as nested in persons or persons nested in items. Both specifications lead to the same results. Because of the analogy with the nested design, we will use the specification structure of items in persons. In a two-facet design, the structure is more complicated because of the large number of interaction effects. Although it can be specified as a cross-classified multilevel model (Goldstein, 1995), current software cannot analyze data of a realistic size and complexity.

The specification of a one-facet nested design is straightforward. An intercept-only model is specified with two levels. At the lowest level we obtain a direct estimate of the residual variance and at the second level, a direct estimate of the person-level variance. These estimates are exactly the same as the variance components estimated before.

The one-facet crossed design, where all people respond to all items, is specified as a three-level intercept-only model. Although the analysis is set up using three separate levels, it should be clear that conceptually we have two levels, items nested in persons. The lowest level is added to estimate the residual variance; the item and person levels are "dummy" levels with only one unit that covers the entire data set (cf. Hox, 2002). At the lowest level the items are represented by a full set of dummy variables. The fixed coefficients of these dummies are excluded from the model, but their slopes are allowed to vary at the second (item) "dummy" level. The covariances between these dummy variables are all constrained to zero, and their variances are all constrained to be equal. Thus, we estimate one variance component for the items. The specification of the third (the person) level is similar. At the lowest level we obtain a direct estimate of the residual variance, at the second level the item variance is estimated, and at the third level the person variance. The estimates are exactly the same as the variance components estimated with ANOVA. Because the software specification for the multilevel approach requires as many dummy variables as there are subjects in the data set, it is clear that data of a realistic size and complexity pose severe difficulties.

A special case of crossed facet designs is the situation in which people only partially respond to the same items (see Table 19.5, as a special case of Table 19.4). Analyzing these data as a crossed design with the ANOVA approach is not feasible because of the empty cells in the observed data set. Multilevel analysis of these data is straightforward. Following the same procedure as described for the one-facet crossed design, estimates for the variance components for the items, persons, and residual are obtained.

The variance components are estimated as <j2 = 0.006, (J2 =0.019, and a2 .. , = 0.239.

p i residual

Two percent of the variance is associated with persons, 7% with items, and the remainder with the interaction and error.

Item Scores of Eight Respondents on Four Multiple-Choice Items With Incomplete Data


Was this article helpful?

0 0

Post a comment