Replication Lawfulness and Reliability

Absolute consistency is displayed when repeated observations yield identical results on a single dimension of comparison. Several methods could be used for measuring one mode of one type of helpfulness of one individual in one situation at one point in time. One obtains absolute consistency if all methods yield identical results. As was explained earlier, all methods must use the same metric. Otherwise results cannot be compared.

Relative consistency occurs when repeated observations yield corresponding results, with correspondence meaning, for instance, that the difference between two individuals on two dimensions of comparison is equal. Several methods could be used for measuring one mode of one type of helpfulness of several individuals in one situation at one point in time. In such a design, helpfulness could be compared on the dimension of individuals (Neighbors A, B, and C) and on the dimension of methods (Methods 1, 2, and 3). There is perfect relative consistency in this example when the differences between the helpfulness scores of A, B, and C are identical for all three methods. Again, this definition of consistency holds meaning only if all of the methods use the same metric or if the different metrics are transformed into a common metric (e.g., via ^-standardization).

Absolute and relative consistency imply lawfulness and reliability. Both lay judgment and scientific analysis define reliability as replicability (Willoughby, 1935). If a result can be replicated, we conclude that it was generated by a lawful process and that the method we used for obtaining the result was reliable. A single observation is insufficient to determine whether a result was generated by a systematic or a random process. Furthermore, it is impossible to know on the basis of a single observation whether or not the assessment method was reliable.

In the neighbor example, perfect relative consistency suggests that the behavioral differences between Neighbors A, B, and C are lawful and that the methods that revealed these differences are reliable. The consistent differences between the neighbors might result from differences in their altruistic personality. If this interpretation is correct, the methods then reliably assess altruistic personality. However, the interpretation may be wrong. Possibly, the neighbors do not differ in altruistic personality, but rather, in their need for praise. If this is true, the methods measure need for approval rather than altruistic personality.

0 0

Post a comment