Info

Response or the (high) difficulty of giving a free response. On the other side, the third method in this hypothetical example, the numerical response format, interacts with the content area depending on which formula has to be applied and on the choice of the numbers. An easy numerical calculation makes the items easier, whereas the application of a complex formula and or fractions can make the items rather difficult. From this perspective, the main effects and the interaction model are not...

Echo Planar Imaging

In a completely uniform magnetic field, the spatial location of the measured MR signal cannot be determined. However, by temporarily imposing a separate magnetic field that varies linearly across a volume, known as a gradient, it becomes possible to temporarily alter the strength of the external magnetic field in a spatially specific manner. This spatial encoding is achieved by the application of gradient fields in three dimensions at critical times in relation to the RF pulse. Gradients affect...

Pet Experimental Design And Data Analysis

The range of experimental design strategies for PET is limited by the physical constraints associated with delivering and measuring radioisotopes. At a minimum, 150 studies involve an aggregation of activity occurring over 30 seconds or more, and for metabolism and neurotransmitter studies the aggregation covers 20 to 60 minutes. This temporal resolution precludes event-related types of designs and makes it difficult to dissociate the different processes involved in a task. The total number of...

Temporal Resolution

Because of the nature of both radioisotope decay and the slow time course of the hemodynamic response, the temporal resolution of neuroimaging is inherently limited. The sluggish nature of the hemodynamic response prohibits the detection of numerous events that occur on a millisecond time scale and may be conceptualized as a low-pass filter that prevents detection of higher frequency information. The temporal resolution is particularly poor for PET, which is largely insensitive to transient...

Assessing Multimethod Association With Categorical Variables

This chapter provides an introduction to methods for analyzing the associations between categorical variables. The focus is on the analysis of nonordered categorical data, also referred to as nominal data or nominal variables (for the analysis of ordered categorical data, see Rost & Walter, chap. 18, this volume). First, general association indices such as the proportion (or percentage) agreement index, the occurrence (nonoccurrence) agreement index, the chi-square value, and coefficient...

Summary

Tests of human cognitive abilities assess arguably the most scientifically significant individual differences uncovered by psychological science. Tests of general intelligence were the focus of this chapter because the dominant dimension that runs through them accounts for 50 of the variance in heterogeneous tests (across a wide range of talent) and the majority of criterion variance that cognitive abilities can predict in school, training, and work settings. This latter generalization pertains...

Background

I O psychology is, from one viewpoint, a pragmatic psychology. Much research is devoted to the prediction of behavior in organizations. For example, what are the psychological factors that lead a group of employees to attempt to change a dissatisfying or stressful situation by unionizing What factors go into decisions to quit one's job What psychological traits best predict job performance across a wide range of occupations Effort typically goes into identifying measures that accurately predict...

Generalizability and Specificity

Although the implications of multidetermination were discussed with regard to methods, they apply to all facets of a data box. The degree of relative consistency across time, situations, and other facets always depends on the relative weights of common and unique factors. Again, this has important consequences for psychological measurement (e.g., if helping in different situations depends only on altruistic personality, individual differences in helping remain nonspecific across situations)....

Using Meta Analysis to Compare and Integrate Multimethod Research

A critical challenge in the systematic evaluation of multimethod research is how to compare and integrate results from multiple studies that use different outcome measures, different research designs, different samples, different methodologies, and different statistical analyses. An important limitation to the multimethod approach is that an individual study or even a research program is typically too narrow in focus to take full advantage of the multi-method approach. Hence, the potential...

Psychometric Properties Of Neuroimaging Data

The selectivity, sensitivity, criterion validity, and test-retest reliability can be calculated for both PET and MRI studies. Establishing the test-retest reliability of most baseline PET measures is relatively straightforward (Ball, Fox, Herscovitch, & Raichle, 1988 Nyberg, Farde, & Halldin, 1996 Schmidt et al., 1996), although this literature remains surprisingly small considering the increasing use of these measures in clinical diagnosis. Establishing the test-retest reliability of...

Glucocorticoids and Behavioral States Reciprocal Determinism

Glucocorticoids (Cortisol in humans, corticos-terone in rats) are steroid hormones of the adrenal cortex that have potent effects on glucose metabolism and immune function, as well as on psychological processes (Gore & Roberts, 2003 Lovallo & Thomas, 2000 Schimmer & Parker, 1996). Glucocorticoids are classic stress hormones and have been commonly used as biochemical markers of stress reactions (McEwen, 2000). As illustrated in Figure 12.6, the secretion of glucocorticoids is regulated...

Data Analytic Issues

Statistical issues primarily deal with the calculation of accuracy in terms of self-other agreement or consensus. The potential artifacts identified by Cronbach can be estimated easily when multiple informant ratings are replicated across multiple targets. The social relations model provides unique methods that allow separating the very different components of informant ratings, (i.e., genuine elevation, differential elevation, stereotype accuracy, and differential accuracy see Figure 4.1). The...

Simultaneous Measurement of Other Variables to Enhance Understanding of Neuroimaging Data

Although the preceding discussions have focused on the ability of neuroimaging data to provide information on psychological constructs, a multimethod approach can also prove extremely useful in directing the interpretation of neuroimaging data. Group statistical analyses often proceed on the assumption that all subjects performed a cognitive task in a similar manner or responded similarly to procedures aimed at inducing a specific psychological state. Unfortunately, verification of this...

Rasch and IRT Approaches

For unidimensional constructs with dichotomous or ordered categorical items, the Rasch model and the 2-parameter logistic IRT model are commonly used (see Embretson & Reise, 2000). The Rasch model has one parameter (bp for each item representing its difficulty (level), whereas the two-parameter IRT model has both a difficulty parameter (b.) and a discrimination (slope) parameter (a) for each item. Assessment of measurement invariance across time involves checking that the item parameters a....

Innovative Computerized Tests

The previous sections in this chapter have shown great advances in testing and assessment due to the introduction of computer-based tests. However, a legacy of paper-and-pencil testing the multiple-choice question seems to have constrained the way test developers think about assessment This item type has continued as the predominate format. But there are many skills and abilities that do not lend themselves to assessment with multiple-choice items. For instance, how should a physician's ability...

Benefits of Experimental Assessment

What are the benefits of the experimental approach to psychological assessment First, and perhaps most important, the experimental method provides sound solutions to what we have called decomposition problems elsewhere (e.g., Buchner, Erdfelder, & Vaterrodt-Plunnecke, 1995 Erdfelder & Buchner, 1998a, 1998b). Decomposition problems arise because empirical psychological variables (such as test scores, response times, etc.) are almost always affected by more than a single psychological state...

Empirical Evidence And The Postulation Of Mental Systems

Recall that we began this chapter with a series of pithy comments about the ways in which cognitive psychologists derive evidence for theoretical entities. That task begins with an analysis of empirical data and proceeds to a theoretical interpretation only through the lens of a particular model. Although we have not emphasized it here, it is important to remember that any comparison of conditions or measures assumes some underlying model, and that those comparisons that are simple do not...

Nonreactivity as a Continuous Concept

A continuous conception of nonreactivity assumes it to be an ideal state rather than a feature inherent in certain techniques of measurement. Here, measures are assumed to vary in the extent to which they come close to this ideal. According to this notion, the reactivity of a measure increases with the extent to which both the research subject and the researcher are involved in the act of measurement. We will now introduce six types of measures (see Table 14.1) that vary in the extent to which...

Measuring Emotions Implications Of A Multimethod Perspective

Larsen and Zvjezdana Prizmic-Larsen Other chapters of this book address various conceptual concerns about multiple methods, about convergent and divergent validity, and about the distinction between method and error variance. The construct of emotion provides special challenges for assessment, thus bringing these abstract conceptual issues into sharp relief. We therefore begin our chapter on emotion measurement with a few deliberations concerning these special challenges. We then...

Recording Observational Data

Once the hard work of developing coding schemes is past, trained observers are expected to categorize (i.e., code) quickly and efficiently various aspects of the behavior passing before their eyes, audible by their ears, or both. The behavior may be live, an audio or video recording (in either analog or digital form), or a previously prepared transcript, but one basic question concerns the coding unit To what entity is a code assigned Is it a neatly bounded time interval such as the single...

Correlated Trait Correlated Method Ml Model

As an alternative to the CTCM model, Eid (2000) proposed an MTMM model that is not affected by the identification problems of the CTCM model. Eid's model is a special variant of the CTCM model but differs from it in the number of method factors. It contains one method factor less than methods included and is, therefore, called the correlated trait correlated method minus one CTC(M-l) model. The basic idea of the CTC(M-l) model is that one method has to be chosen as the comparison standard. All...

Recall Biases In Selfreport Data

A major theme in the development of diary and momentary capture of self-report data is the possibility that biases contaminate recall. As it turns out, at least three fields of scientific study have contributed to our knowledge of these biases survey research, autobiographical memory research, and cognitive science. Many excellent reviews of factors exist that can influence self-reports (e.g., Bradburn, Rips, & Shevell, 1987 Gorin & Stone, 2001 Schwarz, Wanke, & Bless, 1994 Schwarz,...

Selfreport Methods Need To Be Supplemented With Other Methods

Some fields in the behavioral sciences rely almost exclusively on global self-report surveys of respondents. Skeptics point to instances where self-report instruments have gone wrong and decry the numerous studies in which a bevy of self-report scales are merely correlated with one another. The proponents of the self-report technique cite the validity of self-report measures in some studies, as well as the virtue that people themselves often know information that cannot be obtained by other...

Tests That Adapt To The Test Taker

Although adapting conventional tests for computerized administration has many procedural advantages, the psychometric properties of the tests are not improved. Specifically, highly capable examinees answer easy items correctly with high probability, and weak examinees answer difficult items at near chance levels. As described in this section, the computer's dynamic capabilities can be used to selectively administer items to examinees so that the items are of appropriate difficulty for each...

Absolute Stability and Absolute Consistency

Social comparisons are not the only sources of psychological knowledge. Temporal comparisons can also provide important information about self and others (Albert, 1977). Consider this example Suppose I had asked my neighbor many times to water my plants while 1 was out of town. If my neighbor had always agreed to help, it seems reasonable to predict on the basis of his absolutely stable behavior that he will continue to water my plants in the future. Comparisons across situations can provide...

Summary And Recommendations

In this chapter we first described how CFA can be used to model multitrait by multisource matrices to determine the convergent and discriminant validity of measures in clinical psychology. Here we described the outcomes necessary for measures to have strong construct validity as well as the less-than-ideal outcomes for the construct validity of measures. We hope that our description of how to use CFA to model multitrait by multisource matrices results in more researchers using these procedures...

Variability and Discrimination

Discrimination is a core ability of living beings and adaptive for survival. Evolution has made our information processing systems sensitive to differences and changes to the point of our ability to contrast objects actively At the same time, our sensory systems adapt to, ignore, and actively inhibit invariant input. By ignoring invariant input, we maximize our capacity for more informative input (Lindsay & Norman, 1972). Attending to variation and ignoring invariance are pervasive phenomena...

Multiple Indicator CTCMl Model

A multiple-indicator extension of the CTC(M-l) is depicted in Figure 20.3c. This model is described in detail by Eid et al. (2003). In this model there is a method factor for each combination of a trait and a nonstandard method. The method factors belonging to the same method can be correlated, thus representing the generalizability of method effects across traits. Also the method factors of different methods can be correlated, thus showing whether the nonstandard methods have more in common...

Cross Cultural Evaluations of Theoretical Models The Internal External Frame of Reference

The internal-external frame of reference model describes a seemingly paradoxical pattern of relations between math and verbal self-concepts and corresponding measures of achievement (see Marsh, 1986, 1990c, 1993a Marsh, Byrne, & Shavelson, 1988 Marsh & Hau, 2003 Marsh, Kong, & Hau, 2001) and why math and verbal self-concepts are almost uncorrelated, even though corresponding areas of academic achievement are substantially correlated (typically .5 to .8, depending on how achievement is...

Individual Symptoms Outcomes Required for Construct Validity

Four outcomes are required for the individual symptom ratings to have strong construct validity. First, each symptom rating in Figure 27.1 must have a substantial loading on the appropriate trait factor (i.e., each ADHD-IN symptom rating has a substantial loading on ADHD-IN trait, each ADHD-HI symptom rating has a substantial loading on the ADHD-HI trait, and each ODD symptom rating has a substantial loading on the ODD trait). Such outcomes would provide support for the convergent validity of...

Abilities

Assessing intelligence has traditionally focused on multiple forms of test data (T data). Standard measures of intelligence typically attempt to gather information on a wide variety of traits considered to be at the core of general mental ability. However, numerous efforts have been made to move beyond traditional assessment approaches. These have included measures of specific cognitive abilities, intellectual interests, and self-report measures of intelligence. Testing cognitive abilities has...

Quasi Independence Model

A useful extension of the independence model is the quasi-independence model. In this model a new parameter is introduced that is only implemented for cells on the main diagonal, which represent agreement between methods In ev) ii + + XB. + 8.1 (i j), with This model fits the data in Table 17.3 ( 2 1.38, df 1, p .24) very well. In contrast to the independence model, this model allows higher cell frequencies for cells on the main diagonal, but no overrepresentation in any other cell. For cells...

Analyzing Emotion Abstraction Patterns TASC

Mergenthaler and his research group use text analysis to characterize key moments in psychotherapy sessions. They developed a computer program called TAS C that focuses on two language dimensions emotional tone and abstraction. According to Mer-genthaler's theory, emotion-abstraction patterns occur periodically in psychotherapy sessions with insight processes (abstraction) following emotional events (emotion) with a time lag (Mergenthaler, 1996). For the analysis of emotional tone, defined as...

Methods For Measuring Emotion

As mentioned earlier, the components of emotion can be conveniently categorized into three broad, only somewhat overlapping, output systems (after Lang, 1979) (a) language, which includes evalua tive self-reports of experience, modifications of speech patterns by emotions, and expressive communications (b) behavior, which includes overt acts and functional behavior sequences, such as flight or fight, approach, and threat behavior, as well as the modulation of other behaviors by emotion and (c)...

Limitations To The Trait Approach

Personality traits are stable, cross-situational consistencies in behavior, thought, or experience. Although personality traits could in principle be based on multiple types of data, there has been an overwhelming reliance on self-report. On trait scales, participants are asked to characterize themselves in general, that is, without regard to time frame or situational contingencies. Trait scores are reliable and valid and Preparation of this chapter was assisted by funding from NSF (9817649)...

Using Multiple Self Report Measures for Construct Validation

It can be stated with some certainty that almost all health psychology research, whether it addresses molecular or broad research questions, involves self-reports of some kind. Reasons for this dominance of self-report measures include that objective measures (e.g., physiological or observational measures) are often not easy to obtain and that for many constructs that health psychologists are interested in, no objective indictors exist. For example, no objective measure of self-efficacy is...

Emotion From A Systems Perspective

Before something can be measured it must be defined. Is there an adequate definition of emotion Although emotion researchers do not fully agree on a single, consensual definition (cf. Ekman & Davidson, 1994), researchers have long debated its facets and components. A consideration of these components provides a working definition of emotion useful in planning measurement strategies. Moreover, a working definition is appropriate because it may be refined and extended over time as new findings...

Traits And Implicit Processing Tendencies

Researchers interested in individual differences may wish to capture processes that are more dynamic than those measured by traits (Pervin, 1994). If self-reported traits were somehow exhaustive of personality functioning, we would expect implicit processing measures to correlate with self-reported traits. However, they often do not. Theoretical reasons for such dissociations are discussed next. As in many other areas of psychology, William James (1890) made significant contributions to the...

Exceptions to Our Description

We paint a slightly caricatured picture of the field to make a point. Many organizational researchers can, and do, use alternative methods to collect some of their data. For example, many studies of quitting, retirement, and absence use objective organizational reports of such behavior. Other studies of leadership and citizenship behavior, for example, use peer, subordinate, and spousal reports of such behaviors. These studies are stronger for using multisource data however, it is still the...

Life Story Narrative as a Means of Investigating Personality

Like the first three domains of personality, the use of narrative methods in multimethod research is a novel occurrence, yet has thus far been informative to the understanding of individual differences. Qualitative assessments of personality begin at the most basic level with the case study and progress to rigorously assessed structured interviews (McAdams, 1996, 1999). Qualitative data is frequently gathered in the form of open-ended questions concerning a topic of interest to the researcher....

Brain Imaging And Related Methods

Modern neuroimaging techniques enable researchers to noninvasively assess brain structure and function in humans. The knowledge gained from these techniques has led to a revolution in our understanding of brain-behavior relationships and has dramatically altered the psychological sciences. Several brain imaging techniques are currently in wide use, including computerized tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single photon emission computed...

Using a Combination of Self Reports and Other Methods for Construct Validation Purposes

The most frequent examples of studies that use different methods to assess different aspects of one and the same construct are those that combine observational or physiological measures (described in Stone & Litcher-Kelly, chap. 5, this volume and Mehl, chap. 11, this volume, respectively) with self-report measures or studies that combine proxy ratings (Neyer, chap. 4, this volume) with self-ratings in assessing a construct. The assessment of physio logical indicators or observational data...

Experimental Studies

Another method of research sometimes used to assess a variety of aspects of children's functioning is experimental studies (usually laboratory) in which some aspect of the situation is experimentally manipulated. For example, in studies of young children's regulation, investigators sometimes have manipulated the degree to which the mother is in the room or available to the child when the child is experiencing a potentially stressful situation (Diener & Mangelsdorf, 1999 Grolnick, Kurowski,...

Closing Thoughts

Although the main goal of this chapter is to show the relevance of multimethod assessment for social psychology, we have also tried to broaden the sense of what multimethod research entails in our subdiscipline. One way we have tried to achieve this broader goal is by making a general distinction between multiple methods focused mostly on the independent variable (between-method and within-method replication) and those focused mostly on the dependent variable (multimethod assessment). But these...

Applications To Multilevel Analyses

Psychophysiological measures offer a unique vantage for multilevel analysis as they index physiological processes and events that may intervene between psychological processes and health or behavioral outcomes. Because they represent the operations of integrated physiological systems rather than isolated molecular events, these measures are more proximal to psychological processes than are molecular events. This is in keeping with the corollary of proximity, and the intermediate level of...

Implications

Theories of organizational behavior posit dynamic processes. Inherent in the generation of predictions is how variables operate and interact over time. For example, the observation that conscientiousness should relate to job performance inherently posits that conscientious individuals engage in behaviors that enhance their job performance to a greater extent than nonconscientious individuals. This hypothesis is typically tested across individuals by relating trait conscientiousness to some...

Multidetermination

Both interpretations hold validity to some extent. Individuals differ in altruistic personality (Bierhoff, Klein, & Kramp, 1991) and in their need for approval (Crowne & Marlowe, 1964). Because helping is a social norm, it is likely that individual differences in helping reflect individual differences in both personality characteristics. Furthermore, altruistic personality and need for approval may not be the only determinants of helping (Montada & Bierhoff, 1990). Most psychological...

Physiological Biological Methods Reactivity and Regulation

Developmental psychologists often use physiological measures to assess variables such as stress, hormones, heart rate, and skin conductance (Berntson & Cacioppo, chap. 12, this volume). Typically, these variables are measured along with observations or self-reports of a similar construct (e.g., a stress index is taken before or while Cortisol levels are measured). In terms of their advantages, physiological and biological methods are extremely useful for measuring psychological processes...

Catching The Mind In Action Implicit Methods In Personality Research And Assessment

Robinson and Clayton Neighbors Key to implicit methods is the assumption that many of a person's most important tendencies are revealed only through performance. An intelligent person, for example, is not someone who claims to be intelligent, as such self-ratings may or may not correlate with performance measures (for a pessimistic view, see Brown & Dutton, 1995). Rather, an intelligent person is someone who can process information efficiently and reliably, even with concurrent...

Heart Rate Measures of Psychological States and Processes Multiple Determinism

There is now an extensive history of theory and research on the potential links between psychological states, autonomic regulation, and disease processes. A common measure in this literature has been heart rate. The electrical signature of the heart beat is readily recorded as the electrocardiogram (EKG) by noninvasive surface electrodes, and heart rate has been known for centuries to be sensitive to psychological states. It is theorized, for example, that decreases in heart rate are triggered...

Loneliness and Health Nonadditive Determinism

The utility of multilevel analysis to understanding psychological processes and psychosomatic relations is illustrated by our recent work on loneliness. Social isolation and loneliness are potent but little understood risk factors for broad-based morbidity and mortality (Seeman, 2000). Although loneliness has a heritable component, differences in social cognition provide a better explanation for the physiological characteristics of lonely versus nonlonely individuals than does a model based on...

Types of Web Based Methods

Web-based studies can be categorized as nonreactive Web-based methods, Web surveys, Web-based tests, and Web experiments. Nonreactive Web-based methods refer to the use and analysis of existing databases and text collec tions on the Internet (e.g., server log files or newsgroup contributions). The Internet provides an ocean of opportunities for nonreactive data collection. The sheer size of Internet corpora multiplies the specific strengths of this class of methods Non-manipulable events can be...

Why Innovate

The purpose of innovation is to improve a test and not innovate for the sake of innovation. Jodoin (2003) compared innovative item types and traditional multiple-choice items and found that although innovative item types returned more information per item than multiple-choice items, the amount of time spent by respondents was disproportionately higher than the return. Thus, innovative item types were less efficient in terms of information yield per unit of time than multiple-choice items. So,...

Extracting Word Patterns Latent Semantic Analysis

Latent Semantic Analysis (LSA Foltz et al., 1998 Landauer & Dumais, 1997 Landauer, Foltz, & Laham, 1998) is a semantic text analysis strategy and concerned with the use of words in their context. Compared to most existing semantic text analysis programs, however, LSA does not adopt the top-down strategy of specifying a semantic grammar and looking at the occurrence of S-V-0 constellations. Instead in a bottom-up manner it distills information about the semantic similarity of words by...

Multiple Research Teams as Multiple Methods

Particularly in the heyday of all-encompassing learning theories, it was common for competing research teams to challenge each other's work. Studies by one team would be critiqued, reanalyzed, replicated, or extended by another research team to evaluate predictions based on competing theoretical perspectives. This multimethod approach has also been evident in our self-concept research, in which colleagues have challenged our conclusions or we challenged the conclusions of others Marsh and...

Utilizing Multiple Methods in Successive Steps of Research Programs

Multiple methods can also be used in successive steps of a research program to gain an increasingly better understanding of a construct and its relations with other constructs. Examples for this are research programs that start out with qualitative research methods and subsequently conduct studies in which more traditional, quantitative techniques are used. For example, Johnston, Corban, and Clarke (1999) used a multimethod approach for studying adherence issues in sport and exercise....

Sequential Observational Methods

Observational methods are about measurement. Like most of the assessment methods described in other chapters in this section, they provide ways to extract scores from behavior. Thus observational methods, in common with assessment methods generally, are defined by procedures that when applied to events produce scores. Such scores are usually refined and reduced and then, often in combination with scores from other sources (thus becoming multimethod), are subjected to the sorts of statistical...

Other Forms of Momentary Data Capture

In addition to the paper and electronic diary methods described above, other methods are available for capturing momentary data in the real world, and we briefly mention them here. One method is interactive voice recording or IVR where a dedicated computer is programmed to present questions and response options to participants via telephone. This well-developed technology can be used in several ways, and one common approach is to have participants call the computer at designated times (e.g.,...

Methods Are Traits and States

The terminology introduced by Campbell and Fiske (1959) may be (mis)interpreted that methods are not traits. However, if methods are composites of the causes we want to measure and causes we consider irrelevant, they can be stable dispositions. Both in a substantive sense and on a formal level, no qualitative difference exists between traits and methods. The only difference is that, ideally, traits are single causes (altruistic personality) and methods are composites of causes (altruistic...

Toward A Processoriented View Of Personality

For quite a long time, psychologists have realized that self-reports of personality represent only one approach lurking beneath the surface of self-report are implicit tendencies related to selective attention, accessibility, categorization, and information retrieval. To the extent that one can measure these patterns, one unlocks important clues to what makes us different in our daily transactions with the environment. Self-reported traits, we believe, are not exhaustive of personality....

The Good Trait

Some traits may be more difficult to judge than others. Some traits such as extraversi n, for instance, are easy to judge, whereas others require more specific information and longer acquaintanceship (e.g., Colvin & Funder, 1991 Park & Judd, 1989 Paunonen, 1989). According to the RAM, differences between the judgability of traits may stem from their visibility, their availability and relevance, evaluative properties, and adaptive importance. A large body of research has consistently...

Reciprocal Effects Model Causal Ordering of Academic Self Concept and Achievement

A critical question in self-concept research is the causal ordering of academic self-concept and achievement. Self-concept researchers (e.g., Byrne, 1996 Marsh, 1990c, 1993a Marsh, Byrne, & Yeung, 1999) have attempted to resolve the theoretical chicken-egg debate about whether academic self-concept causes academic achievement or achievement causes academic self-concept. Byrne (1996) noted that much of the interest in the self-concept-achievement relation stemmed from the belief that academic...

Physical Traces

Our physical environment provides various sources for us to recognize behavior without observing it directly and, therefore, without being in danger of influencing what is under investigation (Kazdin, 1979). According to Webb et al. (1966, p. 36), the measurement of these natural traces might be separated into erosion and accretion measures. Erosion refers to the degree of selective wear on some material, whereas accretion measures record the deposit of materials. Natural traces may be used to...

Mode of Data Collection Paper Versus Electronic

Early versions of the ESM used pagers to signal participants to make a diary recording. Later versions used digital watches with auditory alarms and the capability of storing many (e.g., 100) preprogrammed dates and times for alarms. Some of the watches were linkable to personal computers so that stored alarm schedules could be easily downloaded. Both the pager and watch methods have inherent limitations. First, because alarm schedules are preprogrammed based on usual awakening and bedtimes,...

Comparison of Nonhierarchical Loglinear Association Models

As has been shown, the most restrictive models that fit the data are the quasi-independence model with a constrained 5 parameter and the symmetry model. Because both models are not nested, the likelihood ratio difference test cannot be conducted (see Figure 17.1). To decide which model fits best, information criteria have to be considered. Information criteria such as the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) are based on the 2 value, and they weigh the...

Basic Concepts And Fundamental Psychological Principles

Lay and scientific epistemics have much in common (Kruglanski, 1989a). Ordinary people and scientists share a desire for knowledge, use similar methods for acquiring knowledge, need knowledge for similar purposes, collect similar data, and use similar criteria for judging the usefulness of data. Lay and academic psychologists alike want to describe individuals and social situations in psychological terms. Both construct theories for the explanation of behavior and rules for its prediction. Both...

CFA Approach to MTMM Studies of Self Other Agreement

How well do self-report self-concept ratings by self agree with inferred self-concept ratings based on responses by significant others (e.g., teacher, family member, friend) This long-standing debate in self-concept research has important theoretical, substantive, and practical implications. Following Shavelson et al. (1976), Marsh (e.g., Marsh, 1990c, 1993a Marsh & Craven, 1997) stressed that inferred self-concept is a separate construct and should not be confused with self-concept ratings...

Methods Are Hypothetical Constructs

The language in which methods are often treated in the literature suggests that they are something technical, nonpsychological, or different than substantive variables. But method is a summary concept for a multitude of ways in which we obtain psychological information. The result of a method has psychological significance as does the method itself. Returning to the neighbor example, self-report, other-report, and observation were introduced as methods because the different procedures collected...

Latent Variable Models

The correlation and association methods described so far are correlations between observed variables that are usually affected by measurement error. Latent variable models are statistical approaches designed to separate measurement error from true individual differences. Moreover, latent variable models allow the definition of latent variables that represent different sources of influence on the observed variables. The advantage here is that one can model complex structures that link latent...

Prerequisite Psychometric Conditions

The use of CFA to model a multitrait by multisource matrix requires a significant amount of psychometric work on the rating scale for each source prior to this analysis. First, the items and features of the scale (e.g., wording of items, rating interval, rating anchors, and so on) must have good content validity (i.e., the representativeness and relevance of the items for the given construct Haynes, Richard, & Kubany, 1995). Second, the distributional characteristics of measures from each...

Formulating a Response

Once respondents understand a self-report question, they must formulate an answer. The processes involved in constructing an answer vary depending on the type of self-report being made. When reporting on the frequency of a specific behavior, for instance, respondents might be able to search their memory, count the number of occasions on which the behavior occurred, and report the counted value. When reporting an attitude, on the other hand, respondents must search their memory for relevant...

Confirmatory Factor Analysis Approaches

When data are continuous and there are one or more underlying factors, confirmatory factor analysis procedures may be used to test measurement invariance. Meredith (1993) considered the issue of measurement invariance across groups, and he developed a sequence of model comparisons that provide a close parallel to the IRT approach. Widaman and Reise (1997) presented a clear description of these procedures, and Meredith and Horn (2001) have recently extended this approach to testing measurement...

Convergence and Divergence Across Brain Regions

Because neuroimaging experiments provide data on multiple brain regions simultaneously, neuroimaging data need not be limited to a single entry in an MTMM matrix. Rather, different brain regions can be sampled to examine the extent to which activity converges or diverges across brain regions. For instance, in considering a measure related to attention, it may be useful to know that task perform ance correlates with activity within the frontal eye field, parietal cortex, and anterior cingulate...

Gottschalk Gleser Method of Content Analysis

Also in the 1960s, Gottschalk and his colleagues started developing what became known as the Gottschalk-Gleser Method of content analysis (Gottschalk, 1995). The Gottschalk-Gleser Method involves participants giving a 5-minute speech on a personal life experience. The verbatim transcripts are then submitted to a content analysis. Several scales tapping into what Gottschalk calls psychobiological dimensions have been developed and validated. Most of the scales are derived from a psychoanalytic...

Multimethod Item Response Theory

Item response theory (IRT) is a framework for an increasing number of statistical models that refer to the same kind of data structure. The data basis for applying IRT models is a matrix of responses of a (large) number of persons on a (small) number of questions, tasks, stimuli, or whatever, called the items. The item responses may be dichotomous (yes-no correct-incorrect true-false, etc.), ordinal (strongly disagree, disagree, agree, strongly agree), or nominal without a given order of the...

Self Report Emotion Mood and Coping

Self-report methods are commonly used in developmental psychology. However, several important issues should be noted when using self-report data. First, age is an obvious concern because most children under age 8 have difficulty completing paper-and-pencil measures. Nevertheless, as we discuss briefly in this section, methods have been constructed that appear to successfully elicit young children's reports of some constructs. Second, the validity of self-report data sometimes is a concern...

Using Multiple Methods to Assess Different Predictors of an Outcome Variable

Many classical studies in health psychology test the various health behavior models that have been proposed since the 1970s. The most prominent models include structural models such as the health belief model (Becker, 1974 Rosenstock, 1974), the theory of reasoned action planned behavior (Ajzen & Fish-bein, 1980 Fishbein & Ajzen, 1975), and the protection motivation theory (Rogers, 1983), as well as more dynamic health behavior models that describe the processes of health behavior change...

Applications

We will illustrate the five models by applying them to data from an MTMM study exploring the relations between self- and peer-rated frequency of negative emotions. The traits were ear, anger, and sadness. The three methods were self-ratings, ratings by a good friend, and ratings by an acquaintance. The sample consisted of 172 triples of self- and peer raters. This sample was a subgroup of individuals from a larger MTMM study (see Eid et al., 2003 Trierweiler, Eid, & Lischetzke, 2002). While...

Proportion Agreement Index Percentage Agreement Index

The proportion agreement index (p0), which indicates how often two observers' ratings concur, is an intuitive and useful first measure of observer agreement. It is computed by dividing the number of times raters agree by the number of objects rated Jiy denotes the number of cases in the cell ij of the cross-classified table, and nu represents the cells on the main diagonal (where i j), which indicate concordant ratings. The same information is provided by the percentage agreement index (p ),...

Analysis Of Facet Data

Facet design is part of a more general approach called facet theory, which uses the facet structure to generate hypotheses about similarities between items. Facet theory relies almost exclusively on producing low-dimensional geometric representations of the data, which are then interpreted in terms of the properties of the defining facets (cf. Borg & Shye, 1995). Other approaches include confirmatory factor analysis (cf. Mellenbergh, Kelderman, Stijlen, & Zondag, 1979). A problem with...

Reporting a Response

The final step in making a self-reported judgment is to communicate that judgment to the investigator. Anything that impedes the accuracy of this communication will affect the validity of the report. For instance, when reporting on illegal or other socially undesirable behaviors, participants may simply decide not tell the truth. Alternatively, when asked to respond using a Likert response scale, respondents may attempt to provide an accurate response, but different respondents may use the...

Experimental Methods Of Psychological Assessment

What can be gained from applying methods of experimental psychology to problems of psychological assessment Experimental psychology and psychological assessment, although both being important branches of psychology, are clearly distinct scientific disciplines with unique histories, characterized by discipline-specific theories, paradigms, and research methods (see Bringmann, L ck, Miller, & Early, 1997). Consequently, there has been little overlap between research in experimental psychology...

Advantages And Disadvantages Of Selfreport Methods

Self-report methods offer clear advantages over other assessment techniques. These methods are simple, quick, inexpensive, flexible, and often provide information that would be difficult or impossible to obtain any other way Yet each advantage corresponds to specific disadvantages that may go unnoticed by researchers. For example, the ubiquity of self-report techniques results from the fact that they are so easy to administer. However, this ease of use may result in an overreliance on...

Integrating Neuroimaging Data With Other Data in a Multitrait Multimethod Framework

Assessment of psychological constructs has traditionally focused on behaviors that are either directly observable by a researcher or can be reported by the examinee. Neuroimaging can supplement these methods of assessment by providing information at a neural level. Although, one might be tempted to view this at a causal level (i.e., the brain activity causes the behavior, or the behavior causes the brain activity), it need not be viewed as such. Rather, neuroimaging data can be viewed as just...

The Weighted Average Model WAM

Whereas the SRM is a statistical model designed for the decomposition of the components of interpersonal perception, the WAM is a general theoretical model of perception sometimes applied to informant assessment to predict its qualities (e.g., consensus and self-other agreement). The WAM predicts that the qualities of informants' ratings are a weighted function of nine components 1. acquaintance (i.e., amount of information informants have about the target), 2. overlap (i.e., the number of...

Correlation and Association Models

The convergence of methods can be assessed by correlating the methods that are supposed to measure the same trait. Campbell and Fiske (1959) have extended this idea by defining an MTMM correlation matrix. In this matrix there is one indicator for each trait-method unit, and this matrix allows a thorough analysis of convergent and discriminant validity by comparing several correlation coefficients (e.g., the correlations between different methods measuring the same trait versus those between...

CFA Approach to MTMM Studies of Different Self Concept Instruments

What is the relation between scales based on responses from new and existing self-concept instruments Historically, self-concept research was plagued by a surfeit of idiosyncratic instruments that hindered communication among researchers and research syntheses. This situation invited the Jingle-Jangle Fallacy (Marsh, 1994) whereby researchers mistakenly assumed that two scales with the same label measured the same construct or that two scales with different labels measured different constructs....

Representing Observational Data

With many measurement approaches, the question, How should one represent one's data does not arise. The standard rectangular data matrix suffices. Rows represent sampling units (participants, dyads, etc.), columns represent variables, and columns are filled in with the relatively few scores generated by the measurement approach. That is all the standard statistical packages need or expect, and even a preliminary step like scoring the items of a self-esteem scale, for example, is relatively...

The Experimental Method In Psychology

From the very beginning, the experimental method has been closely tied to hypothesis testing and theory evaluation in psychological research (see Boring, 1950 Bredenkamp, 2001 Calfee, 1985 Cook & Campbell, 1979 Davis, 1995 Shadish, Cook, & Campbell, 2002). During the past 50 to 60 years in particular, experiments have been used fairly routinely for testing hypotheses from different branches of psychology. Typical examples include frustration causes aggression (Berkowitz, 1989, p. 61),...

P0xaocLfO17

In this equation, L stands for the likelihood function, and the latent trait distribution is denoted by f(d). Warm (1989) used a prior distribution for J(6), which is equal to the square root of the Fisher information function. Such a prior is called nonin-jormative because it is a constant over the latent dimension and does not contain information about the latent distribution of person abilities. By means of Warm's method, an estimator for 6 is obtained that has a finite value for persons who...

Observational Methods Parent Child Relations

Observational methods are excellent tools for the study of the complex relationships examined in developmental psychology (Bakeman & Gnisci, chap. 10, this volume). One of the key distinctions between the approach of many developmentalists and that of some other disciplines is the importance placed on context. Individuals do not exist in isolation they actively construct their environment and simultaneously are influenced by their environment. The complexity of these interactions is often...

Validity

Validity, one of the key issues of research, concerns the question whether the inferences drawn from the results of a study are true or not (Shadish, Cook, & Campbell, 2002). In particular, with respect to measurement methods, validity represents the degree to which the adequacy and appropriateness of inferences and actions based on the results of a measurement device are supported by empirical evidence and theoretical rationales (Messick, 1989). Multimethod research plays a key role in the...

Reliability Of Observational Data

The standard psychometric concerns of reliability and validity are in no way unique to observational methods. The precision and accuracy of any measuring device needs to be established before weight can be given to the data collected with it. Such considerations apply no matter whether observational methods or other measuring approaches are used. Nonetheless, for some measuring approaches, reliability issues do not loom large. For example, usually we assume that, once calibrated,...

Franz J Ney er

Knowledgeable informants are frequently employed as data-gathering instruments in all domains of research in psychology. Informant assessments correspond with one of the basic data types known in psychology, which has been called L-data (i.e., life data recorded by observers) by Cattell (1957), 0-data (i.e., data generated by observers) by Block (1977), or l-data (i.e., data derived from informants) by Funder (2004). Informants are people who usually share some brief history with a studied...

CFA Approaches to MTMM Data

Using CFA approaches to MTMM data, researchers can define models that posit a priori trait and method factors and test the ability of such models to fit the data. In the general MTMM model (Marsh, 1989 Marsh & Grayson, 1995 Widaman, 1985) (a) there are at least three traits (T 3) and three methods (M 3) (b) T X M measured variables are used to infer T + Ma priori factors (c) each measured variable loads on one trait factor and one method factor but is constrained so as not to load on any...

Structural Equation Models For Multitraitmultimethod Data

Michael Eid, Tanja Lischetzke, and Fridtjof W Nussbeck Models of confirmatory factor analysis (CFA) or structural equation modeling (SEM) have generally become the most often applied methodological approaches besides Campbell and Fiske's traditional approach of inspecting correlation matrices (e.g., Eid, 2000 Eid, Lischetzke, Nussbeck, & Trierweiler, 2003 Kenny, 1976, 1979 Marsh, 1989 Marsh & Grayson, 1995 Saris & van Meurs, 1991 Widaman, 1985). This is mainly due to the fact that SEM...

Characteristics Of Sequential Observational Methods

As we define matters here, coding schemes are a central defining characteristic of sequential observational methods. Sometimes it useful to use the phrase systematic observation to distinguish the sorts of methods we are talking about from simply looking at behavior or producing narrative, journalistic reports. Then a brief definition of systematic observation might be the application of predefined coding schemes to sequences of live or recorded behavior (or transcripts of behavior) based on...

Examples of Multimethod Assessment Interpersonal Conflict and Aggression

William Graziano and his colleagues have conducted a program of research that also illustrates well the strengths and advantages of a multimethod approach in social psychology. We chose this particular study (Graziano, Jensen-Campbell, & Hair, 1996) to describe in some detail because it is a good example of all three forms of multimethod research multi-method assessment of the dependent variables, between-method replication of the independent variable, and...

Personality Is a Multilevel Phenomenon

A key component of our neosocioanalytic perspective on personality is that the domains of traits, motives, abilities, and narratives can be differentiated in hierarchical terms (see Hooker, 2002 Hooker & McAdams, 2003 Mayer, 1995 Roberts & Pomerantz, in press). For example, at the broadest level of the trait domain one finds the personality traits found in standard omnibus personality inventories. These are often the traits that make up the now ubiquitous measures of the Big Five. The...

Examining Stability Autoregressive Models

Autoregressive models are used to examine the stability of the relative standing of individuals over time. Figure 21.1 illustrates an autoregressive model for a three-wave data set. In this data set (Biesanz, West, & Millevoi, 2004), 188 college students were assessed at weekly intervals on a measure of the personality trait of conscientiousness (Saucier & Ostendorf, 1999). According to Saucier and Ostendorf, conscientiousness is comprised of four closely related facets orderliness,...