Multiple Research Teams as Multiple Methods

Particularly in the heyday of all-encompassing learning theories, it was common for competing research teams to challenge each other's work. Studies by one team would be critiqued, reanalyzed, replicated, or extended by another research team to evaluate predictions based on competing theoretical perspectives. This multimethod approach has also been evident in our self-concept research, in which colleagues have challenged our conclusions or we challenged the conclusions of others Marsh and...

Utilizing Multiple Methods in Successive Steps of Research Programs

Multiple methods can also be used in successive steps of a research program to gain an increasingly better understanding of a construct and its relations with other constructs. Examples for this are research programs that start out with qualitative research methods and subsequently conduct studies in which more traditional, quantitative techniques are used. For example, Johnston, Corban, and Clarke (1999) used a multimethod approach for studying adherence issues in sport and exercise....

Simultaneous Longitudinal and Measurement Modeling

Structural equation modeling permits simultaneous modeling of the measurement structure and the longitudinal model of stability or change. In the measurement portion of the model, each latent construct is hypothesized to be error free and normally distributed on an interval scale. The structural part represents the relationships between the latent constructs. This modeling approach can also be extended to two or more ordered categories (Muthen, 1984). This approach assumes that each dichotomous...

Sequential Observational Methods

Observational methods are about measurement. Like most of the assessment methods described in other chapters in this section, they provide ways to extract scores from behavior. Thus observational methods, in common with assessment methods generally, are defined by procedures that when applied to events produce scores. Such scores are usually refined and reduced and then, often in combination with scores from other sources (thus becoming multimethod), are subjected to the sorts of statistical...

Other Forms of Momentary Data Capture

In addition to the paper and electronic diary methods described above, other methods are available for capturing momentary data in the real world, and we briefly mention them here. One method is interactive voice recording or IVR where a dedicated computer is programmed to present questions and response options to participants via telephone. This well-developed technology can be used in several ways, and one common approach is to have participants call the computer at designated times (e.g.,...

Methods Are Traits and States

The terminology introduced by Campbell and Fiske (1959) may be (mis)interpreted that methods are not traits. However, if methods are composites of the causes we want to measure and causes we consider irrelevant, they can be stable dispositions. Both in a substantive sense and on a formal level, no qualitative difference exists between traits and methods. The only difference is that, ideally, traits are single causes (altruistic personality) and methods are composites of causes (altruistic...

Info

Fear, anger, sadness Trait factors. Method factors are denoted by SR (self-report), FR (friend report), AR (acquaintance report). Covariances are depicted in the upper (right) triangular, variances appear in the main diagonal, and correlations are presented in the lower right portion of the table. Empty cells indicate nonadmissible covariances or correlations. Parameters that differ significantly from 0 (a .05) appear in boldface type. traits. These differences in correlations, however,...

Toward A Processoriented View Of Personality

For quite a long time, psychologists have realized that self-reports of personality represent only one approach lurking beneath the surface of self-report are implicit tendencies related to selective attention, accessibility, categorization, and information retrieval. To the extent that one can measure these patterns, one unlocks important clues to what makes us different in our daily transactions with the environment. Self-reported traits, we believe, are not exhaustive of personality....

There Are Dramatic Advances In The Statistics Of Measurement

Most researchers were exposed to classical test theory during their training and so understand meas ures in terms of ideas related to reliability and validity. Oftentimes they have memorized some types of reliability and several types of validity. This simple approach to measurement can aid an investigator to proceed with his or her work, but is woefully inadequate compared to the psychometric sophistication we now possess. Some investigators have also been exposed to generalizability theory,...

The Good Trait

Some traits may be more difficult to judge than others. Some traits such as extraversi n, for instance, are easy to judge, whereas others require more specific information and longer acquaintanceship (e.g., Colvin & Funder, 1991 Park & Judd, 1989 Paunonen, 1989). According to the RAM, differences between the judgability of traits may stem from their visibility, their availability and relevance, evaluative properties, and adaptive importance. A large body of research has consistently...

The Coalescence of Statistical Models

Several statistical models have been described so far as distinct families of approaches that makes the flexible modeling of multimethod data possible. Although the distinctness helps us understand the peculiarities of each method, it conceals the close relationships between the methods. For example, IRT models can also be formulated as multilevel models for categorical variables (Rijmen, Tuer-lincks, de Boeck, & Kuppens, 2003). Moreover, an increasing endeavor by psychometric researchers...

Techniques Against Common Errors and Problems

Every coin has two sides, and so the great advantage of revealing assessment materials to a large worldwide audience via the Internet also means that the collected information may be accessible for many people. There is evidence that confidential data is often openly accessible (an estimate runs at 25 -33 , and this is a cause for concern) because of configuration errors on the part of the researcher that can be easily made in certain operating systems (Reips, 2002b). Several measures help...

Reciprocal Effects Model Causal Ordering of Academic Self Concept and Achievement

A critical question in self-concept research is the causal ordering of academic self-concept and achievement. Self-concept researchers (e.g., Byrne, 1996 Marsh, 1990c, 1993a Marsh, Byrne, & Yeung, 1999) have attempted to resolve the theoretical chicken-egg debate about whether academic self-concept causes academic achievement or achievement causes academic self-concept. Byrne (1996) noted that much of the interest in the self-concept-achievement relation stemmed from the belief that academic...

Physical Traces

Our physical environment provides various sources for us to recognize behavior without observing it directly and, therefore, without being in danger of influencing what is under investigation (Kazdin, 1979). According to Webb et al. (1966, p. 36), the measurement of these natural traces might be separated into erosion and accretion measures. Erosion refers to the degree of selective wear on some material, whereas accretion measures record the deposit of materials. Natural traces may be used to...

Multivariate Multilevel Measurement

Raudenbush et al. (1991) extended the measurement model by combining items from several different scales in one analysis. The constant in the multilevel model is then replaced by a set of dummy variables that indicate to which scale each item belongs. This is similar to a confirmative factor analysis, but with the restriction that the loadings of all items that belong to the same scale are equal and that there is one common error variance. These are strong restrictions, and multilevel...

Mode of Data Collection Paper Versus Electronic

Early versions of the ESM used pagers to signal participants to make a diary recording. Later versions used digital watches with auditory alarms and the capability of storing many (e.g., 100) preprogrammed dates and times for alarms. Some of the watches were linkable to personal computers so that stored alarm schedules could be easily downloaded. Both the pager and watch methods have inherent limitations. First, because alarm schedules are preprogrammed based on usual awakening and bedtimes,...

Method Variance And Neuroimaging

Neuroimaging researchers have often highlighted sources of method variance associated with specific technical steps in neuroimaging, such as the effects of using different techniques for spatial normalization, movement correction, or modeling the hemo dynamic response. In contrast, because neuroimaging data has usually been collected in relative isolation, much less attention has been paid to overall sources of method variance when attempting to include neuroimaging data as part of a larger...

Major Distinctions Of Methodological Approaches For Analyzing Multimethod Data

It is important to understand the distinctions among methods before choosing an approach to any research question under consideration. To this end, I will discuss five general distinctions among methodological approaches single versus multiple trait methods, single-indicator versus multiple-indicator methods, interchangeable versus structurally different methods, temporal versus nontemporal methods, and metrical versus categorical data methods. Methods for Measuring Single Versus Multiple...

Is Verbal Data Psychometrically Good Data

There might be many good reasons to use text data for psychological research. From a measurement perspective, one of the most important questions concerns the extent to which verbal data is psychometrically good data. Unfortunately, it is common for text analysis researchers, after developing a new method, to proceed to its application without establishing its psychometric properties. Thorough construct validation in the area of text analysis is yet rare. However, at least two notable...

Generalized Rasch Models for Multimethod Data

The multimethod Rasch model introduced in the previous section can be considered to be a main effects model in the terminology of analysis of variance. Because there are three factors persons, items, and methods in this main effects model, three first-order interactions can be distinguished interactions between methods and items, between persons and methods, and between items and persons. The last-mentioned interaction, however, would contradict the principle of Rasch models to separate the...

Fmri Experimental Design And Analysis

Because images can be collected at the level of seconds (TR for a single slice can be < 1 s, whole brain coverage < 3 s), it becomes possible to collect hundreds of images consecutively, with the primary limiting factors being subject to fatigue or movement over time, or hardware processing constraints. This allows a wide range of study designs. The prototypical fMRI experimental design involves a boxcar in which two behavioral tasks alternate over the course of a scanning session, and the...

Ethical Limits

In highly nonreactive research, the subjects or participants are not aware of being the objects of research. They are not aware of person(s) or installations recording data related to themselves. They may not be aware that others in their proximity or those who are able to observe them by using technical equipment or reading the traces of behavior are in the positions of scientist, experimenter, or confederate. Consequently, they are unable to refuse their participation. This fact has been...

Criterion Validity

Robinson, Solberg, et al. (2003) suggested that criterion-related correlations for implicit tests are often surprisingly high given reliability concerns by contrast, those for explicit tests are often fairly disappointing given their internal consistency. This conclusion was based on prior knowledge concerning the validity of implicit and self-report tests (Bornstein, 1999), as well as findings from the investigation at hand (Robinson, Solberg, et al., 2003). Bornstein (1999), for example, has...

Complementary Qualitative Methods

Thus far we have described diverse quantitative methods that have been highly effective in addressing key issues relevant to the self-concept and which have been critical to the conduct of our overall research program. However, we have not yet addressed the potentially illuminating role of qualitative methods in combination with quantitative research. We argue that qualitative methods provide complementary, nonoverlapping advantages beyond those that quantitative methods can provide (and vice...

Comparison of Nonhierarchical Loglinear Association Models

As has been shown, the most restrictive models that fit the data are the quasi-independence model with a constrained 5 parameter and the symmetry model. Because both models are not nested, the likelihood ratio difference test cannot be conducted (see Figure 17.1). To decide which model fits best, information criteria have to be considered. Information criteria such as the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) are based on the 2 value, and they weigh the...

Basic Concepts And Fundamental Psychological Principles

Lay and scientific epistemics have much in common (Kruglanski, 1989a). Ordinary people and scientists share a desire for knowledge, use similar methods for acquiring knowledge, need knowledge for similar purposes, collect similar data, and use similar criteria for judging the usefulness of data. Lay and academic psychologists alike want to describe individuals and social situations in psychological terms. Both construct theories for the explanation of behavior and rules for its prediction. Both...

CFA Approach to MTMM Studies of Self Other Agreement

How well do self-report self-concept ratings by self agree with inferred self-concept ratings based on responses by significant others (e.g., teacher, family member, friend) This long-standing debate in self-concept research has important theoretical, substantive, and practical implications. Following Shavelson et al. (1976), Marsh (e.g., Marsh, 1990c, 1993a Marsh & Craven, 1997) stressed that inferred self-concept is a separate construct and should not be confused with self-concept ratings...

Methods Are Hypothetical Constructs

The language in which methods are often treated in the literature suggests that they are something technical, nonpsychological, or different than substantive variables. But method is a summary concept for a multitude of ways in which we obtain psychological information. The result of a method has psychological significance as does the method itself. Returning to the neighbor example, self-report, other-report, and observation were introduced as methods because the different procedures collected...

Latent Variable Models

The correlation and association methods described so far are correlations between observed variables that are usually affected by measurement error. Latent variable models are statistical approaches designed to separate measurement error from true individual differences. Moreover, latent variable models allow the definition of latent variables that represent different sources of influence on the observed variables. The advantage here is that one can model complex structures that link latent...

Prerequisite Psychometric Conditions

The use of CFA to model a multitrait by multisource matrix requires a significant amount of psychometric work on the rating scale for each source prior to this analysis. First, the items and features of the scale (e.g., wording of items, rating interval, rating anchors, and so on) must have good content validity (i.e., the representativeness and relevance of the items for the given construct Haynes, Richard, & Kubany, 1995). Second, the distributional characteristics of measures from each...

Formulating a Response

Once respondents understand a self-report question, they must formulate an answer. The processes involved in constructing an answer vary depending on the type of self-report being made. When reporting on the frequency of a specific behavior, for instance, respondents might be able to search their memory, count the number of occasions on which the behavior occurred, and report the counted value. When reporting an attitude, on the other hand, respondents must search their memory for relevant...

Confirmatory Factor Analysis Approaches

When data are continuous and there are one or more underlying factors, confirmatory factor analysis procedures may be used to test measurement invariance. Meredith (1993) considered the issue of measurement invariance across groups, and he developed a sequence of model comparisons that provide a close parallel to the IRT approach. Widaman and Reise (1997) presented a clear description of these procedures, and Meredith and Horn (2001) have recently extended this approach to testing measurement...

Gottschalk Gleser Method of Content Analysis

Also in the 1960s, Gottschalk and his colleagues started developing what became known as the Gottschalk-Gleser Method of content analysis (Gottschalk, 1995). The Gottschalk-Gleser Method involves participants giving a 5-minute speech on a personal life experience. The verbatim transcripts are then submitted to a content analysis. Several scales tapping into what Gottschalk calls psychobiological dimensions have been developed and validated. Most of the scales are derived from a psychoanalytic...

Self Report Emotion Mood and Coping

Self-report methods are commonly used in developmental psychology. However, several important issues should be noted when using self-report data. First, age is an obvious concern because most children under age 8 have difficulty completing paper-and-pencil measures. Nevertheless, as we discuss briefly in this section, methods have been constructed that appear to successfully elicit young children's reports of some constructs. Second, the validity of self-report data sometimes is a concern...

Using Multiple Methods to Assess Different Predictors of an Outcome Variable

Many classical studies in health psychology test the various health behavior models that have been proposed since the 1970s. The most prominent models include structural models such as the health belief model (Becker, 1974 Rosenstock, 1974), the theory of reasoned action planned behavior (Ajzen & Fish-bein, 1980 Fishbein & Ajzen, 1975), and the protection motivation theory (Rogers, 1983), as well as more dynamic health behavior models that describe the processes of health behavior change...

Applications

We will illustrate the five models by applying them to data from an MTMM study exploring the relations between self- and peer-rated frequency of negative emotions. The traits were ear, anger, and sadness. The three methods were self-ratings, ratings by a good friend, and ratings by an acquaintance. The sample consisted of 172 triples of self- and peer raters. This sample was a subgroup of individuals from a larger MTMM study (see Eid et al., 2003 Trierweiler, Eid, & Lischetzke, 2002). While...

Proportion Agreement Index Percentage Agreement Index

The proportion agreement index (p0), which indicates how often two observers' ratings concur, is an intuitive and useful first measure of observer agreement. It is computed by dividing the number of times raters agree by the number of objects rated Jiy denotes the number of cases in the cell ij of the cross-classified table, and nu represents the cells on the main diagonal (where i j), which indicate concordant ratings. The same information is provided by the percentage agreement index (p ),...

Analysis Of Facet Data

Facet design is part of a more general approach called facet theory, which uses the facet structure to generate hypotheses about similarities between items. Facet theory relies almost exclusively on producing low-dimensional geometric representations of the data, which are then interpreted in terms of the properties of the defining facets (cf. Borg & Shye, 1995). Other approaches include confirmatory factor analysis (cf. Mellenbergh, Kelderman, Stijlen, & Zondag, 1979). A problem with...

Reporting a Response

The final step in making a self-reported judgment is to communicate that judgment to the investigator. Anything that impedes the accuracy of this communication will affect the validity of the report. For instance, when reporting on illegal or other socially undesirable behaviors, participants may simply decide not tell the truth. Alternatively, when asked to respond using a Likert response scale, respondents may attempt to provide an accurate response, but different respondents may use the...

Experimental Methods Of Psychological Assessment

What can be gained from applying methods of experimental psychology to problems of psychological assessment Experimental psychology and psychological assessment, although both being important branches of psychology, are clearly distinct scientific disciplines with unique histories, characterized by discipline-specific theories, paradigms, and research methods (see Bringmann, L ck, Miller, & Early, 1997). Consequently, there has been little overlap between research in experimental psychology...

Advantages And Disadvantages Of Selfreport Methods

Self-report methods offer clear advantages over other assessment techniques. These methods are simple, quick, inexpensive, flexible, and often provide information that would be difficult or impossible to obtain any other way Yet each advantage corresponds to specific disadvantages that may go unnoticed by researchers. For example, the ubiquity of self-report techniques results from the fact that they are so easy to administer. However, this ease of use may result in an overreliance on...

Integrating Neuroimaging Data With Other Data in a Multitrait Multimethod Framework

Assessment of psychological constructs has traditionally focused on behaviors that are either directly observable by a researcher or can be reported by the examinee. Neuroimaging can supplement these methods of assessment by providing information at a neural level. Although, one might be tempted to view this at a causal level (i.e., the brain activity causes the behavior, or the behavior causes the brain activity), it need not be viewed as such. Rather, neuroimaging data can be viewed as just...

The Weighted Average Model WAM

Whereas the SRM is a statistical model designed for the decomposition of the components of interpersonal perception, the WAM is a general theoretical model of perception sometimes applied to informant assessment to predict its qualities (e.g., consensus and self-other agreement). The WAM predicts that the qualities of informants' ratings are a weighted function of nine components 1. acquaintance (i.e., amount of information informants have about the target), 2. overlap (i.e., the number of...

Correlation and Association Models

The convergence of methods can be assessed by correlating the methods that are supposed to measure the same trait. Campbell and Fiske (1959) have extended this idea by defining an MTMM correlation matrix. In this matrix there is one indicator for each trait-method unit, and this matrix allows a thorough analysis of convergent and discriminant validity by comparing several correlation coefficients (e.g., the correlations between different methods measuring the same trait versus those between...

CFA Approach to MTMM Studies of Different Self Concept Instruments

What is the relation between scales based on responses from new and existing self-concept instruments Historically, self-concept research was plagued by a surfeit of idiosyncratic instruments that hindered communication among researchers and research syntheses. This situation invited the Jingle-Jangle Fallacy (Marsh, 1994) whereby researchers mistakenly assumed that two scales with the same label measured the same construct or that two scales with different labels measured different constructs....

Representing Observational Data

With many measurement approaches, the question, How should one represent one's data does not arise. The standard rectangular data matrix suffices. Rows represent sampling units (participants, dyads, etc.), columns represent variables, and columns are filled in with the relatively few scores generated by the measurement approach. That is all the standard statistical packages need or expect, and even a preliminary step like scoring the items of a self-esteem scale, for example, is relatively...

The Experimental Method In Psychology

From the very beginning, the experimental method has been closely tied to hypothesis testing and theory evaluation in psychological research (see Boring, 1950 Bredenkamp, 2001 Calfee, 1985 Cook & Campbell, 1979 Davis, 1995 Shadish, Cook, & Campbell, 2002). During the past 50 to 60 years in particular, experiments have been used fairly routinely for testing hypotheses from different branches of psychology. Typical examples include frustration causes aggression (Berkowitz, 1989, p. 61),...

Observational Methods Parent Child Relations

Observational methods are excellent tools for the study of the complex relationships examined in developmental psychology (Bakeman & Gnisci, chap. 10, this volume). One of the key distinctions between the approach of many developmentalists and that of some other disciplines is the importance placed on context. Individuals do not exist in isolation they actively construct their environment and simultaneously are influenced by their environment. The complexity of these interactions is often...

Validity

Validity, one of the key issues of research, concerns the question whether the inferences drawn from the results of a study are true or not (Shadish, Cook, & Campbell, 2002). In particular, with respect to measurement methods, validity represents the degree to which the adequacy and appropriateness of inferences and actions based on the results of a measurement device are supported by empirical evidence and theoretical rationales (Messick, 1989). Multimethod research plays a key role in the...

Reliability Of Observational Data

The standard psychometric concerns of reliability and validity are in no way unique to observational methods. The precision and accuracy of any measuring device needs to be established before weight can be given to the data collected with it. Such considerations apply no matter whether observational methods or other measuring approaches are used. Nonetheless, for some measuring approaches, reliability issues do not loom large. For example, usually we assume that, once calibrated,...

CFA Approaches to MTMM Data

Using CFA approaches to MTMM data, researchers can define models that posit a priori trait and method factors and test the ability of such models to fit the data. In the general MTMM model (Marsh, 1989 Marsh & Grayson, 1995 Widaman, 1985) (a) there are at least three traits (T 3) and three methods (M 3) (b) T X M measured variables are used to infer T + Ma priori factors (c) each measured variable loads on one trait factor and one method factor but is constrained so as not to load on any...

Structural Equation Models For Multitraitmultimethod Data

Michael Eid, Tanja Lischetzke, and Fridtjof W Nussbeck Models of confirmatory factor analysis (CFA) or structural equation modeling (SEM) have generally become the most often applied methodological approaches besides Campbell and Fiske's traditional approach of inspecting correlation matrices (e.g., Eid, 2000 Eid, Lischetzke, Nussbeck, & Trierweiler, 2003 Kenny, 1976, 1979 Marsh, 1989 Marsh & Grayson, 1995 Saris & van Meurs, 1991 Widaman, 1985). This is mainly due to the fact that SEM...

Characteristics Of Sequential Observational Methods

As we define matters here, coding schemes are a central defining characteristic of sequential observational methods. Sometimes it useful to use the phrase systematic observation to distinguish the sorts of methods we are talking about from simply looking at behavior or producing narrative, journalistic reports. Then a brief definition of systematic observation might be the application of predefined coding schemes to sequences of live or recorded behavior (or transcripts of behavior) based on...

Sociotechnological Changes Chance and Risk for Nonreactive Measurement

One reason why we have chosen to describe nonreactive online research in more detail is because this field has recently expanded very quickly and will soon lead to a multitude of new nonreactive research strategies that offer analyses of incomparable potency with regard to the availability of information and the efficiency of analysis. The other reason can be found in the exemplary character this type of research has on the influence that technological developments may exert on the development...

Examples of Multimethod Assessment Interpersonal Conflict and Aggression

William Graziano and his colleagues have conducted a program of research that also illustrates well the strengths and advantages of a multimethod approach in social psychology. We chose this particular study (Graziano, Jensen-Campbell, & Hair, 1996) to describe in some detail because it is a good example of all three forms of multimethod research multi-method assessment of the dependent variables, between-method replication of the independent variable, and...

Personality Is a Multilevel Phenomenon

A key component of our neosocioanalytic perspective on personality is that the domains of traits, motives, abilities, and narratives can be differentiated in hierarchical terms (see Hooker, 2002 Hooker & McAdams, 2003 Mayer, 1995 Roberts & Pomerantz, in press). For example, at the broadest level of the trait domain one finds the personality traits found in standard omnibus personality inventories. These are often the traits that make up the now ubiquitous measures of the Big Five. The...

Examining Stability Autoregressive Models

Autoregressive models are used to examine the stability of the relative standing of individuals over time. Figure 21.1 illustrates an autoregressive model for a three-wave data set. In this data set (Biesanz, West, & Millevoi, 2004), 188 college students were assessed at weekly intervals on a measure of the personality trait of conscientiousness (Saucier & Ostendorf, 1999). According to Saucier and Ostendorf, conscientiousness is comprised of four closely related facets orderliness,...

Multicohort Multioccasion Designs to Cross Validate Developmental Trends

Marsh (1998 also see Baltes & Nesselroade, 1979) argued that multicohort-multioccasion designs provide a stronger basis for assessing developmental self-concept differences than a typical cross-sectional methodology (comparisons of different age cohorts collected on a single occasion) or a true longitudinal methodology (comparisons of responses by a single age cohort collected on multiple occasions). In particular, the juxtaposition of the age effects based on the (cross-sectional) age...

Trait Source and Error Variance

Trait (construct) variance represents the systematic variance in a specific manifest variable associated with a particular latent trait, whereas source variance represents the systematic variance in a specific manifest variable associated with a particular latent source. The error variance in a specific manifest variable involves two different aspects residual systematic variance (i.e., reliable variance not associated with trait and source factors) and nonsystematic effects (i.e., measurement...

Thematic Content Analysis

Thematic content analysis is used here as a summary label for a number of approaches that have been developed in the context of motivational psychology (Smith, 1992). Generally, these approaches have human judges identify critical thematic references in a text. Ratings are made either each time a theme occurs or as global ratings reflecting the prevalence of a theme across an entire text. In either case, the analyses are based on standardized coding systems that define a psychological construct...

Other Informants Child and Infant Temperament

Probably one of the most common forms of assessment in developmental research is the use of questionnaires or interviews that assess some aspect of a participant's functioning from someone else's perspective, such as that of a parent, teacher, or peer (Neyer, chap. 4, this volume). Paper-and-pencil measures are often used because of their efficiency. Many participants can be questioned at one time, so obtaining these reports is much less time consuming and less expensive than observational...

Multiple Methods and Personality Traits

As we noted, one of the persistent disputes in personality psychology is between those who believe that self-reports or observer methods should hold priority in the field. The programmatic efforts of David Funder and his colleagues demonstrate that multiple methods bring multiple perspectives to our efforts to understand the behavioral manifestation of personality traits. For example, people judging the behaviors of others perceive different cues as more relevant to personality than the...

Evaluating the Constructs Underlying Ability Achievement and Other Tests of Cognitive Competencies

It is important to appreciate that the vertex of the hierarchical model (derived factor analytically) and 2To be sure, other frameworks have been proposed, involving emotional, moral, multiple, and practical intelligence but these have yet to generate meaningful empirical advances beyond what conventional cognitive ability assessments afford (cf. Brody, 2003 Gottfredson, 2003b Hunt, 1999 Lubinski & Benbow, 1995). Messick (1992) in particular has skillfully demonstrated that many of these...

Generalizability Theory GT

Generalizability Theory (Cronbach, Rajaratnam, & Gleser, 1963 Cronbach et al., 1972 Gleser, Cronbach, & Rajaratnam, 1965 Shavelson & Webb, 1991) combines Brunswik's request for representative multifacet designs with the true-score model of Classical Test Theory (CTT Lord & Novick, 1968 Spearman, 1910). Like CTT, GT assumes that each person (or other object of-measurement) has a true score on the measured attribute. In GT, this score is called the universe score. Whereas CTT treats...

The Criterion Problem Of Accurate Informant Assessment

In terms of Brunswik's lens model, accurate perception is characterized by the convergence of cue validity and cue utilization. When researchers examine the accuracy of informant assessment, they inevitably face the criterion problem. Kruglanski (1989b) discussed three distinct notions of accuracy criteria used throughout the literature (i.e., the correspondence between a judgment and one or more independent indicators of the psychological construct, interpersonal consensus, and pragmatic...

Nonreactive Methods In Psychological Research

Participants in social science research usually think. That is, they interpret the actions of a researcher and relate it to their own beliefs, emotions, and intentions. Accordingly, the behavior they exhibit during investigations is controlled by what they think is appropriate, depending on their interpretation of the study situation and their motivation to comply with these assumed requirements. Hence, what is often measured is not a natural state but the participant's intentional...

Latent Trait Effects Outcomes Required for Construct Validity

In the first example Figure 27.1 , the manifest variables were the individual symptom ratings. More typically, however, these procedures use summary scores as manifest variables e.g., the summary score for the nine ADHD-IN, nine ADHD-HI, and eight ODD symptoms for the four sources , the focus of our second example. Figure 27.2 shows the model for the summary scores. This model involves three latent trait factors ADHD-IN, ADHD-HI, and ODD and four latent FIGURE 27.2. Heuristic representation of...

Advantages Of Webbased Methods

One of the principal reasons why Web-based methods are so popular is the fundamental asymmetry of accessibility What is programmed to be accessible from any Internet-connected place in the world will surely also be accessible in a university laboratory but what is programmed to work locally may most likely not be accessible anywhere else. A laboratory experiment, for instance, cannot simply be turned into a Web experiment by connecting the host computer to the Internet. But any Web experiment...

The Realistic Accuracy Model RAM

Realistic Accuracy Model

The realistic accuracy model RAM by Funder 1995 begins with the premise that personality traits are real and observable. As a consequence, the RAM assumes that informants reach consensus not because they share similar meaning systems or because of overlap, but rather because their judgments about a target's personality are at least partly accurate. According to the RAM, the path between a target's personality and the accurate informant judgment can be described in four steps, each associated...

Convergent and Discriminant Validity

Given the lack of knowledge about the true attributes of objects, convergence across different methods for the same attribute is often the best alternative. This type of validity has been called convergent validity Campbell amp Fiske, 1959 . Demonstrating convergent validity is not sufficient, however, because convergence alone does not yet guarantee that the methods measure what they should measure. It only shows that the methods measure the same factors. As previously outlined, some or even...

Multipleindicator Models

In contrast to single-indicator models, multiple-indicator models are able to separate measurement error from trait-specific method influences. Moreover, the hypothesis that method effects are trait specific and do not perfectly generalize across traits can be statistically tested. We will only describe three multiple-indicator extensions a general model that is able to estimate the latent correlations between different trait-method units, a model that is related to the CTCU model, and a model...

Analysis of Variance Generalizability Theory and Multilevel Modeling

The application of analysis of variance ANOVA models has a long tradition in multimethod research. To analyze the conver gence of several methods measuring the same trait, ANOVA models are routinely applied Mill-sap, 1995b Tinsley amp Weiss, 2000 . In general, two types of factors can be considered in ANOVA models random and fixed factors. Random factors are considered when the levels of a factor are a random sample from a population and the research goal is to generalize...

Linguistic Inquiry and Word Count

Linguistic Inquiry and Word Count LIWC Pennebaker et al., 2001 was originally developed in the context of Pennebaker's work on emotional writing. It was designed to reveal aspects of writing about negative life experiences that predict subsequent health improvements Pennebaker amp Francis, 1996 Pennebaker, Mayne, amp Francis, 1997 . More recently LIWC has been used to analyze language use in a wide variety of text sources including literature, personal narratives, press conferences, and...

Reducing And Analyzing Observational Data

In contrast with both self-report or questionnaire methods, and more similar with automatic collection of physiological data, observational methods often result in voluminous data. Thus data reduction is often a necessary prelude to analysis. A useful strategy is to collect slightly more detailed data than one intends to examine, thus initial data reduction may consist of combining some codes. Other data reduction may involve computation of conceptually targeted indices e.g., an index of the...

Betweenmethod Replication

When social psychologists think of multiple methods, we are perhaps just as likely to focus on the independent variable as the dependent variable. There is clear virtue even in an exact or strict replication of a finding e.g., Hendrick, 1990 , but social psychologists prefer a conceptual replication, using a different operationalization of the independent variable e.g., Gerard amp Mathewson, 1966 Pratka-nis, Greenwald, Leippe, amp Baumgardner, 1988 . If alternative manipulations replicate the...

Emotion Assessed Through Language

Self-report measures, where participants provide an evaluation of their emotional experience, form the most diverse yet most widely used set of assessment tools for measuring emotion Larsen amp Fredrickson, 1999 . Measures range from rating scales and adjective checklists, to analog scales and real-time rating dials. Proponents of self-report measures e.g., Baldwin, 2000 assume that participants are in a privileged position to monitor, assess, and integrate...

Emotion Assessed Through Physiology

Emotion output that can be assessed with physiological methods can be divided into two categories the somatic changes and changes reflecting autonomic or central nervous system activity. The somatic changes most useful to emotion researchers concern muscle movements associated with emotional expression, particularly those somatic changes on the face. Measures of somatic change. One useful measurement strategy is to have an observer rate how much emotion a target participant appears to be...

Emotion Assessed Through Behavior

Behaviors that are linked to emotions range from the very simple, such as defensive reflex actions, to the complex, such as sequences of action tendencies. Emotions likely evolved to produce adaptive actions, such as to approach desired objects or to withdraw from dangerous objects, as well as to support more flexible action tendencies associated with survival. Researchers may take advantage of these behavioral outputs to estimate emotions. Behavior action tendencies. One behavioral...

The Organization Of Cognitive Abilities

There are literally hundreds of ability tests Carroll, 1993 Cattell, 1971 Jensen, 1980, 1998 Sternberg, 1994 ,1 and a framework is needed to organize them. Over the years, proposals have been made to Support for this article was provided by a Templeton Award for Positive Psychology, a NICHD Grant P30HD15052 to the John E Kennedy Center at Vanderbilt University, and a Cattell Sabbatical Award. Earlier versions of this manuscript profited from many excellent suggestions by Camilla P Ben-bow....

Introduction To The Basic Item Response Models

Item response theory IRT see Baker, 1992 Hamble-ton amp Swaminathan, 1989 usually is considered a class of statistical models for categorical data to which the Rasch model RM and the two-parameter logistic model belong, but not latent class analysis LCA . However, there are at least two reasons why LCA can and should be seen as an IRT model. Whereas the Rasch model tries to measure a latent trait of the persons, LCA tries to identify latent classes of persons. Both measure a latent variable,...

Some Alternative Metrics

In an ideal world of construct validity, we would collect data whose assessment methods share a minimum of facets e.g., structured vs. semistruc- tured vs. unstructured, paper-and-pencil vs. observations by others, verbal vs. behavioral, accessible vs. inferred by observers, transparent vs. masked with other assessments of these constructs. As this book details, other areas of psychology have developed metrics that could and should be applied to I O research. These other assessments could...

Temporal Duration Emotions as States Emotions as Traits

Emotions are dynamic processes that take place over time often rapidly, but sometimes gradually involving a cascade of different response systems, such as those mentioned earlier. Each response system has its own dynamic and duration. For example, the central nervous system is very fast with negative stimuli producing changes at around milliseconds Smith, Cacioppo, Larsen, amp Chartrand, 2003 , the cardiac system somewhat slower with changes taking place on a beat-by-beat basis , the skin...

Weintraubs Analysis of Verbal Behavior

Weintraub's 1981, 1989 work on verbal mannerisms was inspired by the clinical observation that individuals speaking under stress often reveal important information about their psychological adjustment. Drawing on his medical training and practice, Weintraub argued that psychological defense mechanisms manifest themselves in speech patterns obtained under mildly stressful conditions. He assessed these defense mechanisms from the language that participants spontaneously use when they talk for 10...

Wextor

First, we use WEXTOR Reips amp Neuhaus, 2002 , a Web service, to create, store, and visualize experimental designs and procedures for experiments on the Web and in the laboratory. WEXTOR dynamically creates the customized Web pages needed for the experimental procedure. It supports complete and incomplete factorial designs with between-subjects, within-subjects, and quasi-experimental natural factors, as well as mixed designs. It implements client-side, response time measurement and contains a...

Text Analysis Example From Social Psychology

How do people respond to physical symptoms, and what makes them decide whether or not to seek treatment The methodological toolbox in psychology is large, and there are a number of potential ways to address this question. Yet the default strategy has been to rely on just one tool the questionnaire. In this case, for example, a researcher might create a health-decision questionnaire consisting of a number of Likert-scaled items, such as How serious do your symptoms have to be before you see a...

How Far Can We Go With Counting Words

Given that word count-based measures possess rather good psychometric properties, how far can we go with counting words Frequently researchers voice their scientific disdain for text analysis programs that are unable to distinguish between sentences as simple as the dog bit the man and the man bit the dog Hart, 2001, p. 53 . Its blindness to context makes word-count approaches sometimes appear painfully dumb. Not only are they unable to pick up irony or sarcasm e.g., Thanks a lot, accompanied...

Understanding the Question

When an individual responds to a self-report measure, he or she must first make sense of the question being asked Schwarz, 1999 Tourangeau et al., 2000 . To do this, the respondent must understand the literal meaning of the question, and anything that impedes this understanding e.g., vague or unfamiliar words, complicated sentence structure will undermine the quality of the self-report measure. Psychological assessment and survey methodology textbooks suggest that to avoid misunderstandings,...

Multilevel Models for Generalizability Analysis

Generalizability theory can be viewed as a special case of multilevel analysis. In the one-facet nested design, the nesting structure is clear The items are nested in the persons. In the one-facet crossed design, the nesting structure is arbitrary Items can be seen as nested in persons or persons nested in items. Both specifications lead to the same results. Because of the analogy with the nested design, we will use the specification structure of items in persons. In a two-facet design, the...

In Search Of Depressogenic Thought Processes

A variety of studies using a variety of cognitive methods have sought to discover the depressogenic thought processes responsible for depression for reviews, see MacLeod amp Mathews, 1994 Segal, 1988 Segal amp Ingram, 1994 . However, one problematic result emerges from this research. Specifically, it is extremely difficult to distinguish formerly depressed participants from never depressed participants on measures of cognitive bias e.g., Segal amp Ingram, 1994 . That is, depressive biases in...

Analysis of Artistic Change Martindales Regressive Imagery Dictionary

To identify regularities underlying changes in artistic work over time, Martindale 1990 developed a word count program that is based on the Regressive Imagery Dictionary. Martindale's 1990 theorizing starts from the observation that artistic work shows a steady increase in complexity over time. He explains this increase by drawing on two funda mental psychological processes humans' preference for medium levels of arousal and hence moderately complex sensory input and the physiological mechanism...

Campbell and Fiske

Brunswik Lens Model

Multimethod thinking in psychological assessment was influenced most strongly by the seminal paper of Campbell and Fiske 1959 . No other publication so importantly shaped researchers' awareness of the crucial role multimethod designs play in the construction and validation of measurement instruments Shrout amp Fiske, 1995 . Although Campbell and Fiske 1959 did not make reference to Brunswik's work, their proposals were guided by similar insights and ideas. Campbell and Fiske 1959 introduced the...