The Experimental Method In Psychology

From the very beginning, the experimental method has been closely tied to hypothesis testing and theory evaluation in psychological research (see Boring, 1950 Bredenkamp, 2001 Calfee, 1985 Cook & Campbell, 1979 Davis, 1995 Shadish, Cook, & Campbell, 2002). During the past 50 to 60 years in particular, experiments have been used fairly routinely for testing hypotheses from different branches of psychology. Typical examples include frustration causes aggression (Berkowitz, 1989, p. 61),...

P0xaocLfO17

In this equation, L stands for the likelihood function, and the latent trait distribution is denoted by f(d). Warm (1989) used a prior distribution for J(6), which is equal to the square root of the Fisher information function. Such a prior is called nonin-jormative because it is a constant over the latent dimension and does not contain information about the latent distribution of person abilities. By means of Warm's method, an estimator for 6 is obtained that has a finite value for persons who...

Observational Methods Parent Child Relations

Observational methods are excellent tools for the study of the complex relationships examined in developmental psychology (Bakeman & Gnisci, chap. 10, this volume). One of the key distinctions between the approach of many developmentalists and that of some other disciplines is the importance placed on context. Individuals do not exist in isolation they actively construct their environment and simultaneously are influenced by their environment. The complexity of these interactions is often...

Validity

Validity, one of the key issues of research, concerns the question whether the inferences drawn from the results of a study are true or not (Shadish, Cook, & Campbell, 2002). In particular, with respect to measurement methods, validity represents the degree to which the adequacy and appropriateness of inferences and actions based on the results of a measurement device are supported by empirical evidence and theoretical rationales (Messick, 1989). Multimethod research plays a key role in the...

Reliability Of Observational Data

The standard psychometric concerns of reliability and validity are in no way unique to observational methods. The precision and accuracy of any measuring device needs to be established before weight can be given to the data collected with it. Such considerations apply no matter whether observational methods or other measuring approaches are used. Nonetheless, for some measuring approaches, reliability issues do not loom large. For example, usually we assume that, once calibrated,...

Franz J Ney er

Knowledgeable informants are frequently employed as data-gathering instruments in all domains of research in psychology. Informant assessments correspond with one of the basic data types known in psychology, which has been called L-data (i.e., life data recorded by observers) by Cattell (1957), 0-data (i.e., data generated by observers) by Block (1977), or l-data (i.e., data derived from informants) by Funder (2004). Informants are people who usually share some brief history with a studied...

CFA Approaches to MTMM Data

Using CFA approaches to MTMM data, researchers can define models that posit a priori trait and method factors and test the ability of such models to fit the data. In the general MTMM model (Marsh, 1989 Marsh & Grayson, 1995 Widaman, 1985) (a) there are at least three traits (T 3) and three methods (M 3) (b) T X M measured variables are used to infer T + Ma priori factors (c) each measured variable loads on one trait factor and one method factor but is constrained so as not to load on any...

Structural Equation Models For Multitraitmultimethod Data

Michael Eid, Tanja Lischetzke, and Fridtjof W Nussbeck Models of confirmatory factor analysis (CFA) or structural equation modeling (SEM) have generally become the most often applied methodological approaches besides Campbell and Fiske's traditional approach of inspecting correlation matrices (e.g., Eid, 2000 Eid, Lischetzke, Nussbeck, & Trierweiler, 2003 Kenny, 1976, 1979 Marsh, 1989 Marsh & Grayson, 1995 Saris & van Meurs, 1991 Widaman, 1985). This is mainly due to the fact that SEM...

Characteristics Of Sequential Observational Methods

As we define matters here, coding schemes are a central defining characteristic of sequential observational methods. Sometimes it useful to use the phrase systematic observation to distinguish the sorts of methods we are talking about from simply looking at behavior or producing narrative, journalistic reports. Then a brief definition of systematic observation might be the application of predefined coding schemes to sequences of live or recorded behavior (or transcripts of behavior) based on...

Sociotechnological Changes Chance and Risk for Nonreactive Measurement

One reason why we have chosen to describe nonreactive online research in more detail is because this field has recently expanded very quickly and will soon lead to a multitude of new nonreactive research strategies that offer analyses of incomparable potency with regard to the availability of information and the efficiency of analysis. The other reason can be found in the exemplary character this type of research has on the influence that technological developments may exert on the development...

Examples of Multimethod Assessment Interpersonal Conflict and Aggression

William Graziano and his colleagues have conducted a program of research that also illustrates well the strengths and advantages of a multimethod approach in social psychology. We chose this particular study (Graziano, Jensen-Campbell, & Hair, 1996) to describe in some detail because it is a good example of all three forms of multimethod research multi-method assessment of the dependent variables, between-method replication of the independent variable, and...

Personality Is a Multilevel Phenomenon

A key component of our neosocioanalytic perspective on personality is that the domains of traits, motives, abilities, and narratives can be differentiated in hierarchical terms (see Hooker, 2002 Hooker & McAdams, 2003 Mayer, 1995 Roberts & Pomerantz, in press). For example, at the broadest level of the trait domain one finds the personality traits found in standard omnibus personality inventories. These are often the traits that make up the now ubiquitous measures of the Big Five. The...

Examining Stability Autoregressive Models

Autoregressive models are used to examine the stability of the relative standing of individuals over time. Figure 21.1 illustrates an autoregressive model for a three-wave data set. In this data set (Biesanz, West, & Millevoi, 2004), 188 college students were assessed at weekly intervals on a measure of the personality trait of conscientiousness (Saucier & Ostendorf, 1999). According to Saucier and Ostendorf, conscientiousness is comprised of four closely related facets orderliness,...

Multimethod Approaches In Health Psychology

Health psychology is a fairly new discipline, having emerged as a formally organized subdiscipline of psychology only in the late 1970s (Division 38, Health Psychology, of the American Psychological Association was founded in 1978). As a consequence, its boundaries are still somewhat fuzzy. Matarazzo, in 1980, defined health psychology as the aggregate of the specific educational, scientific, and professional contributions of the discipline of psychology to the promotion and maintenance of...

Multicohort Multioccasion Designs to Cross Validate Developmental Trends

Marsh (1998 also see Baltes & Nesselroade, 1979) argued that multicohort-multioccasion designs provide a stronger basis for assessing developmental self-concept differences than a typical cross-sectional methodology (comparisons of different age cohorts collected on a single occasion) or a true longitudinal methodology (comparisons of responses by a single age cohort collected on multiple occasions). In particular, the juxtaposition of the age effects based on the (cross-sectional) age...

MTMM Extensions The Multiple Indicator Approach

The Campbell-Fiske guidelines are frequently criticized for being based on correlations among observed variables rather than among latent constructs. Ironically, in the typical CFA MTMM approach, a single scale score often an average of multiple items is used to represent each trait-method combination. Marsh (1993b Marsh & Hocevar, 1988), however, argued that it is stronger to incorporate the multiple indicators explicitly into the MTMM design. When multiple indicators are used to represent...

Trait Source and Error Variance

Trait (construct) variance represents the systematic variance in a specific manifest variable associated with a particular latent trait, whereas source variance represents the systematic variance in a specific manifest variable associated with a particular latent source. The error variance in a specific manifest variable involves two different aspects residual systematic variance (i.e., reliable variance not associated with trait and source factors) and nonsystematic effects (i.e., measurement...

Thematic Content Analysis

Thematic content analysis is used here as a summary label for a number of approaches that have been developed in the context of motivational psychology (Smith, 1992). Generally, these approaches have human judges identify critical thematic references in a text. Ratings are made either each time a theme occurs or as global ratings reflecting the prevalence of a theme across an entire text. In either case, the analyses are based on standardized coding systems that define a psychological construct...

Other Informants Child and Infant Temperament

Probably one of the most common forms of assessment in developmental research is the use of questionnaires or interviews that assess some aspect of a participant's functioning from someone else's perspective, such as that of a parent, teacher, or peer (Neyer, chap. 4, this volume). Paper-and-pencil measures are often used because of their efficiency. Many participants can be questioned at one time, so obtaining these reports is much less time consuming and less expensive than observational...

Nonreactive Measures as Distinct Techniques

In the literature, the term nonreactive measurement is used in at least two senses, in a dichotomous as well as a continuous sense. Textbooks often think of nonreactive measures as representing a distinct set of procedures sharply different from reactive methods. Actually, there is a comparatively stable core of measures that are commonly subsumed under this heading. In social psychology, for instance, one of the most cited and consequently most prototypical nonreactive method is the so- called...

Multiple Methods and Personality Traits

As we noted, one of the persistent disputes in personality psychology is between those who believe that self-reports or observer methods should hold priority in the field. The programmatic efforts of David Funder and his colleagues demonstrate that multiple methods bring multiple perspectives to our efforts to understand the behavioral manifestation of personality traits. For example, people judging the behaviors of others perceive different cues as more relevant to personality than the...

Evaluating the Constructs Underlying Ability Achievement and Other Tests of Cognitive Competencies

It is important to appreciate that the vertex of the hierarchical model (derived factor analytically) and 2To be sure, other frameworks have been proposed, involving emotional, moral, multiple, and practical intelligence but these have yet to generate meaningful empirical advances beyond what conventional cognitive ability assessments afford (cf. Brody, 2003 Gottfredson, 2003b Hunt, 1999 Lubinski & Benbow, 1995). Messick (1992) in particular has skillfully demonstrated that many of these...

Generalizability Theory GT

Generalizability Theory (Cronbach, Rajaratnam, & Gleser, 1963 Cronbach et al., 1972 Gleser, Cronbach, & Rajaratnam, 1965 Shavelson & Webb, 1991) combines Brunswik's request for representative multifacet designs with the true-score model of Classical Test Theory (CTT Lord & Novick, 1968 Spearman, 1910). Like CTT, GT assumes that each person (or other object of-measurement) has a true score on the measured attribute. In GT, this score is called the universe score. Whereas CTT treats...

Summary and Evaluation

This section reviewed nine influential text analysis strategies in psychology. The selected approaches span a broad spectrum of methodological and theoretical orientations. How should a researcher decide which one to use The most immediate question is whether the options are restricted to computerized solutions or whether the burden of manual coding appears tolerable (Smith, 1992 Weintraub, 1981). Another question concerns what kind of analysis a researcher is interested in. The...

The Criterion Problem Of Accurate Informant Assessment

In terms of Brunswik's lens model, accurate perception is characterized by the convergence of cue validity and cue utilization. When researchers examine the accuracy of informant assessment, they inevitably face the criterion problem. Kruglanski (1989b) discussed three distinct notions of accuracy criteria used throughout the literature (i.e., the correspondence between a judgment and one or more independent indicators of the psychological construct, interpersonal consensus, and pragmatic...

How Are the SRM the WAM and the RAM Different

The described models, the SRM, the WAM, and the RAM, sometimes complement and sometimes compete with each other. The models are more or less based on Brunswik's approach and share the assumption that interpersonal perception should be observed in real settings. Each of them has contributed to the revival of interpersonal perception research and has yielded important, potentially useful insights when using informant assessment in research. The SRM is a statistical model that enables data to be...

Singleindicator Models

The starting point of single-indicator models is the classical MTMM matrix. However, because models of SEM are covariance structure models and the covariance matrix is more informative, SEM of MTMM data is based on the MTMM covariance matrix, not on the correlation matrix (for problems analyzing correlation matrices with SEM, see Cud-eck, 1989). In single-indicator models there is one indicator (observed variable) Yjk for each combination of a trait j and a method k. These observed variables...

Latent Trait Effects Outcomes Required for Construct Validity

In the first example Figure 27.1 , the manifest variables were the individual symptom ratings. More typically, however, these procedures use summary scores as manifest variables e.g., the summary score for the nine ADHD-IN, nine ADHD-HI, and eight ODD symptoms for the four sources , the focus of our second example. Figure 27.2 shows the model for the summary scores. This model involves three latent trait factors ADHD-IN, ADHD-HI, and ODD and four latent FIGURE 27.2. Heuristic representation of...

The Realistic Accuracy Model RAM

The realistic accuracy model RAM by Funder 1995 begins with the premise that personality traits are real and observable. As a consequence, the RAM assumes that informants reach consensus not because they share similar meaning systems or because of overlap, but rather because their judgments about a target's personality are at least partly accurate. According to the RAM, the path between a target's personality and the accurate informant judgment can be described in four steps, each associated...

P

Indicates how kappa is computed .84 in this case . Agreement expected due just to chance is subtracted from both numerator and denominator, thus kappa gives the proportion of agreement corrected for chance. Exact second-by-second agreement may be too stringent. Given human reaction time and other considerations, investigators may be willing to permit their coders some tolerance, which the GSEQ program allows. For the tallies in Figure 10.3, a 2-second tolerance was specified, thus agreements...

Convergent and Discriminant Validity

Given the lack of knowledge about the true attributes of objects, convergence across different methods for the same attribute is often the best alternative. This type of validity has been called convergent validity Campbell amp Fiske, 1959 . Demonstrating convergent validity is not sufficient, however, because convergence alone does not yet guarantee that the methods measure what they should measure. It only shows that the methods measure the same factors. As previously outlined, some or even...

Multipleindicator Models

In contrast to single-indicator models, multiple-indicator models are able to separate measurement error from trait-specific method influences. Moreover, the hypothesis that method effects are trait specific and do not perfectly generalize across traits can be statistically tested. We will only describe three multiple-indicator extensions a general model that is able to estimate the latent correlations between different trait-method units, a model that is related to the CTCU model, and a model...

Analysis of Variance Generalizability Theory and Multilevel Modeling

The application of analysis of variance ANOVA models has a long tradition in multimethod research. To analyze the conver gence of several methods measuring the same trait, ANOVA models are routinely applied Mill-sap, 1995b Tinsley amp Weiss, 2000 . In general, two types of factors can be considered in ANOVA models random and fixed factors. Random factors are considered when the levels of a factor are a random sample from a population and the research goal is to generalize...

Linguistic Inquiry and Word Count

Linguistic Inquiry and Word Count LIWC Pennebaker et al., 2001 was originally developed in the context of Pennebaker's work on emotional writing. It was designed to reveal aspects of writing about negative life experiences that predict subsequent health improvements Pennebaker amp Francis, 1996 Pennebaker, Mayne, amp Francis, 1997 . More recently LIWC has been used to analyze language use in a wide variety of text sources including literature, personal narratives, press conferences, and...

Section Ii Assessment Methods

Baird, Brendan M. CHAPTER 5 Stone, Arthur A. Litcher-Kelly, Leighann CHAPTER 6 Reips, Ulf-Dietrich Web-Based Methods pp 73-85 CHAPTER 7 Drasgow, Fritz Chuah, Siang Chee Computer-Based Testing pp 87-100 CHAPTER 8 Lubinski, David Ability Tests pp 101-114 CHAPTER 9 Robinson, Michael D. Neighbors, Catching the Mind in Action Implicit Methods in Personality Research and CHAPTER 10 Bakeman, Roger Gnisci, Augusto Sequential Observational Methods Quantitative Text Analysis...

Betweenmethod Replication

When social psychologists think of multiple methods, we are perhaps just as likely to focus on the independent variable as the dependent variable. There is clear virtue even in an exact or strict replication of a finding e.g., Hendrick, 1990 , but social psychologists prefer a conceptual replication, using a different operationalization of the independent variable e.g., Gerard amp Mathewson, 1966 Pratka-nis, Greenwald, Leippe, amp Baumgardner, 1988 . If alternative manipulations replicate the...

Emotion Assessed Through Language

Self-report measures, where participants provide an evaluation of their emotional experience, form the most diverse yet most widely used set of assessment tools for measuring emotion Larsen amp Fredrickson, 1999 . Measures range from rating scales and adjective checklists, to analog scales and real-time rating dials. Proponents of self-report measures e.g., Baldwin, 2000 assume that participants are in a privileged position to monitor, assess, and integrate...

Emotion Assessed Through Physiology

Emotion output that can be assessed with physiological methods can be divided into two categories the somatic changes and changes reflecting autonomic or central nervous system activity. The somatic changes most useful to emotion researchers concern muscle movements associated with emotional expression, particularly those somatic changes on the face. Measures of somatic change. One useful measurement strategy is to have an observer rate how much emotion a target participant appears to be...

Emotion Assessed Through Behavior

Behaviors that are linked to emotions range from the very simple, such as defensive reflex actions, to the complex, such as sequences of action tendencies. Emotions likely evolved to produce adaptive actions, such as to approach desired objects or to withdraw from dangerous objects, as well as to support more flexible action tendencies associated with survival. Researchers may take advantage of these behavioral outputs to estimate emotions. Behavior action tendencies. One behavioral...

The Organization Of Cognitive Abilities

There are literally hundreds of ability tests Carroll, 1993 Cattell, 1971 Jensen, 1980, 1998 Sternberg, 1994 ,1 and a framework is needed to organize them. Over the years, proposals have been made to Support for this article was provided by a Templeton Award for Positive Psychology, a NICHD Grant P30HD15052 to the John E Kennedy Center at Vanderbilt University, and a Cattell Sabbatical Award. Earlier versions of this manuscript profited from many excellent suggestions by Camilla P Ben-bow....

Introduction To The Basic Item Response Models

Item response theory IRT see Baker, 1992 Hamble-ton amp Swaminathan, 1989 usually is considered a class of statistical models for categorical data to which the Rasch model RM and the two-parameter logistic model belong, but not latent class analysis LCA . However, there are at least two reasons why LCA can and should be seen as an IRT model. Whereas the Rasch model tries to measure a latent trait of the persons, LCA tries to identify latent classes of persons. Both measure a latent variable,...

Some Alternative Metrics

In an ideal world of construct validity, we would collect data whose assessment methods share a minimum of facets e.g., structured vs. semistruc- tured vs. unstructured, paper-and-pencil vs. observations by others, verbal vs. behavioral, accessible vs. inferred by observers, transparent vs. masked with other assessments of these constructs. As this book details, other areas of psychology have developed metrics that could and should be applied to I O research. These other assessments could...

Temporal Duration Emotions as States Emotions as Traits

Emotions are dynamic processes that take place over time often rapidly, but sometimes gradually involving a cascade of different response systems, such as those mentioned earlier. Each response system has its own dynamic and duration. For example, the central nervous system is very fast with negative stimuli producing changes at around milliseconds Smith, Cacioppo, Larsen, amp Chartrand, 2003 , the cardiac system somewhat slower with changes taking place on a beat-by-beat basis , the skin...

Weintraubs Analysis of Verbal Behavior

Weintraub's 1981, 1989 work on verbal mannerisms was inspired by the clinical observation that individuals speaking under stress often reveal important information about their psychological adjustment. Drawing on his medical training and practice, Weintraub argued that psychological defense mechanisms manifest themselves in speech patterns obtained under mildly stressful conditions. He assessed these defense mechanisms from the language that participants spontaneously use when they talk for 10...

How Far Can We Go With Counting Words

Given that word count-based measures possess rather good psychometric properties, how far can we go with counting words Frequently researchers voice their scientific disdain for text analysis programs that are unable to distinguish between sentences as simple as the dog bit the man and the man bit the dog Hart, 2001, p. 53 . Its blindness to context makes word-count approaches sometimes appear painfully dumb. Not only are they unable to pick up irony or sarcasm e.g., Thanks a lot, accompanied...

Understanding the Question

When an individual responds to a self-report measure, he or she must first make sense of the question being asked Schwarz, 1999 Tourangeau et al., 2000 . To do this, the respondent must understand the literal meaning of the question, and anything that impedes this understanding e.g., vague or unfamiliar words, complicated sentence structure will undermine the quality of the self-report measure. Psychological assessment and survey methodology textbooks suggest that to avoid misunderstandings,...

Multilevel Models for Generalizability Analysis

Generalizability theory can be viewed as a special case of multilevel analysis. In the one-facet nested design, the nesting structure is clear The items are nested in the persons. In the one-facet crossed design, the nesting structure is arbitrary Items can be seen as nested in persons or persons nested in items. Both specifications lead to the same results. Because of the analogy with the nested design, we will use the specification structure of items in persons. In a two-facet design, the...

In Search Of Depressogenic Thought Processes

A variety of studies using a variety of cognitive methods have sought to discover the depressogenic thought processes responsible for depression for reviews, see MacLeod amp Mathews, 1994 Segal, 1988 Segal amp Ingram, 1994 . However, one problematic result emerges from this research. Specifically, it is extremely difficult to distinguish formerly depressed participants from never depressed participants on measures of cognitive bias e.g., Segal amp Ingram, 1994 . That is, depressive biases in...

Analysis of Artistic Change Martindales Regressive Imagery Dictionary

To identify regularities underlying changes in artistic work over time, Martindale 1990 developed a word count program that is based on the Regressive Imagery Dictionary. Martindale's 1990 theorizing starts from the observation that artistic work shows a steady increase in complexity over time. He explains this increase by drawing on two funda mental psychological processes humans' preference for medium levels of arousal and hence moderately complex sensory input and the physiological mechanism...

Campbell and Fiske

Brunswik Lens Model

Multimethod thinking in psychological assessment was influenced most strongly by the seminal paper of Campbell and Fiske 1959 . No other publication so importantly shaped researchers' awareness of the crucial role multimethod designs play in the construction and validation of measurement instruments Shrout amp Fiske, 1995 . Although Campbell and Fiske 1959 did not make reference to Brunswik's work, their proposals were guided by similar insights and ideas. Campbell and Fiske 1959 introduced the...