The 1990s witnessed accelerating growth in the level of interest and development of behavioral health care outcomes programs. The interest in and necessity for outcomes measurement and accountability in this era of managed care provide a unique opportunity for psychologists to use their training and skills in assessment (Maruish, 1999a). However, the extent to which psychologists and other trained professionals become a key and successful contributor to an organization's outcomes initiative will depend on their understanding of what outcomes and their measurement and applications are all about.
What Are Outcomes?
Outcomes is a term that refers to the results of the specific treatment that was rendered to a patient or group of patients.
Along with structure and process, outcomes is one component of whatDonabedian (1980,1982,1985) refers to as "quality of care." The first component is structure. This refers to various aspects of the organization providing the care, including how the organization is organized, the physical facilities and equipment, and the number and professional qualifications of its staff. Process refers to the specific types of services that are provided to a given patient (or group of patients) during a specific episode of care. These might include various tests and assessments (e.g., psychological tests, lab tests, magnetic resonance imaging), therapeutic interventions (e.g., group psychotherapy, medication), and discharge planning activities. Outcomes, on the other hand, refers to the results of the specific treatment that was rendered.
In considering the types of outcomes that might be assessed in behavioral health care settings, a substantial number of clinicians would probably identify symptomatic change in psychological status as being the most important. However, no matter how important change in symptom status may have been in the past, psychologists and other behavioral health care providers have come to realize that change in many other aspects of functioning identified by Stewart and Ware (1992) are equally important indicators of treatment effectiveness. As Sederer et al. (1996) have noted,
Outcome for patients, families, employers, and payers is not simply confined to symptomatic change. Equally important to those affected by the care rendered is the patient's capacity to function within a family, community, or work environment or to exist independently, without undue burden to the family and social welfare system. Also important is the patient's ability to show improvement in any concurrent medical and psychiatric disorder. . . . Finally, not only do patients seek symptomatic improvement, but they want to experience a subjective sense of health and well being. (p. 2)
Following are considerations and recommendations for the development and implementation of outcomes assessment by psychologists. Although space limitations do not allow a comprehensive review of all issues and solutions, the information that follows touches upon matters that are most important to psychologists who wish to incorporate outcomes assessment into their standard therapeutic routine.
The specific aspects or dimensions of patient functioning that are measured as part of outcomes assessment will depend on the purpose for which the assessment is being conducted. Probably the most frequently measured variable is that of symptomatology or psychological/mental health status. After all, disturbance or disruption in this dimension is probably the most common reason why people seek behavioral health care services in the first place. However, there are other reasons for seeking help. Common examples include difficulties in coping with various types of life transitions (e.g., a new job, a recent marriage or divorce, other changes in the work or home environment), an inability to deal with the behavior of others (e.g., spouse, children), or general dissatisfaction with life. Additional assessment of related variables may therefore be necessary or even take precedence over the assessment of symptoms or other indicators.
For some patients, measures of one or more specific psychological disorders or symptom clusters are at least as important as, if not more important than, overall symptom or mental health status. Here, if interest is in only one disorder or symptom cluster (e.g., depression), one may choose to measure only that particular set of symptoms using an instrument designed specifically for that purpose (e.g., the BDI-II would be used with depressed patients). For those interested in assessing the outcomes of treatment relative to multiple psychological dimensions, the administration of more than one disorder-specific instrument or a single, multiscale instrument that assesses all or most of the dimensions of interest (e.g., BSI) would be required. Again, instruments such as the SA-45 or the BSI can provide a quick, broad assessment of several symptom domains.
It is not always a simple matter to determine exactly what should be measured. However, careful consideration of the following questions should greatly facilitate the decision: Why did the patient seek services? What does the patient hope to gain from treatment? What are the patient's criteria for successful treatment? What are the clinician's criteria for the successful completion of the current therapeutic episode? What, if any, are the outcomes initiatives within the provider organization? Note that the selection of the variables to be assessed may address more than one of the above issues. Ideally, this is what should happen. However, one needs to ensure that the task of gathering outcomes data does not become too burdensome. The key is to identify the point at which the amount of data that can be obtained from a patient or collaterals and the ease at which they can be gathered are optimized.
Once the decision of what to measure has been made, one must then decide how it should be measured. In many cases, the most important data will be those that are obtained directly from the patient using self-report instruments. Underlying this assertion is the assumption that valid and reliable instrumentation, appropriate to the needs of the patient, is available to the clinician; the patient can read at the level required by the instruments; and the patient is motivated to respond honestly to the questions asked. Barring one or more of these conditions, other options should be considered.
Other types of data-gathering tools may be substituted for self-report measures. Rating scales completed by the clinician or other members of the treatment staff may provide information that is as useful as that elicited directly from the patient. In those cases in which the patient is severely disturbed, unable to give valid and reliable answers (as in the case of younger children), unable to read, or otherwise an inappropriate candidate for a self-report measure, clinical rating scales, such as the Brief Psychiatric Rating Scale (BPRS; Faustman & Overall, 1999; Overall & Gorham, 1962) and the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1994), can serve as a valuable substitute for gathering information about the patient. Related to these instruments are parent-completed instruments for child and adolescent patients, such as the Child Behavior Checklist (CBCL; Achenbach, 1991) and the Personality Inventory for Children-2 (PIC-2; Lachar & Gruber, 2001). Collateral rating instruments and parent-report instruments can also be used to gather information in addition to that obtained from self-report measures. When used in this manner, these instruments provide a mechanism by which the clinician, other treatment staff, and parents, guardians, or other collaterals can contribute data to the outcomes assessment endeavor.
There are no hard and fast rules or widely accepted conventions related to when outcomes should be assessed. The common practice is to assess the patient at least at treatment initiation and again at termination or discharge. Additional assessment of the patient on the variables of interest can take place at other points as part of postdischarge follow-up.
Many would argue that postdischarge or posttermination follow-up assessment provides the best or most important indication of the outcomes of therapeutic intervention. In general, postdischarge outcomes assessment should probably take place no sooner than 1 month after treatment has ended. When feasible, waiting 3-6 months to assess the variables of interest is preferred. A longer interval between discharge and postdischarge follow-up should provide a more valid indication of the lasting effects of treatment. Comparison of the patient's status on the variables of interest at the time of follow-up with that found at the time of either treatment initiation or termination will provide an indication of the more lasting effects of the intervention. Generally, the variables of interest for this type of comparison include symptom presence and intensity, feeling of well-being, frequency of substance use, and social or role functioning.
Although it provides what is arguably the best and most useful outcomes information, a program of postdischarge follow-up assessment is also the most difficult to successfully implement. There must be a commitment of staff and other resources to track terminated patients; contact them at the appropriate times to schedule a reassessment; and process, analyze, report, and store the follow-up data. The task is made more difficult by frequently noted difficulties in locating terminated patients whose contact information has changed, or convincing those who can be located to complete a task from which they will not directly benefit. However, those organizations and individual clinicians who are able to overcome the barriers will find the fruits of their efforts quite rewarding.
There are two general approaches to the analysis of treatment outcomes data. The first is by determining whether changes in patient scores on outcomes measures are statistically significant. The other is by establishing whether these changes are clinically significant. Use of standard tests of statistical significance is important in the analysis of group or population change data. Clinical significance is more relevant to change in the individual patient's scores.
The issue of clinical significance has received a great deal of attention in psychotherapy research during the past several years. This is at least partially owing to the work of Jacobson and his colleagues (Jacobson, Follette, & Revenstorf, 1984, 1986; Jacobson & Truax, 1991) and others (e.g., Christensen & Mendoza, 1986; Speer, 1992; Wampold & Jenson, 1986). Their work came at a time when researchers began to recognize that traditional statistical comparisons do not reveal a great deal about the efficacy of therapy. In discussing the topic, Jacobson and Truax broadly define the clinical significance of treatment as "its ability to meet standards of efficacy set by consumers, clinicians, and researchers" (p. 12).
From their perspective, Jacobson and his colleagues (Jacobson et al., 1984; Jacobson & Truax, 1991) felt that clinically significant change could be conceptualized in one of three ways. Thus, for clinically significant change to have occurred, the measured level of functioning following the therapeutic episode would either (a) fall outside the range of the dysfunctional population by at least 2 standard deviations from the mean of that population, in the direction of functionality; (b) fall within 2 standard deviations of the mean for the normal or functional population; or (c) be closer to the mean of the functional population than to that of the dysfunctional population. Jacobson and Truax viewed option (c) as being the least arbitrary, and they provided different recommendations for determining cutoffs for clinically significant change, depending upon the availability of normative data.
At the same time, these investigators noted the importance of considering the change in the measured variables of interest from pre- to posttreatment in addition to the patient's functional status at the end of therapy. To this end, Jacobson et al. (1984) proposed the concomitant use of a reliable change (RC) index to determine whether change is clinically significant. This index, modified on the recommendation of Christensen and Mendoza (1986), is nothing more than the pretest score minus the posttest score divided by the standard error of the difference of the two scores.
The demand to demonstrate the outcomes of treatment is pervasive throughout the health care industry. Regulatory and accreditation bodies are requiring that providers and provider organizations show that their services are having a positive impact on the people they treat. Beyond that, the behavioral health care provider also needs to know whether what he or she does works. Outcomes information derived from psychological assessment of individual patients allows the provider to know the extent to which he or she has helped each patient. At the same time, in aggregate, this information can offer insight about what works best for whom under what circumstances, thus facilitating the treatment of future patients.
Was this article helpful?
Are You Depressed? Heard the horror stories about anti-depressants and how they can just make things worse? Are you sick of being over medicated, glazed over and too fat from taking too many happy pills? Do you hate the dry mouth, the mania and mood swings and sleep disturbances that can come with taking a prescribed mood elevator?