Projective Techniques

Despite trying to remain objective, individuals’ personalities and personal interpretations might unconsciously influence their responses to personality assessments. Projective techniques attempt to tap into this unconsciousness. Unlike the directness of self-report inventories, these performance-based measures rely on interpreting the performance of clients as they respond to ambiguous stimuli. Projective techniques include the Thematic Apperception Test (TAT), the House-Tree-Person (HTP), the Tell-Me-A-Story (TEMAS), and the Incomplete Sentence Blank tests (ISB). As a psychologist, in what circumstances might projective techniques be an effective assessment method?

For this Discussion, review this week’s Learning Resources. Select two projective instruments from the following list: Thematic Apperception Test (TAT), House-Tree-Person (HTP) Test, Incomplete Sentence Blank (ISB), and Tell-Me-A-Story (TEMAS) Assessment. Consider the similarities and differences between the two instruments you selected and the insights you gained from your comparison. Finally, think about three challenges related to the use of the instruments you selected.

With these thoughts in mind:

Write a comparison (similarities and differences) of the two projective instruments you selected and explain what insight you gained from making that comparison. Explain three challenges related to the use of the projective instruments you selected.

Reference:

• Anastasi, A., & Urbina, S. (2009). Psychological testing (7th ed.). Upper Saddle River, NJ: Prentice Hall.
  • Chapter 15, “Projective Techniques,” (pp. 410–442)
• Flanagan, R., & Di Guiseppe, R. (1999). Critical review of the TEMAS: A step within the development of thematic apperception instruments. Psychology in the Schools, 36(1), 21–30.
  Retrieved from the Walden Library databases.
• Garb, H. N., Wood, J. M., Lilienfeld, S. O., & Nezworski, M. T. (2002). Effective use of projective techniques in clinical practice: Let the data help with selection and interpretation. Professional Psychology: Research and Practice, 33(5), 454–463.
  Retrieved from the Walden Library databases.

A Sample Answer For the Assignment: Projective Techniques

Title: Projective Techniques

Psychology in the Schools, Vol. 36(1), 1999 © 1999 John Wiley & Sons, Inc. CCC 0033-3085/99/010021-10 CRITICAL REVIEW OF THE TEMAS: A STEP WITHIN THE DEVELOPMENT OF THEMATIC APPERCEPTION INSTRUMENTS rosemary flanagan Baldwin Union Free School District and St. John’s University raymond di giuseppe St. John’s University The TEMAS (acronym for Tell-Me-a-Story)—an objectively scored, projective thematic personality instrument for children and adolescents—is analyzed, reviewed, and critiqued with regard to theoretical underpinnings and rationale for development, administration, scoring, psychometric properties, and research to date. The TEMAS appears to be an improvement over existing projective personality measures used by school psychologists. Although it requires more training than other projective techniques, competency in administration, scoring, and interpretation can be achieved within a one semester course in personality assessment. The test has evidence of reliability and validity, and it is a multicultural alternative to the TAT and other thematic apperception instruments. The use of the TEMAS by psychologists may achieve more accurate assessment of Black and Hispanic children. Limitations include geographically limited standardization samples and little research conducted by individuals other than the authors. © 1999 John Wiley & Sons, Inc. School psychologists continue to debate the utility of projective personality assessment instruments such as Figure Drawings (Hammer, 1980), the Thematic Apperception Test (TAT; Murray, 1943), and the Rorschach prior to the development of the Comprehensive System (Exner, 1993). Although these instruments are popular among school psychologists, empirical research supporting their reliability and validity is lacking (Batsche & Petersen, 1983; Knoff, 1983; Martin, 1988; Motta, Little, & Tobin, 1993). Some psychologists argue that projective measures should not be used because of the lack of demonstrable validity (Martin, 1983, 1988). Others maintain that projective techniques can be qualitatively useful in assessing children’s emotional functioning (Rabin, 1986; Teglasi, 1993). Despite this controversy, surveys of school psychologists indicate these tests remain popular (Goh & Fuller, 1983). Recent surveys suggest there may be a small tendency away from the use of projective techniques in favor of objective measures of social and emotional functioning (Hutton, Dubes, & Muir, 1992; Kennedy, Faust, Willis, & Piotrowski, 1994). Theoretical Underpinnings and Rationale for Development The projective hypothesis (Frank, 1939), which maintains that all individuals view the world in terms of their own experience, forms the basis of the individualized clinical utility of such instruments. Test stimuli are frequently ambiguous and responses are open ended. It is generally assumed that the purpose of the assessment and the nature of the desired information are not readily apparent to the examinee. Practitioners desire the scientific properties of objective personality measures while maintaining the richness of projective tests to reveal idiosyncratic aspects of the individual. A case in point is the Rorschach, developed as a perceptual–cognitive task and later shown to be a stimulus to projection (Exner, 1993). When administered, scored and interpreted according to the Comprehensive Scoring System (Exner), the Rorschach can be defended as a reliable and valid assessment instrument. Meta-analyses of studies (Atkinson, 1986; Parker, Hanson, & Hunsley, 1988) of the Minnesota Multiphasic Personality Inventory (MMPI; Hathaway & Mc Kinley, 1943) and the Rorschach indicate that the MMPI and the Comprehensive System have comparable psychometric properties. The use of the Comprehensive System does not preclude the freedom to interpret chil- Correspondence to: Rosemary Flanagan, Department of Psychology, St. John’s University, 8000 Utopia Parkway, Jamaica, NY 11439. 21 22 Flanagan and Di Giuseppe dren’s responses in a more traditional subjective manner. The rationale for objective scoring and impressionistic interpretation is that scoring yields information regarding level of functioning on various dimensions; impressionistic interpretation provides additional information that can be used to generate ideographic descriptions of the child’s functioning. The TEMAS Thematic Apperception Test was developed by Costantino, Malgady, and Rogler (1988) in this tradition. Psychological tests generally have been criticized for insensitivity to ethnic minorities and recent immigrants. Two types of bias exist: ethnocultural insensitivity in regard to the actual test materials and interpretive bias on the part of the examiner (Fuchs, 1986). Given the rise in multicultural awareness, the diversity of ethnic and social backgrounds of children served by school psychologists, and the criticism of cultural bias in IQ tests, one would expect concern among psychologists regarding possible bias in personality assessment (American Psychological Association, 1993). Psychologists unfamiliar with examinees’ culture may misinterpret a projective response to symbolize something different from what it represents in the examinees’ culture. Costantino, Malgady, and Rogler (1988) addressed these concerns by developing the TEMAS with norms for Black, Puerto Rican, other Hispanic, and White children. Most projective tests are based on the assumption that the more ambiguous the test stimuli the more likely a person is to project his or her personal concerns and issues into perceptions of the stimulus (Rabin, 1986) and elicit responses believed to reflect the deep unconscious concerns of the client, despite research on the ambiguity hypothesis (Epstein, 1966; Goldstein, & Barthol, 1960; Kenny & Bijou, 1953) which demonstrated inconsistent findings. Murstein (1972), in a critique of the literature, concluded that the level of ambiguity of the stimulus could not be assumed to be a key variable in pulling for the most valid information and further indicated that significant research exists indicating that many aspects of the cards influence examinees’ responses, such as the stimulus, context, purpose of the test, and examiner–examinee interaction. Costantino, Malgady, and Rogler (1988) considered the inconsistent data when developing the TEMAS stimulus cards and chose to consider the research supporting the use of unambiguous stimuli (Epstein; Goldstein & Barthol; Sobel, 1981; Veroff, 1968) because such research suggested that reliability and validity of the projective responses could be increased in this manner. Consistent with expectations, greater verbalization occurred when the stimulus materials were more closely identified with the ethnicity, gender, and age of the examinee (Costantino & Malgady, 1983; Costantino, Malgady, & Vasquez, 1981). In addition, greater verbalization on the TEMAS occurred for Spanish language dominant children when they responded in Spanish, suggesting that the limited verbalization often obtained on tests such as the TAT for ethnic minority respondents is related to factors other than cognitive variables. Most adherents of projective tests would argue that greater verbalization would increase the likelihood that more clinically useful data would be obtained. Thus, the more similar the stimulus materials are to the individual, the more likely projection is to occur. Costantino, Malgady, and Rogler (1988) maintain that the major characters in stimulus cards should reflect the demographic characteristics of the examinee. Additional research is needed to determine which demographic characteristics enhance the examinee’s identification with the stimulus card and the optimal level of specificity needed to reflect the demographic characteristics of the examinee. It is not known whether separate stimulus cards for Blacks and Hispanics would yield better results. Noteworthy is that an Asian version has been developed but remains unpublished (G. Costantino, personal communication, 1997). The TEMAS stimulus cards differ from the traditional TAT in two other respects. First, the stimulus materials are concretely designed to represent conflicts that children and adolescents usually encounter, while the TAT cards do not because these were generally designed for adults. The TEMAS cards depict everyday situations in terms of a conflict to be resolved. Adherents of projective instruments might argue that highly specific stimuli will limit projection as well as its extent. We believe Critical Review of the TEMAS 23 this is acceptable because the gains more than offset the limitations. These gains are the application of scoring categories to all responses given, the use of a normative data base, the opportunity to conduct research that is more defensible, and the opportunity to compare research results across studies. The structured conflict-laden situations depicted in the TEMAS cards offer a unique opportunity to observe a youngster’s social problem solving skills or coping skills; this is often desired information from a TAT administration, but its collection by the evaluator can only be hoped for. Such information is desirable because it carries implications for diagnosis, prognosis, and development of individualized interventions. Unfortunately, the authors of the TEMAS have not effectively communicated this point. Another major difference is that the TEMAS stimulus materials are in color, while all other thematic apperception tests are in black and white. Research indicates that colored Rorschach inkblots result in greater verbalization by the respondent (Exner, 1993; Exner & Weiner, 1994). Similarly, early research on the TAT suggested that colored cards led to greater expression of affect in clinical groups (Brackbill, 1951) and produced greater verbalization (Thompson & Bachrach, 1951). It appears that more theoretical and empirical variables have been incorporated in the design and development of the TEMAS than in other thematic apperception instruments. Materials The TEMAS test kit consists of two parallel (minority and nonminority) sets of stimulus cards, a manual, record forms, and a laminated administration card, on which the instructions to be read to the child and the structured inquiries are printed in bold faced type. Each set contains 12 cards appropriate to both genders and 11 cards with male and female versions. One stimulus card is offered in four versions, male and female, child and adolescent. The majority of main characters depicted in the cards are youngsters, with adults appearing in secondary roles. The child’s actual responses (stories) are recorded verbatim on plain paper. Ample space is provided on the first page of the record booklet to record demographic information about the child. The second and third pages contain scoring boxes for every function for each of the 23 cards. The intended psychological conflicts pulled for by a given card are clearly indicated by a triangle in the scoring box. The scoring boxes for the short form cards are highlighted in gray, with asterisks in the left-hand margin. The last page is a scoring profile for the quantitative indicators. Data are plotted as t scores and percentiles. Space is also provided to note critical cut-off points for conflicts “not pulled.” The materials are “user-friendly.” Administration The test is similar to the TAT in that examinees are requested to tell a story with a beginning, middle, and end in response to pictures that they will be shown. Structured inquiries are conducted after each story, as needed. These inquiries address the relationships among the characters in the story, where the story is taking place, what the characters are saying and doing, what happened previously, what will happen next, and the thoughts and feelings of the main character. Reaction time and the time telling the story and responding to inquiries are recorded. Children should be encouraged to respond for at least 2 minutes and encouraged to finish their story after 5 minutes. Test administration procedures are highly specific, providing for greater standardization of administration. Scoring and Interpretation of the TEMAS Each TEMAS story is scored on emotional, personality, or cognitive (similar to formal features of the story) constructs called functions. Scoring is quantitative or qualitative, which is based on the skewness of the normative data. For some functions, the distribution of scores was only slightly skewed and data are reported quantitatively. These include all 8 Personality Functions, 4 of 7 Af- 24 Flanagan and Di Giuseppe fective Functions, and 4 of 11 Cognitive Functions. Normalized t scores and percentiles are available for these variables. t scores of 70 and above or 30 and below place the examinee in a clinical range, depending upon the variable assessed. Extra attention to detail on the part of the examiner will prevent errors, as the direction of interpretation of the variables is obvious (e.g., t ⫽ 70 on Happy is interpreted differently from t ⫽ 70 on Sad). The normative data for all other functions are scored qualitatively. The examinee is considered to have scored high on the variable and the function is interpreted only when the t score is above a critical value for that function which is the 90th percentile. The Personality Functions are scored on a continuum from 1 to 4,1 ⫽ highly maladaptive and 4 ⫽ highly adaptive conflict resolution. Lower scores indicate weaker coping and problem solving abilities. The Personality Functions are Interpersonal Relations, Aggression, Anxiety/Depression, Achievement Motivation, Delay of Gratification, Self-Concept, Sexual Identity, and Moral Judgment. Each stimulus card is designed to tap some of the personality functions. Personality functions that are expected to be tapped by a particular card and do not appear in the child’s story are scored as not pulled. Affective Functions are operationally defined as the main character’s affect at the resolution of the conflict/conclusion of the story. Cognitive Functions which are scored qualitatively represent requested aspects of the story which are missing from the story. These are failure to recognize a conflict situation, appropriately sequence the story elements, demonstrate imagination on the part of the storyteller, describe the relationships among characters, as well as structured inquiries not responded to and transformation of characters, setting, or event. Normative Characteristics Norms The availability of norms for four different groups is unique for any objective or projective measure of social emotional functioning. There are norms for Whites, Blacks, Puerto Ricans, and other Hispanics for both the long and short forms for three age groups (5–7, 8–10, 11–13). Norms for adolescents are currently being developed (G. Costantino, personal communication, 1997). The TEMAS can only be used qualitatively for this age group. The standardization sample was obtained from public schools in New York City. The mean age of the 281 males and 361 females was 8.9 years (SD ⫽ 1.9). The tabled data for conversion of quantitative scale raw data to percentile scores were normalized because of skewness of the raw data. Also, the norms for the short form were derived from the administration of the long form rather than from a separate administration of only the cards in the short form. These procedures make extreme scores less likely but may render results from the short form more tenuous. Clearly the establishment of such norms for so many scales derived from a projective measure is an accomplishment, although not without limitations. Nevertheless, psychological evaluations based on the TEMAS should be an improvement over those obtained from the use of unnormed thematic apperception tests. Serious problems with standardization include the small size of the normative populations and their severe geographic restriction. Costantino, Malgady, Casullo, and Castillo (1991) later found that native Puerto Rican children responded differently to the TEMAS than New York Puerto Ricans and Argentine children responded with fear to some stimuli; all of these differences were demonstrated on scorable indices of the TEMAS. The authors concluded that responses are significantly influenced by very specific cultural variables. Test users must consider these factors and exercise caution when interpreting data from individuals from other geographic areas. Reliability Internal consistency for each function scored was established using Cronbach’s ␣. Data in the manual indicate that the median ␣ coefficient for the function scales for the Black sample is .62; the average ␣ coefficients for the Hispanic sample is .73. The median coefficient ␣ for the long form Critical Review of the TEMAS 25 quantitative indicators for the total sample is .83; for the short form, the median coefficient was .68. The decrease in internal consistency for the short form is attributable to less available data on all scored functions; it appears that a larger sample size is needed to more adequately establish the internal consistency of the short form. Some functions are clearly inadequate to be interpreted for an individual, but may have research utility when studying groups. The test user must be cautious and know which function scales can be interpreted with confidence by referring to the manual. Test–retest reliability over an 18-week period has been described as low to moderate. The authors indicated that this may be attributed to possible inconsistency on the part of the subjects and the use of different raters at pretest and posttest. The restricted range of the actual functions may have lead to attenuated correlations. Interrater reliability, according to card and function rated, is generally moderate, with values ranging from .33 to 1.00 (68% of reported values are at least .70). A later study using the short form (Costantino, Malgady, Casullo & Castillo, 1988) demonstrated higher interrater reliability. Interrater agreement for scoring the personality functions (the most problematic area) ranged from 75 to 95%, with the mean agreement being 81%. More research is needed to demonstrate the interclinician reliability for the TEMAS. Validity Studies of validity reported in the manual suggest that the TEMAS shows promise for differentiating normal from clinical groups, although there were methodological limitations in the work which make data interpretation difficult. Follow-up studies (Costantino, Malgady, Colon–Malgady, & Bailey, 1992; Costantino, Malgady, Rogler, & Tsui, 1988) demonstrated that profiles of the TEMAS personality functions differentiated clinical and nonclinical (public school) samples based on the discriminant function. Although both follow-up studies are flawed in regard to the lack of representativeness of the samples studied, control for examiner effects/biases, and the definitions of clinical (DSM-III diagnoses and/or psychiatric interviews) and normal (public school attendance and parental report), the data suggest that the TEMAS offers promise as a clinical tool for both minority and nonminority individuals. In these studies, 86 –89% of the cases were correctly classified as clinical or nonclinical. The TEMAS has been used to assess Attention Deficit Hyperactivity Disorder (ADHD; Costantino, Colon–Malgady, Malgady & Perez, 1991). Diverse samples of nonclinical, public school children were compared with a outpatients who met the diagnostic criteria for ADHD. The data indicated that ADHD children made more omissions of all types as well as omissions of the conflict and other details that are assessed by the cognitive functions of the TEMAS than normals. Given that the impact of ADHD is not generally observed in a systematic manner through psychological testing, such information may be useful to school psychologists. These results suggest that the TEMAS may be useful for providing supplementary information for intervention/treatment planning. Flanagan (1997) examined the relationship between the TEMAS and the Self-Report of Personality of the Behavior Assessment System for Children (BASC; Reynolds & Kamphaus, 1992) for a diverse sample of youngsters. TEMAS Personality Functions were generally found to bear inverse relation to BASC Scales that measure clinical or school problems but were positively related to the Personal Adjustment Composite and its components. This is an expected pattern of findings, as lower t scores on the TEMAS indicate poorer ability to cope with conflict/solve problems. These data provide an indication that the TEMAS is measuring what it is purported to measure. Clearly, more information needs to be become available before it can be said with certainty that the TEMAS is a tool that will yield valid assessments. TEMAS and the Status of Thematic Apperception Tests Research support for the thematic apperception technique has been mixed (Murstein, 1963; Vane, 1981). Thomas and Dudek (1985) concluded that TAT can be reliable and valid when scoring 26 Flanagan and Di Giuseppe systems are devised for specific personality constructs such as achievement motivation (Mc Clelland, Atkinson, Clark, & Lowell, 1953), empathy (Kaliopska, 1982) and need for intimacy (Mc Adams, 1980). Research is available assessing the reliability and validity of specific constructs. Practitioners often seem to use thematic apperception tests hoping to measure any and all constructs. Each practitioner may use such tests to assess different constructs, depending on his or her theoretical orientation. Worchel and Dupree (1990) question whether thematic apperception techniques should be considered psychometric instruments or strategies for exploring unique aspects of the individual. No validity studies exist for thematic apperception tests for the manner in which these are used in routine clinical practice. Use of the TEMAS circumscribes and delimits these issues, while offering a possible solution to a dilemma in personality assessment. The conflict surrounding projective tests may exist because psychologists usually seek two types of information from their evaluations. The first is diagnostic information which concerns the level and type of disturbance. This type of question seems best answered by objective psychometrically validated instruments because there are normed data available for comparison. The traditional, clinical, unstructured, interpretation of projective tests has failed to demonstrate validity to answer such questions and has been criticized. Objective self-report or parent and teacher rating measures have been more effective for answering these questions. Information on the type and degree of emotional disturbance is helpful for designing interventions. Another type of information that psychologists seek, which might be called motivationalschematic, concerns the motivations, schema, circumstances, and constructs of the individual which drive the emotions. Practitioners are often more interested in the ideographic aspects of assessment, as scores on an objective rating measure are compared to nomothetic data and do not generally provide information on why a problem exists. Arnold (1962) maintained that this information can be obtained relatively quickly and accurately from the TAT. Murray originally designed the TAT to access this type of information for use in therapy (Worchel & Dupree, 1990). Motivational-schematic information without normed reference to the level of functioning may not reflect whether the child has a disturbed level of emotions or whether the motivational, conflictual schema interfere with functioning. Psychologists who use unnormed projective tests may overpathologize the examinee’s functioning. This occurs because the clinical, interpretive use of projective tests may not be valid for this type of question. All individuals, whether disturbed or not, may reveal the major conflicts, schema, and motivations which rule their lives. Although this information is important, it may be of limited usefulness for deciding whether a person is disturbed, but may be very helpful in organizing, implementing, and guiding the intervention. Psychologists need different techniques to answer different questions. Objective measures can be used to assess the type and level of emotional disturbance. Projective measures can provide information concerning the idiosyncratic conflicts and schema for each examinee. Only scoring and clinically interpreting the protocols can provide both types of information validly. McClelland, Kostener, and Weinberger (1989) proposed that both projective and objective methodologies are needed to obtain a broader and more complete assessment of personality. It is recommended that practitioners separate the two questions and answer each separately. The TEMAS, like the Comprehensive System for the Rorschach (Exner, 1993), can provide answers to both questions if it is scored objectively and interpreted clinically. There is one concern. While the objective measures of anxiety, depression, and anger, which are scored from the TEMAS response can be normed, researched, and validated, the motivational/schematic interpretations are still suspect. It is difficult to test the validity of the clinical interpretations of conflicts, motives or schema based on projective responses. This occurs because the practitioner may assess a different construct for each examinee because there may be individual variation in the meaning of such constructs. Given that the material gleaned from each protocol may reflect Critical Review of the TEMAS 27 different constructs from client to client, the construct or theme interpreted from each individual’s response(s) can not be aggregated and tested for validity. Although information provided by projective instruments can be important and useful, one can not be unconcerned about the validity of interpretations. One means by which validity might be estimated is through concordance (diagnostic criteria or specific symptoms) data, using the ␬ statistic. Perhaps a productive research model for testing the validity of clinical interpretation of projective material is the research (Luborsky, Crits–Christoph, Mintz, & Auerbach, 1988) on the accuracy of interpretations in psychodynamic psychotherapy. Luborsky and his associates developed a strategy to assess the validity of the psychotherapists’ interpretations of clients’ core conflictual relationship theme (CCRT) in psychotherapy. The researchers noted that the CCRTs bear resemblance to scoring categories or interpretations derived from the TAT but are construed from the material revealed by the patient in psychotherapy (Luborsky et al.). The CCRT data have the following characteristics: (a) there is no main theme for each person, (b) the pattern is distinctive for each person, (c) the theme is partly out of awareness, (d) the theme is consistent over time, (e) the pattern may change slightly over time, (f ) the pattern is evident in the relationship to the therapist, (g) early-in-life patterns of relationships are like the current pattern. The validity of each interpretation for that client is assessed so that the uniqueness of the interpretation is maintained. The accuracy of the CCRT interpretations are assessed by determining if these are related to the effectiveness of therapy. A model based on the valid and reliable scoring system of Luborsky and his associates, which has demonstrated treatment utility, may be the best method to demonstrate the validity for this type of information that clinicians find so helpful from thematic apperception tests. Such an approach to scoring thematic stories may be necessary because it is very difficult to teach clinical judgment that is supposed to guide the interpretation of the projectives (Vane & Guarnaccia, 1989). The TEMAS and its Competitors The main competitor of the TEMAS is the Roberts Apperception Test for Children (RATC; Roberts & Mc Arthur, 1982). The limitations of the RATC were addressed when developing the TEMAS. Among the improvements are the use of chromatic stimuli, norms for three ethnic minority groups, a greater number of scorable dimensions, and the use of more specific structured inquiries throughout the test. The RATC has slightly better reliability data, but that is expected, given that the scoring system is less complex. In a comparative study, the TEMAS assessed good adjustment more accurately in a sample of Mexican American children living in California than the RATC (Bernal, 1991), suggesting greater utility of the TEMAS for assessing Hispanic populations. More research in this area is needed. Additional features merit separate discussion. The “pull” of the TEMAS cards is considerably more obvious than that of the RATC; in addition, the conflict situations depicted in the cards are frequently depicted as two situations, either in “thinking bubbles” or side by side. The RATC is similar to the TAT in that the stimulus card may be conflict-laden for the examinee. The TEMAS, therefore, casts the examinee into a situation in which she or he is directly faced with conflict and asked to solve a problem. Such features may result in limiting the information obtained, yet increase the likelihood that the desired information will be obtained. The TEMAS standardization sample, although still small and geographically limited by objective test standards, is larger and more clinically diverse than the sample for the RATC, which was standardized on 200 teacher-nominated welladjusted White children. It has been our experience that separate scoring categories for anxiety and depression, as on the RATC, leads to interrater disagreement. Combining anxiety and depression into a single category on the TEMAS eliminates this source of disagreement and is more consistent with research, suggesting that these two constructs have high comorbidity (Brady & Kendall, 1992). Despite its limitations, the TEMAS may be the superior instrument. 28 Flanagan and Di Giuseppe Conclusions The TEMAS is a projective personality instrument that appears appropriate for use by school psychologists. Some aspects of standardization and psychometric properties are weak. Despite these limitations, its degree of empirical validity appears superior to that of other thematic apperception measures (e.g., RATC, TAT). Therefore, it appears to be a better choice than the other instruments in the diagnosis of social–emotional problems for those school psychologists wishing to use a projective test. The TEMAS may prove useful in the differentiation of special education from regular education students if used as part of a comprehensive assessment paradigm such as the one developed by Mc Conaughy and Achenbach (1990). Research in this area is needed. The objective scores are more readily understandable to parents and members of the Committee on Special Education. In addition, interpretation according to clinician judgment permits the same level of flexibility that is afforded by the TAT. Clearly, these topics need to be studied systematically. A number of limitations of personality tests currently in use are addressed, such as the provision of culturally relevant stimuli, norms for several ethnic minority groups, chromatic stimuli, objective scoring, and available information on psychometric properties. A strength of the scoring system is that all scorable dimensions are operationally defined, including dimensions that are often considered informally when using projective techniques such as the TAT. The use of a specific set of stimulus cards is an improvement over general clinical practice with regard to the TAT because most examiners do not specify which TAT cards were administered. Under such circumstances, it is difficult to state that the TAT is the same test each time it is administered. A secondary benefit is that research studies of the TEMAS may be more readily compared than those of the TAT. Although the TEMAS is an improvement over available projective techniques, limitations remain. Inferences and judgments about a child’s personality functioning may be made on the basis of data that occurs only a few times in the record, leading to questionable reliability and possibly spurious interpretations. Given the objective of the authors, it would not have been appropriate for the standardization sample to reflect census data, but it would be desirable to have data on ethnic minorities residing in suburban and rural areas. Reliability and validity are below accepted limits for some scored dimensions according to accepted standards for objective self-report or behavior rating scales and the Comprehensive System for the Rorschach (Exner, 1993). Some psychologists may consider that such limitations, as found in the TEMAS, may be a necessary compromise when choosing to use a projective instrument. Continued programmatic research to revise the test and develop more extensive norms could achieve psychometric properties similar to objective tests. Continued study of the TEMAS indicates that it is valid for nonminority children, suggesting broad applicability. Most importantly, it offers a means by which ethnic minority children can be evaluated and compared to a normative sample, instead of undiagnosed or misdiagnosed; this may be the most significant contribution of the authors. More research needs to be conducted to investigate the validity of projective instruments with minority and immigrant populations. The authors have provided a model that adherents of projective tests can expand. The present work on the TEMAS is encouraging. Acknowledgments Appreciation is expressed to Mary Jo Dellino, Laura Povall, and Theresa Anderson for their comments. References American Psychological Association. (1993). Guidelines for providers of psychological services to ethnic, linguistic and culturally diverse populations. American Psychologist, 48, 45 –48. Arnold, M. (1962). Story sequence analysis. New York: Columbia University Press. Critical Review of the TEMAS 29 Atkinson, L. (1986). The comparative validities of the Rorschach and MMPI: A meta-analysis. Canadian Psychology, 27, 238–247. Batsche, G., &; Peterson, D. (1983). School psychology and projective assessment: A growing incompatibility. School Psychology Review, 12, 440 – 445. Bernal, I. (1991). The relationship between level of acculturation, the Roberts Apperception Test for Children and the TEMAS: Tell-Me-A-Story Test. (Doctoral dissertation, California School of Professional Psychology, 1991). Dissertation Abstracts International, 52, 2810B. Brackbill, G.A. (1951). Some effects of color on thematic fantasy. Journal of Projective Techniques, 15, 412–418. Costantino, G., Colon–Malgady, G., Malgady, R.G., & Perez, A. (1991). Assessment of attention deficit disorder using a thematic apperception technique. Journal of Personality Assessment, 57, 87–95. Costantino, G., & Malgady, R.G. (1983). Verbal fluency of Hispanic, Black & White children on TAT and TEMAS, a new thematic apperception test. Hispanic Journal of Behavioral Sciences, 5, 199–206. Costantino, G., Malgady, R.G., Casullo, M.M., & Castillo, A. (1991). Cross-cultural standardization of TEMAS in three Hispanic subcultures. Hispanic Journal of Behavioral Sciences, 13, 48–62. Costantino, G., Malgady, R.G., Colon–Malgady, G., & Bailey, J. (1992). Clinical utility of the TEMAS with non-minority children. Journal of Personality Assessment, 59, 433 – 438. Costantino, G., Malgady, R.G., & Rogler, L.H. (1988). Technical manual: TEMAS thematic apperception test. Los Angeles: Western Psychological Services. Costantino, G., Malgady, R.G., Rogler, L.H., & Tsui, E.C. (1988). Discriminant analysis of clinical outpatients and public school children by TEMAS: A thematic apperception test for Hispanics and Blacks. Journal of Personality Assessment, 52, 670–678. Costantino, G., Malgady, R.G., Vasquez, C. (1981). A comparison of the Murray TAT and a new thematic apperception test for urban Hispanic children. Hispanic Journal of Behavioral Sciences, 3, 291–300. Epstein, S. (1966). Some considerations on the nature of ambiguity and the use of stimulus dimensions in projective techniques. Journal of Consulting Psychology, 30, 183 –192. Exner, J.E. (1993). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (3rd ed.). New York: Wiley. Exner, J.E., & Weiner, I.B. (1994). The Rorschach: A comprehensive system: Vol. 3. Assessment of children and adolescents (2nd ed.). New York: Wiley. Flanagan, R. (1997). The BASC and TEMAS: Convergent validity or complementary measures? Manuscript submitted for publication. Frank, L.K. (1939). Projective methods for the study of personality. Journal of Psychology, 8, 389–413. Fuchs, D. (August, 1986). You can take a test out of a situation, but you cannot take the situation out of a test: Bias in minority assessment. Paper presented at the 94th annual convention of the American Psychological Association, Washington, DC. Goh, D.S., & Fuller, G.B. (1983). Current practices in the assessment of personality and behavior by school psychologists. School Psychology Review, 12, 240 –243. Goldstein, M.J., & Barthol, R.P. (1960). Fantasy responses to subliminal stimuli. Journal of Abnormal and Social Psychology, 60, 22–26. Hammer, E.F. (1980). The clinical application of projective drawings (5th ed.). Springfield, IL: Charles C. Thomas. Hathaway, S.R., & Mc Kinley, J.C. (1943). The Minnesota Multiphasic Personality Inventory. Minneapolis: University of Minnesota. Hutton, J.B., Dubes, R., & Muir, S. (1992). Assessment practices of school psychologists: Ten years later. School Psychology Review, 21, 271–284. Kaliopska, M. (1982). Empathy as measured by the Rorschach and the TAT. British Journal of Projective Psychology and Personality Study, 27, 5–11. Kennedy, M.L., Faust, D., Willis, W.G., & Piotrowski, C. (1994). Social–emotional assessment practices in school psychology. Journal of Psychoeducational Assessment, 12, 228 –240. Kenny, D.T., & Bijou, S.W. (1953). Ambiguity of pictures and extent of personality factors in fantasy responses. Journal of Consulting Psychology, 17, 283 –288. Knoff, H. (1983). Justifying projective/personality assessment in school psychology: A response to Batsche & Peterson. School Psychology Review, 12, 446 – 451. Luborsky, L., Crits–Christoph, P., Mintz, J., & Auerbach, A. (1988). Who will benefit from psychotherapy? New York: Basic Books. Martin, R.P. (1983). The ethical issues in use and interpretation of the Draw-A-Person and other similar projective procedures. The School Psychologist, 37, 6. Martin, R.P. (1988). Assessment of personality and behavior problems. New York: Guilford. Mc Adams, D.P. (1980). A thematic coding system for the intimacy motive. Journal of Research in Personality, 14, 413– 422. 30 Flanagan and Di Giuseppe Mc Clelland, D., Koestner, R., & Weinberger, J. (1989). How do self-attributed and implicit motives differ? Psychological Review, 96, 690–702. Mc Clelland, D., Atkinson, J.W., Clark, R.A., & Lowell, E.L. (1953). The Achievement Motive. NY: Appleton-CenturyCrofts. Mc Conaughy, S.H., & Achenbach, T.M. (1990). Contributions of developmental psychopathology to school services. In T.B. Gutkin & C.R. Reynolds (Eds.), The handbook of school psychology (2nd.ed., pp. 244 –268). New York: Wiley. Motta, R.W., Little, S.G., & Tobin, M.I. (1993). The use and abuse of human figure drawings. School Psychology Quarterly, 8, 162–169. Murray, H.A. (1943). Thematic Apperception Test manual. Cambridge: Harvard University Press. Murstein, B.I. (1963). Theory and research in projective techniques. New York: Wiley. Murstein, B.I. (1972). Normative written TAT responses for a college sample. Journal of Personality Assessment, 36, 109–147. Parker, C.H., Hanson, R.K., & Hunsley, J. (1988). MMPI, Rorschach and WAIS: A meta-analytic comparison of reliability, stability and validity. Psychological Bulletin, 103, 367– 373. Rabin, A.I. (1986). Concerning projective techniques. In A.I. Rabin (Ed.), Projective techniques for children and adolescents (pp. 3–11). New York: Springer. Reynolds, C.R., & Kamphaus, R.W. (1992). Behavior Assessment System for Children. Circle Pines, MN: American Guidance Service. Roberts, G.E., & Mc Arthur, D. (1982). The Roberts Apperception Test for Children. Los Angeles: Western Psychological Services. Sobel, H.J. (1981). Projective methods of cognitive analysis. In T. Merluzzi, C. Glass, & M. Genest (Eds.), Cognitive assessment (pp. 127–148). New York: Guilford. Teglasi, H. (1993). Clinical use of story telling. Needham Heights, MA: Allyn & Bacon. Thompson, C.E., & Bachrach, J. (1951). The use of color in the Thematic Apperception Test. Journal of Projective Techniques,15, 173–184. Vane, J.R. (1981). The Thematic Apperception Test: A review. Clinical Psychology Review, 1, 319–336. Vane, J.R., & Guarnaccia, V.J. (1989). Personality theory and personality assessment measures: How useful to the clinician? Journal of Clinical Psychology, 45, 5 –19. Veroff, J. (1968). Assessment of motives through fantasy. In C.P. Smith (Ed.), Achievement-related motives in children (pp. 226–227). New York: Russell Sage Foundation. Worchel, F.F., & Dupree, J.L. (1990). Projective storytelling techniques. In C. R. Reynolds & R.W. Kamphaus (Eds.), Handbook of psychological and educational assessment of children and adolescents: Personality, behavior and context. New York: Guilford. Copyright of Psychology in the Schools is the property of John Wiley & Sons, Inc. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s express written permission. However, users may print, download, or email articles for individual use. Professional Psychology: Research and Practice 2002, Vol. 33, No. 5, 454 – 463 In the public domain DOI: 10.1037//0735-7028.33.5.454 Effective Use of Projective Techniques in Clinical Practice: Let the Data Help With Selection and Interpretation Howard N. Garb James M. Wood VA Pittsburgh Healthcare System and University of Pittsburgh University of Texas at El Paso Scott O. Lilienfeld M. Teresa Nezworski Emory University University of Texas at Dallas By learning about the validity of individual test scores, psychologists can avoid using scores that are invalid and making judgments that are potentially harmful to their clients. This is important not only for improving clinical and forensic practice but also for avoiding lawsuits. In this article, the effective use of projective techniques is described, with attention to the overperception of psychopathology, diagnosis and the description of symptoms, and the detection of child abuse. Guidelines are offered for using the Rorschach, Thematic Apperception Test, human figure drawings, and the Washington University Sentence Completion Test. tific basis for justifying the use of Rorschach scales in psychological assessment.” Similarly, Grove and Barden (1999) concluded that expert witness testimony based on the Rorschach should not be found admissible in legal settings. In addition to receiving widespread attention in the psychological community, the controversy has been described in popular science and general news outlets, including Scientific American (Lilienfeld, Wood, & Garb, 2001) and The New York Times (Goode, 2001). In this article, we make recommendations for clinical practice in light of the growing controversy surrounding projective techniques. Topics include the overperception of psychopathology, the assessment of psychopathology (including diagnosis and the description of symptoms), and the detection of child physical and sexual abuse. A case history is presented to illustrate problems that can occur with the use of the Rorschach. Throughout this article, we offer guidelines for the use of projective techniques. Criticism of projective techniques, especially the Rorschach, has mounted in the past few years. For example, after conducting a comprehensive review of the research literature, Hunsley and Bailey (1999, p. 266) concluded that “there is currently no scien- HOWARD N. GARB received a double-major PhD in clinical psychology and in psychological measurement and research methodology from the University of Illinois at Chicago in 1984. He is coordinator of the Anxiety and Adjustment Disorders Clinic at the VA Pittsburgh Healthcare System and clinical associate professor at the University of Pittsburgh. His research interests center on psychological assessment and clinical judgment. JAMES M. WOOD received his PhD in clinical psychology from the University of Arizona in 1990. He is an associate professor of psychology at the University of Texas at El Paso. His research focuses on professional decision-making and eyewitness suggestibility in forensic settings, particularly in cases of child abuse. He has also written extensively about the scientific bases of the Rorschach Inkblot Test. SCOTT O. LILIENFELD received his PhD in clinical psychology from the University of Minnesota in 1990. He is an associate professor of psychology at Emory University in Atlanta and editor of the journal The Scientific Review of Mental Health Practice. His primary interests are the assessment of personality disorders and traits, the etiology of antisocial behavior, psychiatric classification and diagnosis, and the problem of pseudoscience in the mental health fields. M. TERESA NEZWORSKI received a dual PhD in clinical psychology and experimental child psychology from the University of Minnesota in 1983. She is an associate professor in the School of Brain and Behavioral Sciences at the University of Texas at Dallas and the director of Psychological Services at the Callier Center. She has a joint appointment as a clinical professor in the Graduate School of Biomedical Sciences at the University of Texas Southwestern Medical Center. She is currently working in the area of behavioral medicine, with projects that include examination of successful coping in cochlear implant patients and personality adjustment following closed head injury. CORRESPONDENCE CONCERNING THIS ARTICLE should be addressed to Howard N. Garb, Behavioral Health (116A-H), VA Pittsburgh Healthcare System, 7180 Highland Drive, Pittsburgh, Pennsylvania 15206-1297. E-mail: [email protected] Case History Failure to recognize the shortcomings of projective techniques, particularly the Rorschach, can lead to poor assessment decisions and harm to clients. The types of problems that can arise are illustrated by the following case history, which describes a psychologist who lost his license after a former girlfriend complained to the state licensing board that he was abusing alcohol.1 After his license was suspended for a year, he was required to undergo psychological testing. Testing indicated that he appeared to have severe psychopathology, and he was unable to regain his license. Finally, his lawyer asked one of us (Howard N. Garb) to review the psychological testing. The case history is described in greater detail below. We refer to it in later sections to illustrate the importance of following guidelines for improving clinical practice. 1 Appreciation is expressed to “Dr. B” for granting us permission to include his case history. Certain facts have been altered to protect his identity (e.g., B is not the initial for his last name). 454 PROJECTIVE TECHNIQUES Dr. B is in his mid- to late 60s. He has a PhD in clinical psychology from a Big Ten University. Although elderly, he continues to work. Several years ago, a former girlfriend told the state licensing board that he was abusing alcohol, thereby implying that he was impaired. The woman had been his student for 2 years. He had started to see her socially 2 years after she graduated from college. While the complaint was being investigated, Dr. B entered into a troubled relationship (with a different woman) and began abusing liquor. He became depressed, was diagnosed as having major depression, and was admitted to a psychiatric hospital. This was his first and only psychiatric hospitalization. Dr. B was placed on an antidepressant, entered psychotherapy, and joined Alcoholics Anonymous. He voluntarily told the state board about his psychiatric hospitalization. His psychotherapist told them that Dr. B. had been impaired for the 3 weeks surrounding the time of his hospitalization (but that he was not impaired afterward). His license was suspended for a year. He agreed to remain in psychotherapy during this time and to undergo psychological testing at the end of his one-year suspension. At the end of one year, Dr. B was given a battery of psychological tests. The psychologist who conducted the testing has a prestigious reputation and is a member of the Rorschach Research Council, which consults with John Exner. With regard to neuropsychological test results, Dr. B was described as being “quite sharp” and “entirely normal.” On the Minnesota Multiphasic Personality Inventory—2 (MMPI–2), he was described as being “essentially normal.” However, on the Rorschach, he was described as having severe psychopathology. As a result, the state board would not return his license. At the request of the attorney for Dr. B, a legal affidavit was prepared based on empirical research. The affidavit described problems with the portions of the test report that were based on the Rorschach. Within a few days, and without any further litigation, a settlement was reached. After his license was returned, Dr. B commented on how his Rorschach results had made him feel: “The whole time there was a feeling that my peers didn’t like me. My self-worth suffered immeasurably. In therapy, I had to learn to redefine myself as someone who is not only a psychologist. I have some scars and bruises and I always will.” To understand why the use of the Rorschach can lead to the overperception of psychopathology, one needs to become familiar with problems that are associated with the norms for Exner’s (1991, 1993) Comprehensive System (CS). Problems with the CS norms, along with recommendations and guidelines for clinical and forensic practice, are described in the following section. The Rorschach and the Overperception of Psychopathology Clinical Guideline 1: Exercise caution when using the CS norms, as research indicates that their use is related to the overperception of psychopathology. In many instances, it may be best not to use the CS norms. The CS is the most popular system for using the Rorschach. In addition to providing detailed rules for administration, scoring, and interpretation, the CS provides sets of norms to describe the results for relatively normal children and adults. The CS scores discussed in this article, along with brief descriptions of the constructs the scores purportedly measure, are listed in Table 1. Table 1 Comprehensive System Scores Discussed in the Article and the Constructs They Purportedly Measure CS score Construct purportedly measured EB style (% of ambitent protocols) Pairs (2) and Reflections Conventional Form (X ⫹ %) Distorted Form (X ⫺ %) Affective Ratio (Afr) Form-Color (FC) Populars Diffuse Shading (Y), sum of Y Texture (T) WSumC Morbid Content (MOR) WSum6 Lambda Pure Human Aggressive Movement (AG) Aggressive Content (AgC) Color (C) Vista (V) Space (S) Adjusted es Inefficient problem solving Self-absorbed, inflated sense of self-worth Conventional and realistic perceptions Distorted perceptions of reality Withdrawal from, or overresponsiveness to, affect Emotional control Conventional modes of thinking Anxiety, constrained expression of emotion Need for affection and dependency Emotional control Pessimism, negative self-image Cognitive distortion or slippage Emotional responsiveness Empathy, interest in people Aggressiveness, hostility Aggressiveness, hostility Emotional control Self-critical introspection Difficulty in handling anger, negativism A chronic condition involving overstimulation (e.g., racing thoughts) and difficulty organizing thoughts Ability to perceive complexity Depression A person’s typical capacity to tolerate stress (minus the influence of current environmental stressors) Current capacity to tolerate stress Interpersonal skills Constrained emotional expressiveness Blends Depression Index (DEPI) Adjusted D D score Coping Deficit Index (CDI) Achromatic Color (C⬘) 455 456 GARB, WOOD, LILIENFELD, AND NEZWORSKI The adult CS normative sample was recently revised after Exner was told that he had made an error of large magnitude. Of the 700 protocols in the 1993 adult normative sample, 221 were duplicates (Exner, 2001, p. 172; J. E. Exner, personal communication, March 23, 2001). That is, the sample of 700 protocols was actually composed of 479 distinct protocols, with 221 protocols counted twice. Although the adult normative sample has been revised, even the 2001 sample has not been found to be error free. Compelling evidence exists that the CS norms for form quality are based on the wrong scoring rules and have been seriously in error since 1983 (Meyer & Richardson, 2001). Form quality refers to how well a client’s responses are related to the form of an inkblot. A disturbed client may disregard the form of an inkblot when creating a response. By comparing a client’s Rorschach scores with the CS norms, one should be able to detect psychopathology. If a client’s scores differ from those for a group of relatively normal individuals, one should be able to infer that psychopathology is present. However, research evidence suggests that the use of the CS norms causes psychologists to overperceive psychopathology. In many ways, this problem raises perhaps the most serious question to date regarding the Rorschach’s clinical utility. Because the results on norms are important for clinical practice, we describe them in detail here. In a landmark study (Shaffer, Erdberg, & Haroian, 1999), the Rorschach was administered to 123 “nonpatient” adults. CS rules for administration were followed. Participants did not have a major medical illness, had never had a psychiatric hospitalization, had not received psychological treatment in the past 2 years, were never convicted of a felony, and had not undergone psychological testing in the past year. On many important Rorschach variables, participants obtained scores that were substantially different from the CS norms. Nearly all the discrepancies tended to make the participants appear pathological. For example, if one used the CS norms, one would conclude that the participants were probably seriously disturbed on measures of perceptual inaccuracy and distorted thinking. For Distorted Form (X ⫺ %), the mean score for the Shaffer et al. sample was more than 2 standard deviations higher than the mean score listed for the CS norms (0.21 vs. 0.07). For Conventional Form (X ⫹ %), the mean score for the Shaffer et al. sample was more than 3 standard deviations lower than the mean score listed for the CS norms (0.51 vs. 0.79). Negative findings have also been reported for the CS norms for children (Hamel, Shaffer, & Erdberg, 2000). Hamel et al. administered the Rorschach to a group of relatively healthy children. Three children were excluded because they had received psychotherapy for emotional or behavioral disorders, 5 were excluded because they had been evaluated or treated for attention-deficit/ hyperactivity disorder, and 1 child was excluded because of a history of having been suspended from school more than once. The children included in this study (N ⫽ 100) demonstrated healthier than average behaviors, as measured by the Conners’ Parent Rating Scale–93 (Conners, 1989). When the Rorschach was administered, striking discrepancies from the CS norms were obtained. As noted by Hamel et al. (2000): tion. In the main, these children may be described as grossly misperceiving and misinterpreting their surroundings and having unconventional ideation and significant cognitive impairment. Their distortion of reality and faulty reasoning approach psychosis. These children would also likely be described as having significant problems establishing and maintaining interpersonal relationships and coping within a social context. They apparently suffer from an affective disorder that includes many of the markers found in clinical depression. (p. 291) To determine if discrepancies from the CS norms can be found in other samples, Wood, Nezworski, Garb, and Lilienfeld (2001a) reviewed Rorschach studies that included groups of nonpatient adults. In these studies, a clinical group (e.g., clients with an anxiety disorder) was compared with a control group (a nonpatient comparison group). The nonpatient adult comparison groups were often composed of undergraduate students or community volunteers, although other groups were used as well (e.g., “normal married women”). Thirty-two studies were located. For the nonpatient comparison groups, the results for the following 14 CS variables were examined: EB style (percentage of ambitent protocols), Reflection responses, Conventional Form (X ⫹ %), Distorted Form (X ⫺ %), Affective Ratio (Afr), Form-Color (FC), Populars, sum of Diffuse Shading (sum of Y), sum of Texture (sum of T), WSumC, Morbid Content (MOR), WSum6, Lambda, and Pure Human. These 14 indexes were selected because they (a) are crucial to CS interpretation, (b) exhibited substantial discrepancies from the CS norms in the study by Shaffer et al. (1999), and (c) had been examined repeatedly in the 32 studies. For all 14 scores, the nonpatient adults appeared to exhibit significant psychopathology when their results were compared with the CS norms. All differences were statistically significant. The median difference between the comparison groups and the CS norms was large in size (d ⫽ .73).2 Wood et al. (2001a) concluded the following: If Rorschach scores for a normal adult are interpreted using the CS norms, the adult will appear relatively self-focused and narcissistic (elevated Reflection scores), unconventional with impaired judgment and distorted perceptions of reality (low X ⫹ %, low Populars, high X ⫺ %), depressed, anxious, tense, and constrained in emotional expression (elevated MOR, elevated sum Y, low WSumC), insecure and fearful of involvement (elevated Lambda), vacillating and inefficient (elevated number of ambitents), with low empathy (low Pure H), a tendency to withdraw from emotions (low Afr), and poor emotional control (low FC). (p. 356) Some psychologists have argued that the use of the CS norms does not lead to the overperception of psychopathology. For example, Meyer (2001b) analyzed the results from nine international CS studies that involved a total of 2,125 nonclinical participants. Instead of analyzing the results for the 14 variables that we examined, he analyzed the results for 69 Rorschach scores. He found that the international sample was only about four tenths of a standard deviation more impaired than the CS normative sample. Wood et al. (2001b) requested Meyer’s (2001b) data and analyzed the results for the 14 variables that they had examined in their A median value of d ⫽ .73 indicates that for the median comparison, the mean for the clinical group was .73 of a standard deviation from the mean for the comparison group. 2 If we were writing a Rorschach-based, collective psychological evaluation for this sample, the clinical descriptors would command atten- PROJECTIVE TECHNIQUES review. The findings for the international sample were remarkably similar to the findings that had been obtained for the aggregated sample of nonpatient participants in 32 studies. For the 14 variables, the international sample was about eight tenths of a standard deviation more impaired than the CS normative sample. Thus, for these 14 critical variables, the results are even more striking for the international sample than for the aggregated nonpatient sample. The controversy concerning the CS norms is important because it suggests that psychologists using the Rorschach have been systematically overperceiving psychopathology. With regard to forensic practice, it is troubling to realize that the use of the CS norms may contribute to detrimental outcomes, such as a parent being unfairly denied child custody or a prisoner being denied parole. Implications for clinical practice are also important. Some psychologists may have been treating clients for problems they did not have (e.g., narcissism). Worse yet, in some cases, they may have persuaded clients that they had problems they did not really have. However, these errors may be less likely to occur when the Rorschach is used as an aid for exploration in psychotherapy rather than as an assessment device (Wood, Garb, Lilienfeld, & Nezworski, 2002), a point that we address later. In conclusion, empirical evidence indicates that the use of the CS norms leads to the overperception of psychopathology. This result has been consistently obtained across well-designed studies conducted by independent investigators. When a client’s scores deviate significantly from the CS norms, one should not necessarily infer that psychopathology is present. Application to Case History Although Dr. B appeared pathological when his Rorschach scores were compared with the CS norms, he does not appear pathological when his scores are compared with results obtained by researchers other than Exner. For example, Dr. B had a score of 4 on the Rorschach Experience Actual (EA). According to the Exner (1993, 2001) norms, such a score is abnormally low, and according to Weiner (1998, p. 140), “individuals in whom EA ⬍ 6 usually have limited coping resources and are more likely than most people to meet life’s demands in an inept and ineffective manner.” Using the Exner (1993, 2001) norms, one would infer that Dr. B is over 2 standard deviations below the mean score for normal adults. However, according to the findings of researchers other than Exner, an EA score of 4 is not abnormal. For example, in their sample of nonpatient adults, Shaffer et al. (1999) found that EA had a mean of 6.26, with a standard deviation of 3.71. Using samples of older adults, one would also conclude that an EA score of 4 does not reveal psychopathology. Paul (1989) reported a mean of 4.82 (SD ⫽ 3.37) among nonpatients who were 65–94 years old. Erstad (1996) reported a mean of 4.56 (SD ⫽ 3.13) for elderly normal adults who were 61–95 years old. As may be seen, many of the normal adults in these studies would be described as abnormal if their Rorschach results were interpreted using the CS norms. Projective Techniques and the Assessment of Psychopathology Clinical Guideline 2: Use scores that are valid for their intended purposes. Scores should be validated in well-designed studies, 457 results should be consistent, and positive findings should be replicated by independent investigators. It is often argued that projective techniques should be used as part of a test battery and that results from the testing should be integrated with history and interview information. This practice is seen as a safeguard: Conventional wisdom dictates that psychologists should weigh test results only if they are in agreement with other results (or, perhaps, only if they make sense in the context of other results). An alternative view is that psychologists should use projective techniques only if indexes have been shown to be valid for their intended purposes. According to this argument, one should not use an invalid test score even if results for this test score are in agreement with findings from other sources of information. If projective results are in agreement with other results, such as interview and test results, then the projective results will tend to make psychologists more confident in their judgments, even if the projective indexes are not valid for this task. If the projective results are not in agreement with other information, psychologists may believe that hypotheses generated by using the other information have not been confirmed, even though invalid projective indexes should not have an effect on their judgments. In research studies, psychologists have frequently become less accurate when projective test information has been made available in addition to other information, although decreases in accuracy have not always been statistically significant (Garb, 1998). For example, in one study (Whitehead, 1985), psychologists and advanced graduate students made diagnoses using (a) the MMPI alone, (b) the Rorschach alone, and (c) the Rorschach and MMPI together. The CS was used to administer, score, and interpret the Rorschach. Judgment tasks were to differentiate (a) back pain patients from psychiatric hospital patients with diagnoses of schizophrenia or bipolar disorder, (2) depressed back pain patients from nondepressed back pain patients, and (3) psychiatric patients with bipolar disorder from psychiatric patients with schizophrenia. When the results for psychologists and advanced graduate students were pooled across judgment tasks, the average hit rates were 76% for the MMPI alone, 58% for the Rorschach alone, and 74% for the Rorschach and MMPI together. Differences between psychologists and advanced graduate students were not statistically significant. Research on using projective techniques to make diagnoses and describe symptoms is presented next. In light of this research, additional comments will be made about the case history of Dr. B. Research on the Assessment of Psychopathology To evaluate the validity of an index, we have proposed using the following criteria: (a) An index must demonstrate a consistent relation to a particular disorder, trait, or symptom; (b) results must be obtained in methodologically adequate studies; and (c) findings must be independently replicated (Wood et al., 1996b). We acknowledge that isolated positive findings have been obtained for a wide range of projective indicators. This result is not surprising given the enormous number of studies conducted on projective techniques. However, using our criteria, the existence of positive findings is not sufficient to result in a recommendation unless those findings are from sound studies and have been replicated by other researchers. 458 GARB, WOOD, LILIENFELD, AND NEZWORSKI With regard to the Rorschach, results show that there are problems with using the CS to assess psychopathology. Only a few CS indexes appear to be well supported (Lilienfeld, Wood, & Garb, 2000). Poor form quality and deviant verbalizations can be used to detect thought disorder. They can also provide helpful information for the diagnosis of schizophrenia and other mental disorders in which thought disorder is sometimes present (e.g., bipolar disorder, schizotypal personality disorder). In addition, the Rorschach Oral Dependency Scale (Masling, Rabie, & Blondheim, 1967) has been reasonably well supported for the assessment of dependent personality traits. In general, unique patterns of results on the Rorschach have not been observed for specific mental disorders. Investigators have searched with little success for Rorschach indexes that show a clear relation to major depressive disorder, anxiety disorders, dissociative identity disorder, conduct disorder, psychopathy, and dependent, narcissistic, and antisocial personality disorders. Positive findings that have been obtained have rarely been replicated (Wood, Lilienfeld, Garb, & Nezworski, 2000). For example, the Depression Index (DEPI; Exner, 1991, 1993) has been the most extensively studied Rorschach indicator of depression. According to Exner (1991, p. 146), an elevated score on the DEPI “correlates very highly with a diagnosis that emphasizes serious affective problems.” However, independent investigators have generally reported that diagnoses of depression are not significantly related to scores on either the original or revised versions of the DEPI (for a detailed review, see Jorgensen, Andersen, & Dam, 2000). This is true for both adults and adolescents. For example, six of eight studies conducted independently of the Rorschach Workshops found no significant relation between the revised DEPI and psychiatric diagnoses (Archer & Krishnamurthy, 1997; Ball, Archer, Gordon, & French, 1991; Caine, Frueh, & Kinder, 1995; Meyer, 1993; Ritsher, Slivko-Kolchik, & Oleichik, 2001; Sells, 1990/ 1991; see also Meyer, 2001a), one study yielded mixed results (Ilonen et al., 1999), and only one study yielded unmixed positive results (Jansak, 1996/1997). Some psychologists believe that Rorschach results can be used along with other diagnostic information to detect antisocial personality disorder (ASPD). For example, Gacono and Meloy (1994, pp. 108 –117, 157–169) compared the CS norms with results for individuals with ASPD. They concluded that individuals with ASPD obtain distinctive and pathological scores on the Rorschach. However, their conclusions are highly problematic because we now know that many individuals in the community, with no known pathology, also score in a pathological direction when their scores are compared with the CS norms (Hamel et al., 2000; Shaffer et al., 1999; Wood et al., 2001a, 2001b). Gacono and Meloy (1994) argued that a large number of Rorschach variables are related to ASPD. However, some of the studies that they cited compared the results for individuals with ASPD to the CS norms. This is a problem because even the results for relatively healthy individuals can be expected to differ from the CS norms. To learn if individuals with ASPD display a unique pattern of results on the CS, their results should be compared with results for nonpatient comparison groups. If one excludes the studies that compared results with the problematic CS norms, it becomes clear that no Rorschach variable (except perhaps Pair responses) has shown a well-demonstrated relationship to ASPD (Wood et al., 2000). Failures to replicate positive findings have been reported for the following scores: Aggressive Movement (AG; Baity & Hilsenroth, 1999; Berg, Gacono, Meloy, & Peaslee, 1994), T (Berg et al., 1994; Blais, Hilsenroth, & Fowler, 1998; Howard, 1998/1999), Aggressive Content (AgC; Baity & Hilsenroth, 1999; Berg et al., 1994), Color (C; Berg et al., 1994; Blais et al., 1998), Pure Shading Response (Y) and Vista (V; Gacono, Meloy, & Berg, 1992; Howard, 1998/1999), and Space (S) and Pure Human responses (Howard, 1998/1999). Similarly, attempts to replicate positive Rorschach findings for the assessment of psychopathy and conduct disorder have nearly always failed (Wood et al., 2000). The Thematic Apperception Test (TAT), like the Rorschach, has infrequently been well supported for assessing psychopathology. For example, in one study (Sharkey & Ritzler, 1985), TAT measures of perceptual distortions, unusual story interpretations, and affect tone did not significantly distinguish samples of normal, depressed, and psychotic individuals. Furthermore, results have been only mixed for the most promising objective scoring system, the Social Cognition and Object Relations Scale (SCORS; Westen, Lohr, Silk, Gold, & Kerber, 1990). The SCORS was designed to assess object relations (i.e., a client’s mental representation of other people), and it yields scores on several dimensions, including moral standards, aggression, and affect–tone. In one study that used the SCORS (Ackerman, Clemence, Weatherill, & Hilsenroth, 1999), patients with ASPD did not differ significantly from patients with other personality disorders on the moral standards variable. Similarly paradoxical results were found for the SCORS aggression variable. In contrast, research provides provisional support for using the SCORS to detect borderline personality disorder (e.g., Gutin, 1997; Malik, 1992; Westen et al. 1990). Although research provides provisional support for using the SCORS to detect borderline personality disorder, clinicians should not conclude that these findings provide support for the routine clinical use of the TAT. Almost all clinicians who use the TAT rely exclusively on subjective interpretations and do not use an objective scoring system (Ryan, 1985; Vane, 1981). Subjective interpretations of the TAT are difficult to defend in light of the paucity of research support for their validity (Lilienfeld et al., 2001). Human figure drawings are also used to help in assessing psychopathology. There are two major approaches to scoring and interpreting human figure drawings. Using the sign approach, one draws inferences from isolated drawing features. For example, if a client draws a small human figure, a clinician might infer that the person is likely to be depressed (an inference that is commonly made but is unlikely to be correct; Joiner, Schmidt, & Barnett, 1996). In contrast, the global approach involves basing a judgment on all of the features of a drawing. A number of reviewers have concluded that the sign approach typically yields inferences that possess negligible or zero validity (e.g., Kahill, 1984; Klopfer & Taulbee, 1976; Motta, Little, & Tobin, 1993; Thomas & Jolley, 1998). Results have been somewhat better for the global approach. For example, the Draw-A-Person: Screening Procedure for Emotional Disturbance (DAP:SPED; Naglieri, McNeish, & Bardos, 1991) has been normed on 2,622 children, ages 5–17, using U.S. Census data to stratify the sample on a number of demographic variables. It can be used as a screening measure for global psychopathology. In one study (Naglieri & Pfeiffer, 1992), the DAP: SPED was used to discriminate 54 normal students from 54 PROJECTIVE TECHNIQUES students with conduct and oppositional disorders. Cutoff scores suggested by Naglieri et al. (1991) were used to make discriminations. Validity was significantly better than chance, but the use of the DAP:SPED led to a large number of classification errors. Correct ratings were made for 77.8% of the normal participants and 48.2% of the clinical sample. Our findings on projective techniques differ sharply from those reached by the Psychological Assessment Work Group (PAWG), which was commissioned by the Board of Professional Affairs of the American Psychological Association to address issues related to the declining popularity of psychological assessment (Kubiszyn et al., 2000). In a recent issue of this journal, PAWG concluded that the Rorschach and the TAT are valuable for the assessment of psychopathology. However, although PAWG consistently described positive findings for the Rorschach and TAT, they almost always omitted negative findings. The PAWG article did not mention the current controversy surrounding projective techniques or the problems with the CS norms. In conclusion, one must be extremely cautious when using projective techniques to assist in assessing psychopathology. Research indicates that the Rorschach can be helpful for detecting conditions characterized by thought disorder (e.g., schizophrenia; see Wood et al., 2000). It is also supported for evaluating dependent personality traits. However, research does not support the Rorschach’s use for diagnosing most mental disorders, including major depression, conduct disorder, panic disorder, and antisocial and narcissistic personality disorders. For the TAT, research provides provisional support for using the SCORS (Westen et al., 1990) to detect borderline personality disorder, but the validity of the TAT has not been well established for other tasks involved in the assessment of psychopathology. In particular, there is a scarcity of scientific support for the commonplace reliance on clinical intuition in TAT interpretation. Finally, with regard to projective drawings, there is some evidence that the DAP:SPED (Naglieri et al., 1991) can be used to screen for mental disorders among children, but the overwhelming majority of human figure drawing signs appear to possess negligible or essentially zero validity (e.g., Thomas & Jolley, 1998). Application to Case History The psychologist who administered the Rorschach to Dr. B used a computer program to interpret the results. The computer printout specified the scores on which different interpretive statements were based. Rorschach scores that formed the basis for the negative characterization of Dr. B have received little support in the research literature. For example, Dr. B was described as “overly complex in his cognitions and personality functioning.” This assertion rests on Dr. B’s high Adjusted es score and his high number of Blends. Exner (1993) did not cite a single study to support the validity of Adjusted es (pp. 379 –380). Regarding the number of Blends, Exner (1993) presented no research evidence that Blends are related to overcomplexity in cognitions or personality functioning (pp. 140 –145, 501–504). The hypothesis that Blends are related to overcomplexity is apparently based exclusively on Exner’s own authority. One of the most damaging comments made about Dr. B in the psychologist’s report was that he is “easily overwhelmed by ex- 459 ternal stress secondary to the excess of internal stress which characterizes him.” This statement seems to be based on the following scores: the DEPI, Adjusted D, D score, Coping Deficit Index (CDI), Achromatic Color (C’), and Texture (T). We have already cited negative findings for the DEPI. With regard to the D score and the Adjusted D, two broad problem areas have been noted (Kleiger, 1992; Wood, Nezworski, & Stejskal, 1996a; but also see Exner, 1992). First, about half of the studies are unpublished. Second, the findings of the published studies are equivocal. For example, Kleiger (1992, p. 293) noted that some data (Exner, 1974) are described in an “incomplete” and a “confusing manner,” making it difficult to determine if Exner’s conclusions for these variables follow from his data. Also, the results of a published study (Wiener-Levy & Exner, 1981) had been interpreted as providing support for the validity of the D score and the Adjusted D, even though those results contradicted results from earlier studies (Exner & Bryant, 1975, 1976). Finally, very few studies have been conducted on the relation between stress and C’, T, and the CDI. For the few studies that have been conducted, the results have generally been negative. For example, Frueh and Kinder (1994) reported that Vietnam veterans with posttraumatic stress disorder did not differ significantly from normal male undergraduates on T. Detection of Physical and Sexual Abuse Clinical Guideline 3: Do not use the Rorschach, TAT, or human figure drawings to detect child physical or sexual abuse. One of the most controversial tasks in personality assessment involves using psychological tests to detect physical or sexual abuse. This issue is discussed here because of its obvious social and clinical importance. PAWG concluded that the “Rorschach or the TAT” are capable of “differentiating patients who have experienced physical or sexual trauma from those who have not” (Kubiszyn et al., 2000, p. 121). However, they did not cite any research to support their conclusion about physical abuse and cited only one study to support their claim about sexual trauma (Leifer, Shapiro, Martone, & Kassem, 1991). Although PAWG cited only one study on the detection of child sexual abuse, West (1998) located 12 studies in which projective techniques had been used for this purpose. She conducted a metaanalysis and concluded that “projective techniques have the ability to discriminate between children who have been sexually abused and those who were not abused sexually” (p. 1151). The meta-analysis conducted by West (1998) is seriously flawed. Although it was never made explicit in her article, West included only positive results in her meta-analysis. She excluded negative results even when they were reported in the same articles as the positive results. Thus, only by excluding negative results was West able to conclude that projective techniques are valid for the detection of child sexual abuse. Data from the 12 studies located by West (1998) were reanalyzed (Garb, Wood, & Nezworski, 2000). All of the data, not just the positive findings, were entered in a meta-analysis. With regard to results, the overall effect size (d) was estimated to be between .35 and .46, indicating that the mean score for sexually abused children was about .35–.46 of a standard deviation from the mean score for nonsexually abused children. The following conclusions were reached (Garb, Wood, & Nezworski, 2000): 460 GARB, WOOD, LILIENFELD, AND NEZWORSKI Clinicians should not use test indicators until positive findings have been replicated by independent investigators. Most of the positive findings on detecting sexual abuse have not yet been replicated. There have been a few exceptions involving the Rorschach and Human Figure Drawings, but even these indicators are in need of further study because the findings that were replicated involved comparisons between sexually abused children and children who were not being seen by a mental health professional. (p. 166) Thus, even the few indicators with positive results may not be useful for discriminating sexually abused children from children likely to be seen in a mental health clinic. Similar results were obtained in a second meta-analysis (Garb, Wood, & Lilienfeld, 2000). This meta-analysis was broader in scope and included results from unpublished manuscripts. With the exception of Westen et al.’s (1990) TAT SCORS, no evidence was found that a projective technique indicator could consistently discriminate sexually abused children from nonabused children who are receiving mental health treatment. Although the use of the SCORS to detect sexual abuse is promising and we encourage additional research on its use, it is important to note that adequate normative data are not available. Also, results for the SCORS have varied widely across mental health settings, suggesting that separate cutoff scores will need to be established for individual sites. In conclusion, the use of projective techniques for the detection of child physical and sexual abuse can be harmful. If a psychologist incorrectly concludes that a child has been physically or sexually abused, or incorrectly determines that a child has not been abused, these erroneous decisions may cause considerable suffering and pain for the child and the child’s family as well as for other persons. We are particularly concerned that the use of the Rorschach will lead psychologists to make inaccurate and harmful judgments. As noted earlier, many normal children appear maladjusted when their results are compared with the CS norms. A child may score in a pathological direction because the CS norms are flawed, not because the child has been sexually or physically abused. The Need for Flexible Standards Clinical Guideline 4: Use projective techniques differently depending on whether one is testifying in court as an expert witness, evaluating a client in clinical practice, or using a projective technique as an aid for exploration in psychotherapy. Standards for forensic practice are more rigorous than standards for clinical practice. Similarly, standards should be more rigorous for conducting a psychological evaluation than when using assessment information as an aid for exploration in psychotherapy. Psychologists should only use indexes that have been supported by research, but they can be relatively liberal or conservative in making interpretations depending on the setting (forensic vs. clinical) and the task (formal evaluation vs. generating material for psychotherapy). For example, when using a projective technique as an aid for exploration in psychotherapy, a psychologist may want to discuss an interpretation with a client that is based on the findings from a well-designed study even though the findings have not yet been replicated by an independent investigator. The psychologist should not assume that the interpretation is correct but instead should use this as an opportunity to explore an important issue with the client. Rigorous standards require that interpretations be based on strong empirical evidence. For this reason, expert witnesses in legal settings should expect to be challenged if they (a) use the current CS norms, (b) interpret scores that are not valid for their intended purposes, or (c) use the Rorschach, TAT, or human figure drawings to help detect child physical or sexual abuse. In psychotherapy, a tentative, exploratory approach can be beneficial. In fact, Aronow (2001, p. 384) recommended that the Rorschach be used “as part of the therapy process, particularly when logjams in the therapy are encountered. ” However, when using projective techniques to generate material for therapy sessions, psychologists must be careful not to fall prey to confirmatory bias (Garb, 1998). That is, in addition to considering information that confirms an interpretation, psychologists must weigh information that serves to refute it. Additional Recommendations and Guidelines Perhaps the greatest pitfall to be avoided when using projective techniques is accepting the validity of the CS at face value (Exner, 1991, 1993, 2001). As already discussed, use of the CS norms for interpreting Rorschach protocols can lead psychologists to conclude that relatively normal individuals have severe psychopathology. Also, as illustrated in our discussion of the case history, some CS variables have negligible or essentially zero validity for their intended purposes. Thus, psychologists who interpret Rorschach protocols “by the book” are likely to make incorrect and potentially harmful inferences. With regard to the CS, professional psychologists are left in a difficult predicament. One cannot expect clinicians to become familiar with the research literature on every CS variable. Their schedules will not permit it. But neither can they simply accept the validity of CS scores at face value. In contrast to the CS, a number of textbooks that serve as interpretive manuals for the MMPI–2 have been written by authors who have been willing to adopt a critical stance. For example, Greene (2000) was critical of over half of the MMPI–2 supplementary scales. Consider the following statements from his book: There is virtually no research on the Es [Ego Strength] scale on the MMPI–2. The inconsistent findings in the earlier research may have dissuaded most researchers from investigating the scale. (p. 227) Clinicians should be very cautious in using the MAC [MacAndrew Alcoholism] scale in nonwhite ethnic groups. (p. 230) Research on the O-H [Overcontrolled–Hostility] scale has been very mixed. (p. 246) Research on the Do [Dominance] scale has been exceedingly sparse. (p. 248) There has been virtually no research on the Re [Social Responsibility] scale. (p. 249) Research on the Mt [College Maladjustment] scale is almost nonexistent. (p. 250) [Post Traumatic Stress Disorder Scales] are measures of general distress with little specificity for Post Traumatic Stress Disorder (p. 253) PROJECTIVE TECHNIQUES Any method using the MMPI–2—whether it involves single scales, profile analysis, Supplementary scales, or item analysis—appears disappointing in the prediction of suicide. (pp. 266 –267) This type of interpretive manual, one that adopts a genuinely critical attitude, is sorely needed for the CS. How should information from projective techniques be integrated with other information to make judgments and decisions about clients?3 In addition to providing guidelines for the use of the Rorschach, TAT, and human figure drawings, we describe the Washington University Sentence Completion Test (WUSCT; Loevinger, 1998) and make recommendations for its use. Although the Rorschach, TAT, and human figure drawings are the most commonly used projective techniques, the WUSCT is arguably the most extensively validated one (Lilienfeld et al., 2000). Our primary recommendations for conducting formal psychological evaluations are as follows. First, to detect emotional disturbance among children, psychologists can conduct interviews with children, teachers, and/or parents. If a screening instrument is needed, they can use the Child Behavior Checklist (Achenbach, 1978; Achenbach & Edelbrock, 1983) or the DAP:SPED (Naglieri et al., 1991). It should be noted, however, that the Child Behavior Checklist has greater empirical support than the DAP:SPED. For making diagnoses, psychologists should rely primarily on interview and history information, but results from psychological tests, including self-report personality inventories and projective techniques, can sometimes be helpful. For example, when diagnosing a severe mental disorder, one may find it helpful to assess for the presence of thought disorder. Using the Rorschach, one can do this by detecting deviant verbalizations and evaluating the form quality of responses. However, as already noted, serious problems exist with the CS norms for form quality and deviant thinking, such as X ⫹ %, X ⫺ %, and WSum6. To overcome this problem, one can use the Thought Disorder Index for the Rorschach (Solovay et al., 1986), although it is not part of the CS. One can also use the CS form quality variables but rely on local norms collected in one’s work setting, but this is an imperfect solution because one will still not have norms for individuals in the community who are not impaired. To assist with the diagnosis of borderline personality disorder, one can use the SCORS (Westen et al., 1990) to score and interpret a client’s TAT protocol. The SCORS dimensions have demonstrated encouraging construct validity for the diagnosis of borderline personality disorder (Lilienfeld et al., 2000), but they are difficult to score. Also, adequate norms are not available. This makes it difficult to use the SCORS in clinical practice because clinicians will not know what cutoff scores to use. To predict behavior, psychologists should rely heavily on history and interview data. It may also be possible to use projective techniques to predict certain behaviors of relevance to clinical and counseling psychologists. The Rorschach Prognostic Rating Scale (RPRS; Klopfer, Kirkner, Wisham, & Baker, 1951) has been used for the prediction of psychotherapy outcome, and the McClelland, Atkinson, Clark, and Lowell (1953) TAT scoring system for assessing achievement needs has been used to assist in the prediction of occupational success. However, little research has been conducted on the RPRS since the early 1980s, up-to-date norms are not available, and some investigators have argued that the extant research is of poor methodological quality (Hunsley & 461 Bailey, 1999, p. 274; Shields, 1978). Also, neither of these indexes can be scored easily by clinicians (e.g., the RPRS requires that the Rorschach be administered and scored using the now rarely employed Klopfer system). Moreover, the correlations between McClelland et al.’s scoring system for achievement motivation and real-world achievement are relatively low (Spangler, 1992). To evaluate psychiatric symptoms and personality traits, clinicians should rely on interview and history information, brief selfrated and clinician-rated measures (e.g., measures of panic frequency and severity), self-report personality inventories, and, in selected instances, projective techniques. Assessing symptoms and personality traits is supposed to be the ideal task for projective techniques. However, even for this task, the findings for the Rorschach, TAT, and human figure drawings have rarely been independently and consistently replicated. One exception involves using Rorschach indexes to evaluate thought disorder. Another involves the use of the Rorschach Oral Dependency Scale. This measure can be used to detect dependency, although it is not known if its addition to interview and history information leads to an increase in validity. This index is not currently part of the CS. In contrast to other projective techniques, the WUSCT (Loevinger, 1998) has been extensively validated for evaluating personality. The WUSCT consists of 36 incomplete sentence stems (e.g., “Sometimes she wished that . . .”). Separate forms are used for male and female participants. The test has been well-validated as a measure of ego development (e.g., Lilienfeld et al., 2000, pp. 55–56; Manners & Durkin, 2001). It can be used to evaluate a range of traits, including impulse control, moral development, cognitive style, interpersonal style, and conscious preoccupations. However, some questions remain unresolved. The test has infrequently been used in clinical practice, and research needs to be conducted to demonstrate its utility for this purpose. For example, research can determine if the WUSCT provides helpful information for diagnosing personality disorders. In conclusion, from a scientific standpoint, the use of projective techniques is highly controversial. Although they can be used appropriately and should continue to be the subject of research, there are significant problems with the manner in which they are at times used. In general, psychologists will be on safer ground when they use projective techniques as an aid for exploration in psychotherapy rather than as an assessment device. The PAWG report’s conclusions notwithstanding, psychologists would be well advised to proceed with great caution when using these instruments in clinical and forensic practice. 3 Because we focus on the valid use of projective techniques, a discussion of objective personality tests is beyond the scope of this article. References Achenbach, T. M. (1978). The Child Behavior Profile: I. Boys ages 6 –11. Journal of Consulting and Clinical Psychology, 46, 478 – 488. Achenbach, T. M., & Edelbrock, C. (1983). Manual for the Child Behavior Checklist/4 –18 and Revised Child Behavior Profile. Burlington: University of Vermont, Department of Psychiatry. Ackerman, S. J., Clemence, A. J., Weatherill, R., & Hilsenroth, M. J. (1999). Use of the TAT in the assessment of DSM–IV Cluster B personality disorders. Journal of Personality Assessment, 73, 422– 448. 462 GARB, WOOD, LILIENFELD, AND NEZWORSKI Archer, R. P., & Krishnamurthy, R. (1997). MMPI–A and Rorschach indices related to depression and conduct disorder: An evaluation of the incremental validity hypothesis. Journal of Personality Assessment, 69, 517–533. Aronow, E. (2001). CS norms, psychometrics, and possibilities for the Rorschach technique. Clinical Psychology: Science and Practice, 8, 383–385. Baity, M. R., & Hilsenroth, M. J. (1999). Rorschach aggression variables: A study of reliability and validity. Journal of Personality Assessment, 72, 93–110. Ball, J. D., Archer, R. P., Gordon, R. A., & French, J. (1991). Rorschach depression indices with children and adolescents: Concurrent validity findings. Journal of Personality Assessment, 57, 465– 476. Berg, J. L., Gacono, C. B., Meloy, J. R., & Peaslee, D. (1994). A Rorschach comparison of borderline and antisocial females. Unpublished manuscript. Blais, M. A., Hilsenroth, M. J., & Fowler, J. C. (1998). Rorschach correlates of the DSM–IV histrionic personality disorder. Journal of Personality Assessment, 70, 355–364. Caine, S. L., Frueh, B. C., & Kinder, B. N. (1995). Rorschach susceptibility to malingered depressive disorders in adult females. In J. N. Butcher & C. D. Spielberger (Eds.), Advances in personality assessment (Vol. 10, pp. 165–174). Hillsdale, NJ: Erlbaum. Conners, K. (1989). Manual for Conners’ Rating Scales. North Tonawanda, NY: Multi-Health Systems. Erstad, D. (1996). An investigation of older adults’ less frequent human movement and color responses on the Rorschach (Doctoral dissertation, Marquette University, 1995). Dissertation Abstracts International, 57, 4084B. Exner, J. E. (1974). The Rorschach: A comprehensive system (Vol. 1). New York: Wiley. Exner, J. E. (1991). The Rorschach: A Comprehensive System. Vol. 2: Interpretation (2nd ed.). New York: Wiley. Exner, J. E. (1992). Some comments on “A Conceptual Critique of the EA:es comparison in the Comprehensive Rorschach System.” Psychological Assessment, 4, 297–300. Exner, J. E. (1993). The Rorschach: A comprehensive system. Vol. 1: Basic foundations (3rd ed.). New York: Wiley. Exner, J. E. (2001). A Rorschach workbook for the Comprehensive System (5th ed.). Asheville, NC: Rorschach Workshops. Exner, J. E., & Bryant, E. (1975). Pursuit motor performance and the Ea– ep relation (Workshops Study No. 222). Unpublished manuscript. Exner, J. E., & Bryant, E. (1976). Mirror star tracing as related to different Rorschach variables (Workshops Study No. 240). Unpublished manuscript. Frueh, B. C., & Kinder, B. N. (1994). The susceptibility of the Rorschach Inkblot Test to malingering of combat-related PTSD. Journal of Personality Assessment, 62, 280 –298. Gacono, C. B., & Meloy, J. R. (1994). The Rorschach assessment of aggressive and psychopathic personalities. Hillsdale, NJ: Erlbaum. Gacono, C. B., Meloy, J. R., & Berg, J. L. (1992). Object relations, defensive operations, and affective states in narcissistic, borderline, and antisocial personality disorder. Journal of Personality Assessment, 59, 32– 49. Garb, H. N. (1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association. Garb, H. N., Wood, J. M., & Lilienfeld, S. O. (2000). The detection and assessment of child sexual abuse: An evaluation of the Rorschach, Thematic Apperception Test, and projective drawings. Manuscript in preparation. Garb, H. N., Wood, J. M., & Nezworski, M. T. (2000). Projective techniques and the detection of child sexual abuse. Child Maltreatment, 5, 161–168. Goode, E. (2001, February 20). What’s in an inkblot? Some say, not much. The New York Times, pp. D1, D4. Greene, R. L. (2000). The MMPI–2: An interpretive manual. Needham Heights, MA: Allyn & Bacon. Grove, W. M., & Barden, R. C. (1999). Protecting the integrity of the legal system: The admissibility of testimony from mental health experts under Daubert/Kumho analyses. Psychology, Public Policy, and the Law, 5, 224 –242. Gutin, N. J. (1997). Differential object representations in inpatients with narcissistic and borderline personality disorders and normal controls. Dissertation Abstracts International, 58(03-B), 1532. (UMI No. AAG9723797) Hamel, M., Shaffer, T. W., & Erdberg, P. (2000). A study of nonpatient preadolescent Rorschach protocols. Journal of Personality Assessment, 75, 280 –294. Howard, W. W. (1999). The utility of selected Rorschach indices of distress and attachment for differential diagnosis in a forensic setting (Doctoral dissertation, Georgia State University, 1998). Dissertation Abstracts International, 59, 5578B. Hunsley, J., & Bailey, J. M. (1999). The clinical utility of the Rorschach: Unfulfilled promises and an uncertain future. Psychological Assessment, 11, 266 –277. Ilonen, T., Taiminen, T., Karlsson, H., Lauerma, H., Leinonen, K. M., Wallenius, E., et al. (1999). Diagnostic efficiency of the Rorschach schizophrenia and depression indices in identifying first-episode schizophrenia and severe depression. Psychiatry Research, 87, 183–192. Jansak, D. M. (1997). The Rorschach Comprehensive Sys…