Thinking clearly about reliability: More critical corrections regarding the Rorschach Comprehensive System.

In this brief comment on J. M. Wood, M. T. Nezworski, and W. J. Stejskal's (1997) response to his article (Meyer, 1997a), the author documents how J. M. Wood et al. continue to make allegations based on a limited subset of the available literature. He also points out specifically how their criticisms regarding kappa, test-retest reliability, true score theory, score aggregation, and his meta-analysis are incorrect. He concludes that these new errors provide additional reasons to be cautious about the conclusions proffered in their other articles on the Rorschach. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reliability and validity of the Hamilton Depression Inventory: A paper-and-pencil version of the Hamilton Depression Rating Scale Clinical Interview.

A self-report, paper-and-pencil version of the Hamilton Depression Rating Scale (HDRS; M. Hamilton, 1960) was developed. This measure, the Hamilton Depression Inventory (HDI; W. M. Reynolds & K. A. Kobak, 1995) consists of a 23-item full form, a 17-item form, and a 9-item short form. The 17-item HDI form corresponds in content and scoring to the standard 17-item HDRS. With a sample of psychiatric outpatients with major depression (n?=?140 ), anxiety disorders (n?=?99), and nonreferred community adults (n?=?118), the HDI forms demonstrated high levels of reliability (rα?=?.91 to .94, rtt?=?.95 to .96). Extensive validity evidence was presented, including content, criterion related, construct, and clinical efficacy of the HDI cutoff score. Overall, the data support the reliability and validity of the HDI as a self-report measure of severity of depression. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Correction to Stice et al. (2000).

Reports 2 errors in the original article by E. Stice et al (Psychological Assessment , 2000, vol 12[2], 123–131). On page 130 in Appendix A under heading "Bulimia Nervosa," EDDS Item 8 response should be "greater than or equal to 2." Under heading "Binge-Eating Disorder," EDDS Item 7 response should be "greater than or equal to 2." (The following abstract of this article originally appeared in record 2000-03952-002.): This article describes the development and validation of a brief self-report scale for diagnosing anorexia nervosa, bulimia nervosa, and binge-eating disorder. Study 1 used a panel of eating-disorder experts and provided evidence for the content validity of this scale. Study 2 used data from female participants with and without eating disorders (N?=?367) and suggested that the diagnoses from this scale possessed temporal reliability (mean κ?=?.80) and criterion validity (with interview diagnoses; mean κ?=?.83). In support of convergent validity, individuals with eating disorders identified by this scale showed elevations on validated measures of eating disturbances. The overall symptom composite also showed test–retest reliability (r?=?.87), internal consistency (mean α?=?.89), and convergent validity… (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Big C, little c, big M, little m.

Comments on the article Multicultural experience enhances creativity: The when and how by A. K. Leung et al (see record 2008-03389-003) which discusses how many practices aimed at cultivating multicultural competence in educational and organizational settings (e.g., exchange programs, diversity education in college, diversity management at work) assume that multicultural experience fosters creativity. In line with this assumption, the research reported in this article is the first to empirically demonstrate that exposure to multiple cultures in and of itself can enhance creativity. The current author, Grant J. Rich, feels that despite the reviews of a number of recent studies that seem to justify at least to some degree the value of such programs for enhancing creativity, Leung et al left a number of vital questions unanswered or unasked. Leung et al seemed to focus the majority of their attention on little c creativity and Big M multicultural experience. Rich hopes that their continued endeavors focus on the relationships between other forms of creativity and other forms of multicultural experience as well. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The limitations of behavior-genetic analyses: Comment on McGue, Elkins, Walden, and Iacono (2005).

This article takes issue with the behavior-genetic analysis of parenting style presented by M. McGue, I. Elkins, B. Walden, and W. G. Iacono (see record 2005-14938-011). The author argues that the attribution of their findings to inherited genetic effects was without basis because McGue et al. never indicated how those genetic effects manifested themselves. Instead, McGue et al. neglected important, and inevitable, developmental effects that most developmental psychologists understand to influence parent and adolescent behavior. The author also suggests that there is great merit in adopting the approach of developmental systems theory in understanding McGue et al.'s findings in particular and all developmental phenomena in general. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The etiology of dissociative identity disorder: Reply to Gee, Allen and Powell (2003).

Responds to a comment by T. Gee et al (see record 2003-04125-019) about the authors' original article (see record 1999-03012-003) which studied how clinicians treating dissociative identity disorder (DID) were or were not gathering and utilizing corroborative data. Results of this study show that many of the symptoms associated with DID were displayed by patients before entering therapy or prior to diagnosis. Gee et al argue that these results do not refute the sociocognitive model (a modified iatrogenesis model) of DID, but actually support it. The present authors maintain that their data did clearly contradict the iatrogenesis position. It is asserted that Gee et al's comment illustrates many important flaws of the iatrogenic DID argument, primarily that it is unscientific by virtue of being nondisconfirmable, and that it can only be maintained by adhering to false assumptions about DID and by misinterpreting data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reliability and validity of the situational interview for a sales position.

The reliability and validity of a situational interview (e.g., Latham, Saari, Pursell, & Campion, 1980) were examined for a sales position. In a pilot study, the interrater reliability of the interview was higher than that typically observed for interviews (r?=?.84). However, the estimate was probably inflated because the reliability was computed on the same data used in the item analysis phase of interview development. In a predictive validation study, the situational interview was shown to be valid (r?=?.45) in the prediction of sales productivity. After correcting for attenuation in the criterion, a validity coefficient of .47 was observed. Future research directions with respect to the situational interview are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distinguishing anxiety and depression: Reexamination of the reconstructed Hamilton scales.

The discriminant validities of the original and the reconstructed Hamilton anxiety and depression scales (J. H. Riskind et al, 1987) were compared in patients who had principal Diagnostic and Statistical Manual of Mental Disorders-III—Revised (DSM-III—R) anxiety disorders with or without 1 or more comorbid mood disorders. The reconstructed anxiety and depression scales had better discriminant validity (scale intercorrelation?=?.61) than the original scales did (r?=?.78). However, the reconstructed scales shared considerable variance (about 37%), which was significantly higher than the shared variance (about 2%) reported by Riskind et al. Discriminant analyses showed that the reconstructed scales did not distinguish anxiety patients with comorbid mood disorders from those without comorbid mood disorders better than the original scales did. However, the reconstructed scales eliminate item overlap, an obvious source of artifactual correlations between scores on the Hamilton anxiety and depression scales. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Finding relations between input and outcome in language acquisition.

J. N. Bohannon, R. J. Padgett, K. E. Nelson, and M. Mark (see record 199601760-018) raised numerous points of disagreement with the analyses of J. L. Morgan, K. Bonamo, and L. L. Travis (see record 1995-24603-001) concerning children's use of parental recasts, responses potentially providing negative evidence. Claiming to show that the bivariate time-series analyses used by Morgan et al. were inherently flawed, Bohannon et al. argued that training studies offer a preferable means for examining contributions of language input. In this article, multiple weaknesses of training studies are noted; at best, such studies may yield information on the sufficiency of aspects of language input but can yield none on their necessity. The failure of Bohannon et al.'s time-series analyses to distinguish among varying models of recast function is shown to be attributable to confounding of parameters and idiosyncratic assumptions adopted in generating simulated data from these models. Contrary to Bohannon et al.'s assertions, bivariate time-series analyses of observational data may provide invaluable tools for discerning signs and magnitudes of relations among variables in language development. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of self-evaluation of ability: A review and meta-analysis.

Reviews 55 studies in which self-evaluations of ability were compared with measures of performance to show a low mean validity coefficient (mean r?=?.29) with high variability (SD?=?.25). A meta-analysis by the procedures of J. E. Hunter et al (1982) calculated sample-size weighted estimates of –- r and SDr and estimated the appropriate adjustments of these values for sampling error and unreliability. Among person variables, high intelligence, high achievement status, and internal locus of control were associated with more accurate evaluations. Much of the variability in the validity coefficients (R?=?.64) could be accounted for by 9 specific conditions of measurement, notably (a) the rater's expectation that the self-evaluation would be compared with criterion measures, (b) the rater's previous experience with self-evaluation, (c) instructions guaranteeing anonymity of the self-evaluation, and (d) self-evaluation instructions emphasizing comparison with others. It is hypothesized that conditions increasing self-awareness would increase the validity of self-evaluation. (84 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Problems with a monolithic APA style.

Comments on the article by R. Madigan et al (see record 1995-31466-001) analyzing the epistemology of APA style. The author argues that critics both in and beyond psychology have challenged the epistemology and details of APA style, pointing to the constraints it places on reporting and knowledge-making processes. References are made to specific problems with APA style including its embodiment of a behaviorist set of assumptions about human beings that many psychologists consider flawed. The author states that in their analysis, Madigan et al relied on the work of scholars outside the discipline. Although the Madigan et al take a sunnier view of APA style than the author, he does agree with them that much can be learned by talking to people beyond one's disciplinary boundaries. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Individual differences in prototypicality judgments about trait categories.

Three studies demonstrated substantial individual differences in prototypicality judgments about trait categories. In Study 1, subjects twice rated instances of four personality trait categories, four concrete object categories, and one definitional category. For all nine categories, agreement between individuals was substantially lower (median r?=?.43) than the retest reliability of individual ratings (median r?=?.86). In Studies 2 and 3, subjects twice rated the prototypicality of characters from the film Twelve Angry Men as instances of seven trait dimensions. Agreement between individuals was again found to be substantially lower (median r?=?.56) than the retest reliability of individual ratings (median r?=?.81). These results may help to explain the unreliability of trait ratings based on examples of behavior (e.g., evaluation of employees or patients) and suggest that individual differences in the perception of prototypicality should be taken into account when comparing cross-situational with temporal consistency of behavior. Most generally, the results suggest a reconsideration of the nature of implicit personality theory. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Clinical Judgment in Science: Reply.

Replies to comments published by M. S. Schulz and R. J. Waldinger (see record 2005-11115-010), J. M. Wood and M. T. Nezworski (see record 2005-11115-011), and H. N. Garb and W. M. Grove (see record 2005-11115-012) on the original article by D. Westen and J. Weinberger (see record 2004-19091-002). Schulz and Waldinger (2005) make the important point that just as researchers can capitalize on the knowledge of experienced clinical observers through aggregation, they can aggregate the judgment of lay observers in assessing phenomena such as emotion. The reason, as they articulate, is that skills such as "reading" emotion from facial expression, tone of voice, posture, and the constellation of cues provided in everyday life are an area of expertise for most people, one that is now often called social or emotional intelligence. As psychometricians have known for years, one can increase reliability in many different ways. The comments by Wood and Nezworski (2005) and Garb and Grove (2005) do not address our central thesis--namely, the importance of distinguishing two meanings of clinical. The point of the sentence around which Wood and Nezworski (2005) build their comment was simply that the same biases widely attributed to clinicians are common in scientists as well--a point for which we would be delighted to take credit, but it is one that was actually made much more elegantly by the historian and philosopher of science Thomas Kuhn (1962). The authors respond to Wood and Nezworski's (2005) specific concerns about misrepresentation. In their comment, Garb and Grove (2005) challenge us to document our view that anticlinician prejudice is widespread among many academic clinical psychologists. As research on implicit prejudice suggests, surveys of academic clinical psychologists might indicate little about their implicit attitudes, as evident in Garb and Grove's apparent lack of recognition of the offensive nature of comparing a clinician's attempt to revise his or her understanding when the patient says "I don't think what you just said is right" to astrology and Barnum effects. We appreciate Garb and Grove's (2005) point about potential differential effects of training and experience on reliability and validity of clinical judgment. The data they cite are important and bear consideration. We would offer two caveats, however. Finally, we cannot help but note that this series of comments and our reply to them provide a prototypical example of "clinical" judgment in science--that is, subjective, informal aggregation of data, often leading to a "gestalt" judgment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: Reliability and validity of uniqueness scoring in creativity assessment.

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. In Study 1, Silvia et al. (2008) criticized the uniqueness scoring of Wallach and Kogan (1965). The uniqueness scoring has a virtue that single rater may be enough to rate, and it is characterized by the assignment of points to uncommon responses in a pool of sample's responses. The first criticism for uniqueness scoring is that uniqueness scores increase as a subject produces more responses, resulting in confounding of uniqueness and fluency. The second criticism relates to the ambiguity of statistical rarity pursued by uniqueness scoring in that uniqueness does not guarantee creativity. When a mundane unique response is misperceived as creative, reliability is threatened. Some bizarre, grotesque, or inappropriate responses in the pool of responses may be assigned a point, causing the validity to be threatened. The third criticism raised by the authors is that the uniqueness scoring system penalizes large samples in that it is less probable for a response in a larger sample of people to appear unique. However, the subjective scoring system has other deficits and is never free from the first two criticisms. The third criticism is, however unfounded; rather, the uniqueness scoring system is in a better position to capture the construct of creativity through better accessibility to large samples. The authors' (Silvia et al., 2008) three criticisms will be discussed one by one. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measuring general social support in alcoholic patients: Short forms for perceived social support.

Previous studies identify a positive association between social support and favorable posttreatment functioning. The next step for this line of inquiry is to identify the mechanisms through which social relationships affect alcohol treatment outcome by discrimination of the types and sources of social support. This article presents versions of the Perceived Social Support instrument (M. Procidano & K. Heller, 1983), which measures support from both family and friends. The item sets are reduced from 20 to 7 and ask identical questions from these two support sources. The internal consistency for the Family scale (α?=?.84) and for the Friends scale (α?=?.81) is acceptable. Test-retest reliability is also good for both the Family (r?=?.94) and the Friends (r?=?.88) scales. Correlations with social functioning and alcohol involvement are similar between the short scales and the original 20 items. Normative levels of family and friends support by gender are presented. These scales expand the ability to include a multifaceted social support construct in comprehensive assessments of alcohol treatment populations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Psychometric properties of the Questionnaire on Stress in patients with Diabetes—Revised (QSD-R).

This study reports the psychometric properties of a 45-item diabetes-specific questionnaire, the Questionnaire on Stress in Patients with Diabetes—Revised (QSD-R), a modified and shortened version of the QSD (G. Duran et al; see record 1996-18805-001). The QSD-R was filled out by 1,930 individuals with insulin-dependent diabetes mellitus and noninsulin-dependent diabetes mellitus. Eight consistent scales were identified (values of Cronbach's α: .69–.81). The test-retest reliability for the total score after a 5-week interval was rtt?=?.63. The results provide evidence for the reliability and validity of this instrument. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Consequences and causes of correlations in process dissociation.

In T. Curran and D. L. Hintzman's (see record 1995-42725-001) article, the authors explained how violations of the independence assumption could affect the process-dissociation procedure and presented evidence that was consistent with the hypothesized effects of independence violations. L. L. Jacoby, I. M. Begg, and J. R Toth (see record 84-21424) argued that independence violations could not account for Curran and Hintzman's results. In this reply, the current authors demonstrate that even moderate correlations between recollective and automatic word-stem completion can cause paradoxical dissociations like those the authors previously reported, and they explain how Jacoby et al came to their contrary conclusion. Second, the authors argue that discussion by Jacoby et al of process-dissociation "boundary conditions" is consistent with Curran and Hintzman's original claims. Finally, the authors discuss problems with the evidence for independence that Jacoby et al presented in their Table 1, including their questionable criteria for excluding experiments from the table and the fact that they did not consider statistical power. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Grouped arrays versus object-based representations: Reply to Kramer et al. (1997).

An important issue in attention research concerns the representational format from which attention selects. S. P. Vecera and M. J. Farah (see record 1994-32352-001) presented results that they argued demonstrated attentional selection from a spatially invariant object representation. In their comment, A. F. Kramer, T. A. Weber, and S. E. Watson (see record 84-18014) questioned the interpretation of these results, and they presented evidence consistent with selection from a grouped location-based representation. In this reply, the author argues that although an absence of spatial, or distance, effects may be ambiguous as to whether attention is selecting from an object-based representation or from a location-based representation, there are computational considerations that favor object-based selection in certain tasks. The author concludes with a discussion of how object-based and location-based representations might interact with one another, thereby providing a possible explanation of Kramer et al.'s results. Such an account may lead to an understanding of how multiple forms of attentional selection may coexist in the visual system. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of a semistructured clinical interview for suicidal behaviors in adolescents.

Describes the development of the Suicidal Behaviors Interview (SBI) developed by W. M. Reynolds (in press), a semistructured clinical interview measure of suicidal behaviors designed for use with adolescents. Ss were 352 adolescents, ages 12–19 yrs. Results indicate high internal consistency (rα?=?.92) and interrater reliability (rrr?=?.97, ricc?=?.99) and significant correlations with self-report measures of depression (r?=?.47) and suicidal ideation (r?=?.62 and .68). Factor analysis of SBI items resulted in a 3-factor solution that was parsimonious with the general formulation of suicidal behavior represented by the SBI. Examination of SBI factors indicated relatively high levels of reliability (rα?=?.83 to .90) and moderate intercorrelations (r?=?.49 to .66) among the factors. The results of this investigation suggest that the SBI is a psychometrically sound clinical interview for the evaluation of suicidal behaviors of adolescents. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Support for the discharge model of emotion: A reply to Zuckerman.

Responds to criticisms by M. Zuckerman (see record 1984-11616-001) of a study by the present author et al (see record 1983-05644-001) which reported evidence in support of the discharge model of the relation between facial expressivity and physiological reactivity. It is argued that Zuckerman's challenge to the Notarius et al study and conclusions is unfounded for 2 reasons: (a) Zuckerman's critique contains several interpretative errors of the Notarius et al study, and (b) subsequent data analysis, recommended by Zuckerman, overwhelmingly supports the original interpretation of the study. (2 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)